标签：#scala

Ian Nowland discusses why conventional CI/CD practices break down for stateful, core infrastructure. Drawing from his leadership at AWS and Datadog, he shares actionable techniques for safe progressive deployments, synthetic testing in production, and mitigating blast radius in complex software platforms. By Ian Nowland

API documentation has a reliability problem. The code gets updated; the OpenAPI spec gets forgotten. The spec gets updated; the TypeScript client doesn't regenerate. By the time an enterprise client asks for your API contract, the document you hand them describes a system that no longer exists. Baklava, an open-source library by Iterators , solves this structurally: documentation is generated from the tests that verify your actual API behaviour, so it cannot drift. The problem Documentation drift is the default state of any API that lives long enough. The causes are well-understood: docs and code are maintained separately, documentation updates require extra discipline at every PR, and no automated check catches a route signature change that wasn't reflected in the OpenAPI file. The consequence is real. Clients building against a stale spec hit integration errors in production. Internal teams onboarding to a service spend hours reconciling the documented contract with actual behaviour. TypeScript front-ends break when an API response field changes without a corresponding client update. The problem compounds as the API grows. The solution Baklava integrates into your existing test suite. When routing tests run, baklava observes each request and response, infers the API surface, and generates documentation as a test output, not as a separate build step, not as a manually-maintained file. In baklava, the test is the documentation spec. Instead of a standard assertion block, each route is defined with path() , supports() , and onRequest() scenarios that both verify the API behaviour and describe it for documentation output: ​`// The test IS the documentation spec class UserApiSpec extends AnyFunSpec with BaklavaPekkoHttp[Unit, Unit, ScalatestAsExecution] with BaklavaScalatest[Route, ToEntityMarshaller, FromEntityUnmarshaller] { path("/users/{userId}")( supports( GET, pathParameters = p Long , summary = "Get user by ID" )( onRequest(pathParameters = 1L) .respondsWith Use

Meryem Arik discusses strategies for designing low-cost LLM inference architectures for high-volume, non-real-time workloads. She explains how software architects and engineering leaders can achieve order-of-magnitude cost reductions by making critical trade-offs across hardware, inference runtimes, speculative decoding, and smart queue reordering. By Meryem Arik

This is Part 10 of my "From One User to One Million" series, where we'll build an understanding of System Design by following a simple application as it grows from a single user to millions. Instead of memorising technologies, we'll learn why they exist by solving real problems as they appear. In Part 9, we solved the problem of data that had grown too large for a single database. We split it across multiple shards, each holding its piece of the whole, so that no single machine ever had to carry everything. At that point, the architecture could scale in almost every direction we'd tried to push it. Traffic was distributed across application servers. Repeated database work was absorbed by the cache. Read traffic was spread across replicas. Data itself was partitioned across shards. And yet. We ended Part 9 by noticing something that none of those solutions addressed. Some user requests trigger a lot of downstream work. Saving an order is one thing. But saving the order, sending a confirmation email, generating an invoice, updating inventory, firing off a notification, recording an analytics event, triggering the recommendation engine: that's an entirely different conversation. Right now, all of that happens before the user gets a response. The question we left with was this: what if they didn't have to wait for all of it? -- Section 1: The User Doesn't Need Everything Right Now Before we look at any solution, it's worth asking a simpler question. When a user places an order, what do they actually need to know before they can move on? They need to know the order was received. They need confirmation that the important thing happened: their money was accepted, their items are reserved, the transaction is real. That's it. That's what they're waiting for. They do not need to wait for the confirmation email to land in their inbox. They do not need to wait for the invoice to be generated and stored somewhere. They do not need to wait for the analytics system to record that

Martin Spier explains how agentic workflows dramatically increase code change volume at OpenAI. He discusses the hidden systemic performance costs of rapid shipping beyond GPUs, and shares how deploying always-on AI agents automates profiling, regression detection, and continuous optimization to maintain product speed and scalability at massive global scale. By Martin Spier

JioHotstar explains the distributed architecture behind its real-time ad request workflow, covering ad decisioning, waterfall tiering, pacing algorithms, latency optimization, and service coordination required to select and deliver personalized advertisements during streaming playback at scale. By Leela Kumili

This is Part 7 of my "From One User to One Million" series, where we'll build an understanding of System Design by following a simple application as it grows from a single user to millions. Instead of memorising technologies, we'll learn why they exist by solving real problems as they appear. Last time, we ended on a question that sounded simple but isn't. Aisha updated her profile picture. Her new photo is now saved in the database. But the cache is still holding onto the old one, completely unaware that anything changed. So every request for Aisha's profile gets served the old data. Confidently. Instantly. Incorrectly. How does a cache know when the data it's holding is no longer correct? Think about what we've actually built at this point. We have an application that responds fast, scales horizontally, and avoids hammering the database with repeated identical queries. From a performance standpoint, it looks great. But Aisha's friends are loading her profile and seeing a photo she replaced five minutes ago. The system isn't slow anymore. It's wrong. Speed and correctness are two different things. We optimized hard for one, and quietly broke the other. Engineers have a name for this problem: cache invalidation . It refers to the challenge of keeping the data in your cache consistent with the data in your database, as that underlying data changes over time. It turns out to be one of the genuinely hard problems in building software systems. Not hard in a complicated-algorithm way. Hard in the way that every solution has a catch, and the right answer always depends on what you're willing to accept. Let's think through it together. -- Section 1: When Cached Data Lies It's worth sitting with the problem a little longer before rushing to fix it, because the damage stale data can cause varies enormously depending on what's being cached. Consider a few examples. Your application caches the list of trending articles. An hour later, the list has changed. New articles have ri

"We use Redis Cluster" can mean two very different things: Our dataset is distributed across Redis nodes. Every individual data structure is distributed across Redis nodes. The first can be true while the second is false. That distinction matters for leaderboards. In Podium , each leaderboard uses several Redis keys and atomic Lua scripts. Redis Cluster helps us scale a large fleet of independent leaderboards, but it cannot split one giant sorted set across primaries. We are sharing this architecture because "Redis Cluster scales horizontally" is true only after you define what the system actually shards. TeneficGames / podium High-performance, Redis-backed leaderboards for games and competitive applications. Podium High-performance, Redis-backed leaderboards for games and competitive applications. Podium provides ready-to-run HTTP and gRPC APIs for scores, ranks, seasons, and player-relative views. It is designed for backend teams operating large fleets of independent leaderboards without provisioning each leaderboard in advance. Fair, deterministic ordering when scores are equal. Single and bulk score updates, including multi-leaderboard fan-out. Standalone Redis and real Redis Cluster integration coverage. Deploy one multi-architecture OCI image with Docker, containerd, Kubernetes or another OCI-compatible runtime. Quickstart · Performance · API · Documentation · Helm chart · Docker Hub · GHCR Quickstart Start Redis 8.2 and the latest stable Podium image: docker network create podium docker run --detach --name podium-redis --network podium redis:8.2-alpine docker run --detach --rm --name podium \ --network podium \ --publish 8880:8880 \ --publish 8881:8881 \ --env PODIUM_REDIS_HOST=podium-redis \ --env PODIUM_REDIS_PORT=6379 \ trungdlp/podium:latest start Verify the service: curl http://localhost:8880/healthcheck WORKING Submit two equal scores: curl --request … View on GitHub Here is how the design works, why hash tags are necessary, and where the scaling bounda

The engineering team at Zalando recently described the design and implementation of an in-process, client-side load balancer for a high-throughput API handling around 1 million requests per second. The result was more predictable latency, a drop in infrastructure costs, and better visibility into where failures actually originate. By Renato Losio

In a recent article, Datadog engineer Arnold Wakim shared what worked, what didn't, and the lessons they learned while evolving a critical production system using AI to overcome hard limits in its storage backend and significantly improve performance. By Sergio De Simone

Both vulnerabilities allow untrusted users to gain root privileges.

Netflix engineers introduced dynamic partition splitting for Cassandra to address wide partitions in time series workloads. The metadata-driven approach detects oversized partitions, splits them smaller units, and routes reads across child partitions. Netflix reported lower read latency from seconds to milliseconds, reduced timeouts, and improved cluster stability while maintaining transparency. By Leela Kumili

The product drop had been planned for months. The direct-to-consumer subscription business had run three separate load tests, provisioned extra capacity for the launch window, and staffed a warroom across two time zones. The drop itself went cleanly. Two hours in, an unrelated video from a creator with a large following mentioned the product without warning, and the sign-up flow collapsed under a rush of new members for twenty-eight minutes. Customers were told the site was busy and to try again later. Some did. Most did not. The refund exposure was manageable. The customer acquisition exposure was not. What went wrong is not the interesting question. The system was under-provisioned for a specific traffic shape it had not seen before, and the team fixed it. The interesting question is what happened seven weeks later. A weather event redirected a wave of app traffic in an entirely different sector, at midnight on a Tuesday, without any warning. That system held, because a small group of engineers had spent those seven weeks quietly rebuilding assumptions about when peak load happens and what it looks like. The lesson from the product drop was not "provision more capacity for product drops." The lesson was that the mental model of peak load as a scheduled event had stopped being useful. This is another post in our series on the engineering layer underneath enterprise strategy. The previous post ( Sovereignty Versus Efficiency ) argued that sovereignty has become an architectural property that procurement cannot solve on its own. This post makes an analogous argument about load. Across banking, media, retail, travel, restaurant chains, and sport, the architectures built to survive named events are increasingly the wrong architectures for the traffic these businesses now routinely encounter. The discipline required has moved closer to what telecommunications engineers have always done, while the cost models have not caught up. What peak load used to mean For most of th

Quando comecei a trabalhar com aplicações descentralizadas há mais de uma década, lembro bem da frustração de pagar US$ 50 em taxas de transação para mover alguns tokens na rede Ethereum durante um pico de congestionamento. Era um problema técnico que ameaçava inviabilizar todo o ecossistema. Hoje, observo com entusiasmo profissional como as soluções de Layer 2 transformaram radicalmente esse cenário, abrindo portas para casos de uso que antes eram economicamente impraticáveis — especialmente aqui no Brasil, onde a tokenização de ativos e os pagamentos em stablecoins crescem em ritmo acelerado. O problema fundamental: o trilema da escalabilidade Para entender por que as soluções de segunda camada são tão importantes, precisamos compreender o trilema da blockchain proposto por Vitalik Buterin. Uma rede precisa equilibrar três pilares: descentralização, segurança e escalabilidade. O Ethereum, em sua arquitetura original, priorizou os dois primeiros, processando apenas cerca de 15 a 30 transações por segundo (TPS) na camada base. Para se ter dimensão, redes de pagamento tradicionais como a Visa processam milhares de transações por segundo. Quando o DeFi explodiu em 2020 e 2021, e novamente com o boom dos NFTs, a rede simplesmente não dava conta da demanda. As taxas de gas dispararam, e usuários comuns foram literalmente expulsos pelo custo. Em meus projetos de consultoria, atendi empresas brasileiras que desistiram de iniciativas Web3 justamente porque os custos operacionais inviabilizavam o modelo de negócio. A pergunta que sempre me faziam era: "Como cobrar R$ 5 de um cliente se a taxa da transação custa R$ 30?". A resposta estava — e está — nas camadas de segunda geração. Como funcionam as soluções de Layer 2 O conceito central das soluções de Layer 2 é elegante: em vez de processar todas as transações diretamente na blockchain principal (Layer 1), executamos a maior parte do processamento "fora da cadeia" e depois enviamos apenas uma prova compacta de volta para o

Erik Steiger discusses the operational pain of legacy PDF generation in regulated banking and manufacturing. He explains how transitioning from resource-heavy engines like Puppeteer and LaTeX to a serverless Rust architecture powered by Typst can drop render latencies below 2ms. He shares how applying Git and Docker concepts to template registries ensures ironclad compliance and rapid debugging. By Erik Steiger

Slack has outlined how its AI serving infrastructure evolved through four distinct phases, moving from a self-managed Amazon SageMaker deployment to a multi-cloud architecture spanning AWS Bedrock and Google Cloud Vertex AI. By Matt Foster

The speakers discuss the architectural challenges of executing safe data deletion across distributed datastores. Balancing durability, availability & correctness, they explain how to orchestrate multi-system deletion propagation without impacting live traffic. They share lessons on controlling tombstone accumulation, building continuous audit loops, and gaining trust with a centralized platform. By Vidhya Arvind, Shawn Liu

ᓯᐅᓇᕐᑕᖅ — Inuktitut for "that which lies ahead; a purpose" 🗣️ On the Name Full disclosure: I named this repository at 2 AM, which is probably when most repository names are decided. Siunertaq comes from Kalaallisut (West Greenlandic), a polysynthetic language — the kind where a single word can encode an entire clause's worth of meaning through agglutination and incorporation. I'm a bit of a grammar nerd, and polysynthetic languages have always fascinated me precisely because of how much structure they pack into a single morphological unit. One word carries subject, object, tense, evidentiality, and mood all at once, with none of it ambiguous if you know the grammar. That felt like the right metaphor for what this project is trying to do: pack a build graph's topology, its norm constraints, and its effect ordering into a single type-checkable unit — where the structure does the work, not the runtime. The word itself means something like "that which lies ahead; a purpose" — which seemed fitting for a tool that reasons about what needs to happen before anything actually runs. 🧵 TL;DR What if your task orchestration system couldn't even represent an ill-ordered build? Not "it would fail at runtime" — but "the type system refuses to construct the value in the first place." That's the idea behind Siunertaq : a Scala 3 project that combines Dhall (a total, non-Turing-complete configuration language), Cats Effect (purely functional async runtime), and a BSD Quiver model (directed Banach space graph) to make inconsistent build topologies structurally non-representable . This post walks through the design — with analogies aimed squarely at the Typelevel community — and closes with some thoughts on what modern AI-assisted development actually looks like when you refuse to let the LLM take the easy path. 🤔 Why Yet Another Build/Orchestration Abstraction? Most task orchestrators model their dependency graph as a mutable Map[Task, List[Task]] or similar at runtime, then check for

The engineering team at Meta recently outlined how the company migrated a data ingestion platform that transfers several petabytes of MySQL social graph data daily to improve reliability and operational efficiency. The team used techniques like reverse shadowing and continuous checksum monitoring to ensure zero downtime during the transition. By Renato Losio