标签：#Spring

Episode 2 | Spring Core | Understanding IoC, Dependency Injection & Beans In Episode 1, I covered the WHY behind Spring — tight coupling, and how Spring takes over creating and providing objects (IoC + DI) instead of classes creating their own dependencies. This episode picks up from there with the parts I hadn't covered yet: how Spring actually does that under the hood — Beans, the Spring Container, and Component Scanning. 🔑 Keywords → 🧠 Understand → 💡 Why? → 💻 Practice → 🎯 Interview Questions → 🛠️ Project 🔑 Keywords for This Episode IoC & Dependency Injection (quick recap) Spring Bean Spring Container / ApplicationContext Component Scanning 1️⃣ Quick Recap: IoC & Dependency Injection From Episode 1: instead of a class creating its own dependency — public class TicketService { private TicketRepository repository ; public TicketService () { repository = new TicketRepository (); } } — Spring creates the dependency and hands it to the class. That's Inversion of Control (IoC) . In code, this usually looks like a constructor parameter: public class TicketService { private final TicketRepository repository ; public TicketService ( TicketRepository repository ) { this . repository = repository ; } } TicketService no longer says "let me create a TicketRepository." It says "I need a TicketRepository" — and Spring supplies one. That act of supplying it is Dependency Injection (DI) . IoC = who's in control of creating/managing objects → Spring. DI = how a class actually receives what it needs → passed in, not self-created. That's the recap. Now — where do these objects Spring creates actually come from, and where do they live? 2️⃣ Spring Bean — what Spring actually manages When Spring creates and manages an object for you, that object is called a Bean . This is the vocabulary you'll see everywhere in Spring code and docs, so it's worth being precise about it. @Service public class TicketService { } The @Service annotation is a signal to Spring: "this class should be managed b

Most gateway tutorials stop at "here's how you route a request." That's the easy 20%. The hard part is what happens when a client hammers you with requests, a downstream service falls over mid-traffic, or you're staring at a 500 trying to figure out which of your four services actually caused it. I wanted to build something that hits those problems on purpose, so I put together spring-gateway-sample : a public gateway , an api-server that fans out to two downstream services, and a full observability stack sitting behind all of it. It's not a real product and never will be. But I tried to make it behave like one — including the annoying bits, like config tradeoffs and races that most demos just quietly ignore. Stack, for context: Spring Boot 4.1, Spring Cloud Gateway on WebFlux, Resilience4j, Redis, Postgres, Keycloak, Prometheus/Grafana/Tempo/Loki, and a small Vue 3 app for throwing traffic at it from a browser. The system, in one request Browser (Vue traffic simulator) │ Keycloak PKCE login + API key ▼ Gateway ── JWT + API-key auth, Redis rate limiting ──▶ routes to │ ▼ api-server ── WebClient delegation, circuit breakers, Caffeine cache ──▶ │ │ ▼ ▼ product-service pricing-service (JPA / Postgres) (JPA / Postgres) Every hop re-validates the JWT on its own — defense in depth, so the gateway isn't the single thing standing between the internet and the data. The gateway also checks an API key on top, because a JWT tells you who the user is, not which client application is calling on their behalf. You need that second identity if you want per-client rate limits or the ability to revoke one app's access without touching anyone else's. Two checks, one specific order Every request needs a Keycloak JWT and an API key, and the order they're checked in isn't an accident: Missing or expired JWT → 401 , before the API key is even looked at. Valid JWT, bad API key → 401 , but a different error code. Both valid, wrong role → 403 . Why bother with the ordering? Because "you're no

There are four patterns that bring PQC into a Spring Boot fleet: encrypting payloads between services, locking down database fields, signing documents that need to hold up for decades, and moving service tokens off RS256. Along the way, we discuss why Harvest Now, Decrypt Later is already happening, and why none of this is production-safe until KMS or Vault is in place. By Pankaj Sharma

After a 10-week hiatus since the last batch of Spring ecosystem releases, there was a flurry of activity during the week of August 17th, 2026, highlighting first milestone releases of: Spring Boot, Spring Framework, Spring Data, Spring Security, Spring Integration, Spring HATEOAS, Spring Modulith, Spring Batch, Spring AMQP and Spring for Apache Kafka. By Michael Redlich

Introduction Authentication is one of those things that looks simple in a tutorial and becomes surprisingly complex in production. Between token storage, CSRF protection, refresh flows, and protected routing, there are many places to get it wrong—and getting it wrong has real security consequences. In two earlier posts, I covered pieces of this puzzle: Enabling CSRF in a JWT-Based React + Spring Boot Application and Storing Personal Information in React: sessionStorage vs Context API . This post ties those threads together into a complete, end-to-end authentication flow you can adapt for enterprise applications. We'll walk through the full journey: login → token issuance → secure storage → protected routes → token refresh → logout. Architecture Overview Before the code, here's the high-level flow: ┌──────────────┐ ┌──────────────────┐ │ React │ │ Spring Boot │ │ Frontend │ │ Backend │ └──────┬───────┘ └────────┬─────────┘ │ 1. POST /login │ │─────────────────────────>│ │ │ validate credentials │ 2. JWT (httpOnly cookie)│ issue access + refresh │<─────────────────────────│ │ │ │ 3. GET /protected │ │ (+ CSRF token) │ │─────────────────────────>│ validate JWT + CSRF │ 4. Protected data │ │<─────────────────────────│ │ │ │ 5. POST /refresh │ │─────────────────────────>│ rotate tokens │ │ │ 6. POST /logout │ │─────────────────────────>│ invalidate session Key Design Decisions Decision Choice Rationale Token storage httpOnly cookies Not accessible to JavaScript → mitigates XSS token theft CSRF protection Double-submit / token pattern Required when using cookies Token type Short-lived access + refresh Limits exposure window State management Context API for auth status Centralized, lightweight Why httpOnly cookies over localStorage? As I discussed in the storage blog, localStorage is readable by any script on the page—making it vulnerable to XSS. httpOnly cookies trade that risk for the need to handle CSRF, which we address below. Step 1: Backend — Login and Token Issuance

SkillShare is an academic social network developed to connect students through knowledge exchange, informal tutoring, and collaboration among users with different skills. The project aims to facilitate collective learning through a modern, dynamic, and responsive web platform. The project was developed as a Course Completion Project (TCC) for the Technical Course in Information Technology at the Escola Técnica de Brasilia (ETB).

In the previous parts, we connected Spring AI with cloud-based AI models. But there is one important question: What if you don't want to send your data to an external AI provider? What if you want to: Run an LLM on your own machine Develop AI applications without API costs Work without an internet connection Keep sensitive company data private Experiment with different open-source models Build AI features locally before moving them to production This is where Ollama becomes very useful. In this article, we will learn how to run a local LLM using Ollama and connect it with Spring AI . We will build a simple real-world AI Customer Support Assistant using Java, Spring Boot, Spring AI, and Ollama. What We Are Building Our application will look like this: User | | HTTP Request v +---------------------+ | Spring Boot API | +---------------------+ | v +-------------+ | Spring AI | | ChatClient | +-------------+ | v +--------+ | Ollama | +--------+ | v Local LLM (Llama/Qwen) | v AI Response | v User The important part is that the LLM is running locally . There is no need to send every prompt to OpenAI, Anthropic, or another cloud provider. 1. What Is Ollama? Ollama makes it easy to run open-source LLMs locally. Instead of calling a remote API like: Spring Boot | v OpenAI API | v Cloud LLM we can run: Spring Boot | v Spring AI | v Ollama | v Local LLM Ollama can run models such as: Llama Qwen Gemma Mistral DeepSeek and many other compatible models The exact models available change over time, so always check the Ollama model library before choosing one. 2. Why Run an LLM Locally? Imagine you are building an internal HR application. Employees may send questions such as: What is our maternity leave policy? or: What is the process for requesting annual leave? You may not want internal company information leaving your infrastructure. A local LLM can help: Employee | v Spring Boot | v RAG / Business Logic | v Ollama | v Local LLM This can provide a useful privacy boundary. However

In my last post I wrote about an error that cost me a full evening: my pom.xml had the MongoDB starter, but my code was still full of JPA annotations. The compiler kept saying cannot find symbol: class Entity . That post was about the error. This post is about the fix — every single line I had to change to move my Task Manager project from MySQL to MongoDB. If you are planning the same switch, this is the checklist I wish I had. 1. The dependency Before (MySQL + JPA): <dependency> <groupId> org.springframework.boot </groupId> <artifactId> spring-boot-starter-data-jpa </artifactId> </dependency> <dependency> <groupId> com.mysql </groupId> <artifactId> mysql-connector-j </artifactId> <scope> runtime </scope> </dependency> After (MongoDB): <dependency> <groupId> org.springframework.boot </groupId> <artifactId> spring-boot-starter-data-mongodb </artifactId> </dependency> One starter replaces two dependencies. And this is exactly where my problem started — I added the new one but never removed the old one, so half my code still compiled and half did not. Remove the JPA starter completely. If you leave it in, the jakarta.persistence annotations still resolve, and you will not notice you are mixing two worlds until something breaks at runtime. 2. application.properties Before: spring.datasource.url = jdbc:mysql://localhost:3306/taskmanager spring.datasource.username = root spring.datasource.password = yourpassword spring.jpa.hibernate.ddl-auto = update spring.jpa.show-sql = true After: spring.data.mongodb.uri = mongodb://localhost:27017/taskmanager Five lines became one. No ddl-auto because MongoDB has no schema to create. No dialect because there is no SQL being generated. The database and the collection are created automatically the first time you insert a document. 3. The model class This is where most of the work was. Here is my actual Task class after the migration: package com.taskmanager.task_manager ; import com.fasterxml.jackson.annotation.JsonIgnore ; import org.

The textbook version of the transactional outbox is tight. You save the domain entity and an outbox row in one local transaction. A background scheduler picks up PENDING rows and publishes them to Kafka. You never publish inside the request thread — no dual-write, no atomicity breach. The pattern closes the consistency gap. Then you load-test it. I ran 1,000 authenticated requests through my event-driven platform in 70 seconds. The gateway returned 201 for every one of them. The outbox absorbed every row. The consumer drained everything. By every visible metric the system looked healthy. Underneath that health, I found three production-grade problems the textbook never mentioned. What a correct implementation looks like Before the problems, the shape of the solution. The outbox publisher runs on a @Scheduled virtual-thread worker: @Scheduled ( fixedDelay = 5000 ) @Transactional public void publishPendingEvents () { List < OutboxEvent > batch = outboxRepository . findTop20ByStatusOrderByCreatedAtAsc ( OutboxStatus . PENDING ); for ( OutboxEvent event : batch ) { event . setStatus ( OutboxStatus . PROCESSING ); outboxRepository . save ( event ); try { kafkaTemplate . send ( event . getTopic (), event . getPayload ()). get (); event . setStatus ( OutboxStatus . PUBLISHED ); } catch ( Exception e ) { event . incrementRetryCount (); if ( event . getRetryCount () >= MAX_RETRIES ) { event . setStatus ( OutboxStatus . FAILED ); } else { event . setStatus ( OutboxStatus . PENDING ); } } outboxRepository . save ( event ); } } This is correct. The PROCESSING state prevents another scheduler instance from claiming the same row. The retry cap prevents infinite cycling. The PENDING fallback on transient errors gives the event another chance. The dual-write problem is genuinely closed. Here is what that correctness does not cover. Gap 1: Your throughput ceiling is a config line fixedDelay = 5000 means the scheduler runs every 5 seconds. findTop20 means it picks up 20 rows per cycl

This week, Anthropic's system-prompt release notes became the top story on Hacker News. The page is where Anthropic publishes the exact instructions that steer Claude on claude.ai and its mobile apps. It hit more than 550 points and 230 comments within a day, and the discussion is still going. The most interesting thing about the page is not any single rule. It is the size. Claude Opus 3's system prompt, dated July 12, 2024, is 358 words by my count. Claude Opus 5's, dated July 24, 2026, is 3,235 words. Nine times larger in two years. I have been building production AI systems with Spring Boot and Spring AI for over a year, and I run my own agent infrastructure. When the prompt that controls a frontier model grows ninefold, that is not an Anthropic curiosity. It is a warning and a playbook for every team shipping an AI product. Here is what is actually inside those 3,235 words, and what production teams should copy from them. What Anthropic actually published The release notes ( platform.claude.com/docs/en/release-notes/system-prompts ) are a changelog of system prompts for the consumer chat products. Two details on the page matter: These are not the API prompts. The page says claude.ai and the mobile apps "use a system prompt to provide up-to-date information, such as the current date, to Claude at the start of every conversation," and that "these system prompt updates do not apply to the Claude API." Models are now fixed snapshots. Since the Claude 4.6 generation, "each model ID is a single fixed snapshot," so each model has exactly one entry in the changelog. Simon Willison turned the page into a git repository ( github.com/simonw/research ) containing 29 prompt revisions across 17 models, each committed with the date from the source document. That means you can run git diff between any two versions of Claude's personality. It is a remarkable thing: the product spec of a frontier model, versioned like source code, and public. What the 3,235 words actually contain

Three days ago I knew exactly what a DeepSeek call cost me. I had wired DeepSeek V4 Pro 0813 into a Spring Boot app with Spring AI, and the math was simple: $0.435 per million input tokens, $0.87 per million output tokens, and a cache-hit rate so aggressive that long agent sessions stayed embarrassingly cheap ( I wrote up the integration ). Then the pricing update landed, and tokens suddenly have rush hour. DeepSeek's official announcement introduces peak and off-peak billing: off-peak rates are 50% lower than peak, and the new prices take effect today, August 16, 2026 at 16:00 UTC (10 PM in Dhaka). The headline reads like a discount. The fine print is a price increase, and the difference matters a lot if you run batch workloads or agentic tools. Full disclosure up front: the new billing starts today, so I have not run a real bill through it yet. What I have done is read the price table carefully, watched the Hacker News thread do the math for two days, and built a scheduling pattern in Spring Boot that shifts heavy work into the off-peak window. That pattern is what I want to show you, because the interesting part is not the announcement. It is what the numbers actually mean. What actually changed The pricing page now splits every price into peak and off-peak tiers. Peak hours are 01:00 to 04:00 UTC and 06:00 to 10:00 UTC. Every other hour is off-peak, which is 17 out of 24 hours. Here are the new per-1M-token rates, straight from the page: DeepSeek V4 Flash, off-peak: $0.22 input (cache miss), $0.66 output, $0.007 cache hit. DeepSeek V4 Flash, peak: $0.44 input, $1.32 output, $0.014 cache hit. DeepSeek V4 Pro, off-peak: $0.66 input, $1.98 output, $0.022 cache hit. DeepSeek V4 Pro, peak: $1.32 input, $3.96 output, $0.044 cache hit. The off-peak discount is real: every off-peak number is exactly half of its peak counterpart, which matches the announcement's "50% lower" claim. But compare those off-peak numbers to what DeepSeek charged before this change, and the pic

Yesterday morning my feed exploded with a model release again. But this one was different from the usual frontier drop. Qwen 3.8 27B hit the top of Hacker News and stayed there: at the time I checked, the thread had passed 1,194 points with 713 comments in under a day. That is the kind of heat normally reserved for a $5-per-million-token API announcement. The twist is that this is a dense 27-billion-parameter open model, Apache 2.0 licensed, that people are running on laptops. Simon Willison ran it on an M5 Max MacBook Pro through LM Studio with a 17GB GGUF file and spent 21 minutes watching it think about an SVG ( his comment ). I build production AI systems with Spring Boot and Spring AI, so my first question was not "how smart is it?" It was: can I call this thing from the code I already have, without a second SDK or a cloud account? The answer is yes, and the setup is smaller than the model's license file. Here is what shipped, what the community actually found when they ran it, and the exact Spring Boot wiring for a local Qwen 3.8 27B. What actually shipped Qwen 3.8 is the latest generation of Alibaba's open model family, and 27B is its compact dense member. The model card lists the headline details: A dense 27B vision-language model. A causal language model with a vision encoder, built on the Qwen3.5 architecture. It takes text, images, and video input. 262,144 tokens of native context. The card says it can be extended toward 1 million tokens with RoPE scaling (YaRN), though the card warns static YaRN can hurt performance on shorter inputs. FP8 quantization from the lab. The FP8 repo uses fine-grained fp8 with a block size of 128 and claims "performance metrics are nearly identical to those of the original model." Thinking on by default. Qwen3.8 operates in thinking mode by default, with three reasoning effort levels: xhigh , medium , and low . It also keeps reasoning context from earlier messages ( preserve_thinking ) for multi-step agent work. Multi-token pr

Most Spring Boot services end up with a scheduled job in them somewhere. A nightly reconciliation, a report, an export to some partner system. It starts small, and it goes in the main app because that's where the domain code already is. One artifact, one deployment, one pipeline. That's a real advantage and it's why most teams do it. This post is about when that stops being a good trade, how to split the job out, and when you shouldn't. The memory problem Look at how much memory each workload uses over a day. The API is fairly flat. Warm heap, connection pool, some caches. It moves with traffic but it doesn't swing much. The batch job uses close to nothing for 23 hours, jumps while it runs, then drops back to nothing. When both live in the same JVM, the pod has to be sized for the peak. So every replica of your API holds batch-sized memory all day, for a job that runs once. With three replicas you're reserving that headroom three times over so one job can use it once, at 2am. Memory limits are not like CPU limits CPU is compressible. Go over your CPU limit and the kernel throttles you. The app gets slower and keeps running. Memory doesn't work that way. There's no "run with less" mode. If the container goes over its memory limit, the kernel kills the process. What you get is a container that exited with code 137 (that's 128 + 9, where 9 is SIGKILL). What you don't get is anything useful in the logs. No OutOfMemoryError , no stack trace, no heap dump unless you configured one and it had time to write, no shutdown hook. The JVM was running fine, asked for another page of memory, and got killed for it. So a batch job sharing a pod with your API is a way for a nightly job to take down the pods serving traffic. If the job's working set grows (bigger dataset, a table that keeps growing, one unusually heavy day) the thing that dies is the API. There's a quieter version of the same problem. Even when the job stays under the limit, it allocates heavily and triggers longer GC

Querydsl is one of those libraries that everyone used for years and then quietly stopped updating. The 5.0 release has been "coming soon" since 2019. The GitHub shows commits but no milestone. The issue tracker has a thread titled "Is Querydsl dead?" with hundreds of comments. It's not dead. But if you're starting a new project in 2026 and you're picking between Querydsl and something that's actively maintained, has Spring Boot 4 support, works with MongoDB and in-memory collections, generates OpenAPI docs automatically, and has companion frontend libraries... well, you see where I'm going. This isn't a "Querydsl bad, Spring Filter good" article. Querydsl pioneered type-safe querying for Java and it deserves credit. But migrations happen, and if you're considering one, here's what the conversion looks like. Side-by-side: basic filtering Querydsl: QCar car = QCar . car ; BooleanExpression filter = car . year . gt ( 2020 ) . and ( car . km . lt ( 50000 )) . and ( car . color . eq ( Color . RED )); List < Car > results = new JPAQuery <>( entityManager ) . select ( car ) . from ( car ) . where ( filter ) . fetch (); Spring Filter (query string): @Filter Specification < Car > spec // URL: ?filter=year > 2020 and km < 50000 and color : 'red' List < Car > results = carRepo . findAll ( spec ); Spring Filter (programmatic builder): FilterNode filter = fb . field ( "year" ). greaterThan ( fb . input ( 2020 )) . and ( fb . field ( "km" ). lessThan ( fb . input ( 50000 ))) . and ( fb . field ( "color" ). equal ( fb . input ( Color . RED ))) . get (); Specification < Car > spec = converter . convert ( filter ); List < Car > results = carRepo . findAll ( spec ); Spring Filter (type-safe builder): FilterNode f = CarFilter . where ( fb ) . year (). greaterThan ( 2020 ) . and () . km (). lessThan ( 50000 ) . and () . color (). equal ( Color . RED ) . build (); Specification < Car > spec = converter . convert ( f ); List < Car > results = carRepo . findAll ( spec ); The type-safe bui

On behalf of the core Axelix team, and everybody who has contributed to the community, I want to declare: we finally did it. Axelix, finally, goes GA (Generally Available)! For those who do not know - Axelix is a product with an Open Source core, that allows you to discover the common problems, pitfalls and inefficiencies in Java applications at large scale. We're available on GitHub (btw - give us a star!). In this post, I want to share the story and the motivation behind the product overall. I hope you find it interesting. The Story. Big "Why" Behind Axelix Java is quite an interesting language and ecosystem in general. I think a lot of people will not argue that it is quite old, and it was one of the first so-called "Object-Oriented" languages, that actually gained massive adoption. It both was, and it still is the backbone of modern enterprise. For anyone who claims that Java is dead - I recommend checking the JetBrains State of Developer Ecosystem survey or even the Stack Overflow survey for 2025 (and Stack Overflow has, sadly, become a part of history). It is clear that Java as a language and the "ecosystem" around it (including Kotlin) is still relatively popular, and it remains true. Ecosystems around Languages The experienced developer knows that today's ecosystems that evolve around languages are typically very diverse. For example, let's talk about JavaScript. If we decide to run JavaScript on the server, then we're probably going to work with a database of some sort. Therefore, we're also going to need a framework, a library to work with the database, e.g. an ORM (I know that we may work without it but let's leave that aside). And in JavaScript, we have quite a lot of options: Prisma TypeORM DrizzleORM Kysely and so on. We can pretty safely state that Prisma ORM is probably the most used ORM on JavaScript . But notice that it is far from being the definitive JavaScript ORM. It is not like Prisma is the default choice and is by far the most popular ORM -

Understanding Java's Virtual Threads: Lightweight Concurrency in Action Java 21 introduced virtual threads as a stable feature (JEP 444), fundamentally changing how we approach concurrency on the JVM. In this post, we'll explore what virtual threads are, why they matter, and how to use them effectively. The Problem with Platform Threads Traditional Java threads—now called platform threads —are thin wrappers around operating system threads. Each one consumes roughly 1MB of stack memory and involves the OS scheduler for context switching. This makes them expensive: java // Creating thousands of platform threads is costly for (int i = 0; i < 10_000; i++) { new Thread(() -> { // blocking I/O ties up an OS thread processRequest(); }).start(); } In high-throughput server applications, the classic "thread-per-request" model hits a ceiling because you simply cannot create enough OS threads. Enter Virtual Threads Virtual threads are managed by the JVM rather than the OS. Many virtual threads run on a small pool of carrier platform threads. When a virtual thread blocks (e.g., on I/O), the JVM detaches it from its carrier, freeing that carrier to run other virtual threads. java // Creating a million virtual threads is perfectly fine try (var executor = Executors.newVirtualThreadPerTaskExecutor()) { IntStream.range(0, 1_000_000).forEach(i -> { executor.submit(() -> { Thread.sleep(Duration.ofSeconds(1)); return i; }); }); } Key Benefits Cheap creation : Virtual threads start with a tiny stack that grows on demand. Familiar model : You write straightforward blocking code—no callbacks or reactive chains. Better scalability : Throughput is limited by resources, not thread count. Using Virtual Threads in Spring Boot As of Spring Boot 3.2, enabling virtual threads is a one-line configuration change: properties spring.threads.virtual.enabled=true This makes Tomcat handle each request on a virtual thread, allowing your application to serve many concurrent blocking requests without exha

You open a new Spring Boot project and you create a DTO, then an entity, and you’re staring at getters, setters, constructors, equals() , hashCode() , toString() . Someone on the team suggests to put lombok’s @Data , or “just slap @Builder on it, it’ll be cleaner.” Forty lines become five and it looks great in the PR. Then it hits a real codebase, OpenAPI generation doesn't behave the way the build expects. Hibernate meets an auto-generated equals() and gets confused about identity. Something throws through a generated builder hierarchy at 2 AM, and the method you need to inspect doesn't exist in any file you can open. Eleven years into enterprise Java, my rule is simple: Lombok doesn't touch core application behavior. Not because writing a getter is interesting - it isn't, but because the handful of lines it saves rarely covers the compiler magic, tooling friction, and debugging problems it adds to something that has to survive for years after you've moved on to another project. "It just removes boilerplate" Worth asking what's actually being removed, though. A getter is part of your public API. A setter is a mutation point someone decided to expose. A constructor defines what states an object is allowed to enter. equals() and hashCode() define identity. toString() is what shows up in your logs when things go wrong at 3 AM. Write those by hand and they live in the source - visible, searchable, debuggable, owned by whoever's reading the file. Generate them with Lombok and the behavior is still there, it's just moved somewhere you can't see it without a separate step. The annotation most people reach for first time is @Data : @Data @Entity public class CustomerEntity { @Id @GeneratedValue private Long id ; private String email ; @OneToMany ( mappedBy = "customer" ) private List < OrderEntity > orders ; } One line and you get getters, setters, toString() , equals() , hashCode() across every field. For a JPA entity that's already a problem before you've written any bus

Last week I changed a system prompt based on a feeling. It was the first prompt change after the evaluation harness from Part 8 went live, and I was completely sure about it. The target was the markdown table. Part 8's first nightly run caught the agent answering price comparisons with a markdown table that renders broken in the chat frontend. The fix looked obvious: add one line to the system prompt demanding plain text. I checked six conversations by hand. All six looked better. I was ready to ship it to production. Then I ran the comparison the way Part 8 promised: the same 40 cases, the same judge, two prompts. The old prompt won. Not by a little. It won 18 pairs, lost 10, and tied 12, and the judge's rationales made the reason visible. The plain-text line had also made the agent terse, and terse answers dropped the order summary that customers actually need. My confidence was a sample size of one. The dataset was the jury. This part is about the pattern that settled that argument: pairwise comparison, the LLM-as-a-judge pattern for A/B testing prompts and tool descriptions before they reach production. It is the harness from Part 8, upgraded to answer "which version is better?" instead of "is this version good?" The Problem With Ship-by-Feeling Every prompt edit is an experiment with one sample. You notice one conversation where the agent is verbose, you add "be concise", and the change ships because that one conversation got better. The dataset from Part 8 makes the agent measurable, but a nightly score cannot tell you whether a change helped. One night is noise, three nights is a signal, and by the time you have three nights of data you have already shipped the change to every user. The variable itself is the problem. A system prompt and a tool description are the two things in an agent you cannot unit test. Part 6 proved the code is bug-free. Part 8 proved the answers are good on a fixed dataset. Neither says anything about whether your new wording is better

You put a lot of work into authentication. A gateway validates the Keycloak JWT, maps realm roles to authorities, checks that the caller is allowed. By the time a request reaches your service, you know exactly who is calling. Then the request crosses into async land, and all of that evaporates. This is the story of the point where identity quietly disappears in an event-driven system, why the dead-letter queue is the worst possible place for it to disappear, and how I made the acting user as durable and replay-safe as the event itself. The flow everyone believes is fine The platform is a set of Spring Boot services: an API gateway in front, a user-service on MySQL, a notification-service on PostgreSQL, and Kafka carrying events between them. A user is created, an event is published, a notification is sent. Authentication is handled at the edge. The gateway is an OAuth2 Resource Server; it validates the token once and propagates the caller's identity downstream as headers: // api-gateway — IdentityPropagationFilter (@Order(2), after security) IdentityContext identity = identityContextExtractor . extract ( jwt ); // Always set all three headers (empty when absent) to mask any spoofed values. enrichedRequest . putHeader ( IdentityHeaders . USER_NAME , nullToEmpty ( identity . username ())); enrichedRequest . putHeader ( IdentityHeaders . USER_EMAIL , nullToEmpty ( identity . email ())); enrichedRequest . putHeader ( IdentityHeaders . USER_ROLES , identity . rolesAsString ( DELIM )); One detail here matters more than it looks. The headers are always overwritten , even when a claim is absent. If a client tries to inject X-User-Name: admin on the inbound request, the gateway stomps it with the validated value (or empty). Downstream trust in those headers is only safe because the perimeter guarantees they cannot be forged. Miss that, and you've built an impersonation API. So far, so good. The synchronous hop carries identity. The problem starts one line later. The hidden f

You attach the OpenTelemetry Java agent, point it at a collector, and within minutes Grafana is drawing a service map you never drew. A box for each service, arrows between them, latency on every edge. It feels like magic, and — more dangerously — it feels complete . "The agent traces everything" is the sentence repeated in every onboarding doc. This is the story of the moment that sentence stopped being true on my platform, why I'm glad it did, and the difference between a system that is working and a system you can actually see . The flow everyone trusts The platform is an event-driven set of Spring Boot services: an API gateway in front, a user-service backed by MySQL, a notification-service backed by PostgreSQL, and Kafka carrying events between them. A user is created, an event is published, a notification is sent. I didn't want to draw that topology. A hand-drawn architecture diagram is documentation that drifts — true the day you commit it, slightly wrong a month later, actively misleading after a quarter. I wanted the dependency graph generated from live traffic , so it would always reflect what the system actually does. Grafana Tempo does exactly this. Its service-graphs processor reads matched client/server span pairs out of trace data and emits a metric — traces_service_graph_request_total — that Grafana renders as a node graph. No edge is ever wired by hand. The topology is derived, continuously, from real spans. The edge that wasn't there I generated the graph and the synchronous edges lit up immediately: api-gateway → user-service user-service → MySQL notification-service → PostgreSQL Then I looked for the one edge I actually cared about — user-service → notification-service , the asynchronous hop over Kafka. It wasn't there. The naive conclusion (and why it's wrong) The tempting read is immediate and obvious: the async hop is broken. The event isn't getting across. Go debug the consumer. So I checked. And the consumer was completely fine. notification