标签：#backend

Being a Java Developer, I always thought about the programming language i'm working in, if it's the right one for all along the career ahead. I went through some web-based studies and, completely satisfied with the information I got to know. So, the short answer to the prime question is: Yes, Java is absolutely relevant and, here's why:- Still a Top Language Java has been in the top 3 programming languages worldwide for 2+ decades. Historical Dominance: The Backbone of Enterprise Systems: Since its inception, Java’s mantra of "Write Once, Run Anywhere" (WORA) revolutionized software development. It quickly became the foundation for global financial systems, insurance platforms, healthcare infrastructure, and e-commerce giants. Unrivaled Stability: Indexes like TIOBE and GitHub Octoverse have consistently ranked Java among the top most used languages for over 20 years. Companies do not shift their backend infrastructure on a whim; billions of dollars of existing, mission-critical infrastructure rely on the Java Virtual Machine (JVM). Enterprise Backbone Banks, insurance, e-commerce, and global-scale companies still rely heavily on Java. 95% of enterprise systems use it in some form. Banking and Financial Services (FinTech): Transactional Integrity: Mega-banks require high concurrency and absolute compliance with ACID (Atomicity, Consistency, Isolation, Durability) properties. Java's robust memory management and strict type safety prevent multi-threading errors that could result in catastrophic financial discrepancies. Legacy Settlement Layers: Systems managing global wire transfers, electronic clearing houses (ACH), and high-frequency trading platforms were built on the Java Virtual Machine (JVM) over the last 30 years. Rewriting these multibillion-dollar codebases carries massive operational risk with zero business incentive. Insurance Platforms: Complex Risk Modeling: Insurance giants process enormous volumes of historical actuarial tables and continuous risk data.

This is Part 7 of my "From One User to One Million" series, where we'll build an understanding of System Design by following a simple application as it grows from a single user to millions. Instead of memorising technologies, we'll learn why they exist by solving real problems as they appear. Last time, we ended on a question that sounded simple but isn't. Aisha updated her profile picture. Her new photo is now saved in the database. But the cache is still holding onto the old one, completely unaware that anything changed. So every request for Aisha's profile gets served the old data. Confidently. Instantly. Incorrectly. How does a cache know when the data it's holding is no longer correct? Think about what we've actually built at this point. We have an application that responds fast, scales horizontally, and avoids hammering the database with repeated identical queries. From a performance standpoint, it looks great. But Aisha's friends are loading her profile and seeing a photo she replaced five minutes ago. The system isn't slow anymore. It's wrong. Speed and correctness are two different things. We optimized hard for one, and quietly broke the other. Engineers have a name for this problem: cache invalidation . It refers to the challenge of keeping the data in your cache consistent with the data in your database, as that underlying data changes over time. It turns out to be one of the genuinely hard problems in building software systems. Not hard in a complicated-algorithm way. Hard in the way that every solution has a catch, and the right answer always depends on what you're willing to accept. Let's think through it together. -- Section 1: When Cached Data Lies It's worth sitting with the problem a little longer before rushing to fix it, because the damage stale data can cause varies enormously depending on what's being cached. Consider a few examples. Your application caches the list of trending articles. An hour later, the list has changed. New articles have ri

Introduction In June 2026, the IETF published RFC 10008 - the first new general-purpose HTTP method since PATCH was introduced in 2010. The method is called QUERY . In simple terms: QUERY = Safety of GET + Body of POST You can now send complex search/filter queries in the request body, while the server knows the operation is safe and idempotent . This means caching, automatic retries, and CDNs can all work properly. This single change can finally end the long-standing practice of using POST for search. The Problem We Had 1. Limitations of GET With GET, query parameters go in the URL: GET /products?category=electronics&price_min=1000&price_max=50000&brand=samsung,apple&sort=-rating&page=1&limit=20 When filters become complex (JSON filters, nested conditions, many tags), the URL easily exceeds 8,000 characters. Many servers, proxies, and browsers struggle with this. URLs also get logged, bookmarked, and shared — which is often undesirable. 2. Problems with POST So many developers started using POST for search: POST /products/search Content-Type: application/json { "filters": { "category": "electronics", "price": { "min": 1000, "max": 50000 }, "brands": ["samsung", "apple"] }, "sort": "-rating", "page": 1, "limit": 20 } But POST is not safe and not idempotent . That means: Caches and CDNs cannot safely cache the response Automatic retries after network failures are risky The server may treat it as a state-changing operation We have been pretending that a read operation is a write operation for years. What is the QUERY Method? According to RFC 10008: A QUERY requests that the request target process the enclosed content in a safe and idempotent manner and then respond with the result of that processing. In plain English: You send the query in the request body (like POST) The server processes it and returns the result It does not change any server state (like GET) Sending the same request multiple times produces the same result (idempotent) Comparison Table Property GET Q

I've been going back and forth on whether to share this — it's a pretty niche idea, and I wasn't sure if it's clever or just weird. But here's the technical side of it, which I figure this crowd might actually appreciate regardless. What I built: claimo.me — you claim a short code (2-4 letters, or a custom name) for a one-time fee, no subscription, permanently yours. Each code is configurable as a redirect link, a QR code, or a small profile card. There's also a "Claimo Map" — every possible code is a clickable pixel you can browse, inspired by the old Million Dollar Homepage. The part I actually want to talk about here: it's zero-dependency. No Express, no ORM, no build step — just Node.js 22+'s built-in http module and the new built-in node:sqlite. I wanted to see how far "just the standard library" actually gets you for something real — payments (Stripe), admin moderation, rate limiting, a live interactive map UI, the works. Some things that surprised me building it this way: node:sqlite's DatabaseSync is genuinely pleasant to use, but it's missing conveniences like better-sqlite3's .transaction() helper — I ended up writing a small manual BEGIN/COMMIT/ROLLBACK wrapper. Routing without a framework is maybe 40 lines of code and I stopped missing Express within a day. The real cost isn't runtime performance, it's losing the ecosystem — anything I'd normally npm install for free (input validation, rate limiting, even basic templating) I had to hand-roll. Some of that was genuinely good for me, some of it I'd reconsider on a bigger project. Business side, since half of you will ask: it's a real registered business, payments go through Stripe only (I never touch card data), no crypto, nothing weird. The paid tiers fund keeping a free short-link tier alive too. Honestly — is the zero-dependency thing a genuinely good call for a real production app, or am I just going to regret it in a year? And separately: does "own a short code" as a product idea make any sense to you

I built a free tool that checks a supplier before you pay them. The part that took most of the work, and taught me most, was reading ten national company registers instead of relying on the EU's own VIES service. This is what I found out, mostly so the next person doesn't have to. The problem with "the VAT number is valid" VIES — the European Commission's VAT Information Exchange System — answers one question: is this VAT number currently registered. That sounds like the question you want answered. It isn't. A company that has gone into liquidation keeps a cleanly resolving VAT number in VIES. So does one that has been struck off the register. Deregistration and insolvency are run by different authorities on different timetables, and the gap between "this company has stopped being a going concern" and "the VAT number stops validating" can be months. So you can check a supplier, get a green tick, and be looking at an insolvency estate. The national registers know. VIES doesn't ask them. Ten registers, and what each actually gives you I found free, public, machine-readable-enough sources for ten countries: Bulgaria, Czechia, Estonia, Finland, France, Greece, Latvia, Poland, Romania and Slovenia. They are not equivalent, and this is the thing I'd have liked written down somewhere before I started: Six of them report company *state * — inactive, in liquidation, bankrupt, insolvent, terminated, ceased, struck off: Romania, Estonia, France, Greece, Bulgaria, Latvia. This is the valuable one. Three report whether the company is actually VAT-active — Poland, Romania, Slovenia. That matters more than it sounds, because VIES does not distinguish "this is a real company that isn't VAT-registered" from "this number belongs to nobody". The rest give you a name and not much more. Czechia, for instance, is in the ten but in neither of the other two groups. It confirms a name. That's it. Worth knowing before you build a feature around it. Poland is the interesting one Poland is the

In part 1, we built the shared mental model: call stack, microtask queue, macrotask queue, and the rule that microtasks fully drain before the next macrotask runs. That model is spec-level JavaScript behavior — but it's not the whole story once you actually run code. The event loop isn't part of the JS language spec. It's part of the host environment — the browser or Node — and each one implements it differently around that shared core. This is the post most "event loop" explainers skip, because it means going past the diagram and into how each runtime is actually built. The browser: event loop meets rendering In a browser, the event loop isn't just juggling callbacks — it's also responsible for keeping the page visually responsive. That means rendering has to get a turn too, and the browser has to decide when . Here's the roughly accurate sequence per loop iteration: Execute one macrotask (a click handler, a setTimeout callback, a network event, whatever's next in the queue) Drain the entire microtask queue Maybe render a frame — the browser doesn't render after every single task; it tries to hit ~60fps and will batch work between paints Go back to step 1 The "maybe render" part is where two APIs come in that don't exist in Node at all: requestAnimationFrame(callback) — schedules a callback to run right before the next repaint. It's not a macrotask or microtask in the queue sense — it's tied directly to the rendering pipeline. Use it for anything visual (animations, DOM measurements) instead of setTimeout , because it's synced to when the browser is actually about to paint, not an arbitrary delay. requestIdleCallback(callback) — schedules a callback to run when the browser is idle, after layout and paint, with a deadline. Meant for low-priority work you don't want competing with rendering — analytics, prefetching, non-urgent DOM updates. Here's the key interaction that's easy to miss: microtasks can starve rendering. If a promise chain keeps queueing more microtask

Use metrics APIs for cron-job, API-failure, and business-event charts, then add a separate heartbeat monitor for jobs that never start. That is the smallest stack I would put on call for a small SaaS. A metrics dashboard can show success and failure counts, duration, backlog size, and error-rate trends; it cannot prove that a scheduler actually invoked a job. Healthchecks-style monitoring closes that specific gap. It still isn't full monitoring coverage, and I wouldn't describe it that way in an SLO review. The distinction matters because a failed run and a missing run leave different evidence. An API error usually increments something. A business event can be counted. A cron job that never fires may produce nothing at all — no duration, no failure, no final log line. No signal. How should a backend metrics dashboard combine cron jobs, API failures, and healthchecks? Start with the questions an operator must answer, not with a vendor menu. For cron jobs, I want a success count, a failure count, duration, and any queue backlog that can delay completion. For API failures, I want error counts and an error-rate trend beside request volume, because a raw count without a denominator can make ordinary traffic growth look like a regression. For business events, I want domain verbs: invoices issued, imports completed, or messages accepted. Those widgets belong on one dashboard because they describe the same service from different angles. Heartbeat monitoring is a separate control. A job reports a start or completion ping to Healthchecks, Cronitor, or an equivalent tool; if the expected ping doesn't arrive within its schedule and grace period, that system owns the missing-run signal. Keep that alert outside the metrics query path. Otherwise the component that failed to emit data is also the component being asked to notice its own silence. Silence counts. I've learned to write the failure matrix before drawing the dashboard. In one incident, a call returned 200, but the side e

Use a durable idempotency receipt when feature flag retries can reach a rollout toggle endpoint, otherwise reach for a read-only flag evaluation that cannot create duplicate writes. Short answer: the backend must bind one caller-generated key to one operation and commit the receipt beside the state change; a retry should recover that recorded result, not perform the write again. The flag is not the transaction. Record the invariant at the write boundary My architecture decision is to enforce idempotency inside the backend that owns the mutable state. The caller creates an operation key before its first attempt, sends the same key and operation on every retry, and never manufactures a fresh key inside the retry loop. The backend binds that key to a stable digest of the requested change. If the key and digest have already been committed, it returns the stored result. If the key exists with a different digest, it rejects the integration error as a conflict. The state mutation and receipt belong in one transaction, because two separate commits create an interval in which the state says “done” while the receipt still says nothing. I write the invariant this way: one idempotency key identifies one logical operation within a documented scope; one committed operation has one durable result. The defensible claim is effectively-once mutation within that scope, not exactly-once delivery. Clients, queues, proxies, and deployment controllers can all repeat an attempt, so delivery count isn't a useful correctness boundary. There are three failure boundaries I test. A response can disappear after commit, two workers can race on the same key, and the flag decision can change between attempts. The first requires replaying the stored result. The second requires a uniqueness constraint rather than a check-then-insert sequence. The third requires persisting the evaluated decision with the operation; reevaluating a flag during recovery can turn one logical request into two different his

Modern backend applications handle thousands or even millions of requests every second. Users perform actions simultaneously: buying products, transferring money, updating profiles, sending messages, and more. But what happens when two requests try to modify the same data at the same time? This is where race conditions appear — one of the most subtle and dangerous problems in backend development. A race condition can cause incorrect data, security issues, financial losses, and unpredictable application behavior. Understanding how race conditions happen and how to prevent them is an essential skill for backend developers. What Is a Race Condition? A race condition occurs when multiple processes or requests access and modify shared data at the same time, and the final result depends on the order in which those operations execute. The problem is that the developer expects operations to happen in a specific sequence, but the computer executes them based on timing, network delays, database speed, and system load. Simple Example: Bank Account Withdrawal Imagine a user has: Account Balance: $100 Two withdrawal requests arrive at the same time: Request A: Withdraw $80 Request B: Withdraw $50 The backend checks the balance: Request A: Balance >= 80? Yes Request B: Balance >= 50? Yes Both requests continue because they saw the original balance of $100. The system processes: $100 - $80 = $20 $100 - $50 = $50 The final balance might become: $50 instead of: -$30 (which should have been rejected) The application has allowed money to be withdrawn that does not exist. This is a race condition. How Race Conditions Happen in Express.js Express.js applications are often built around asynchronous operations: Database queries API calls File operations Background jobs Message queues Consider this simple inventory system: app . post ( " /purchase " , async ( req , res ) => { const product = await Product . findById ( req . body . productId ); if ( product . stock > 0 ) { product . stock -

After working on enterprise applications and distributed microservices, I have realized that the biggest challenges rarely come from writing business logic. They come from handling production traffic, failures, concurrency, and unexpected edge cases. Here are seven lessons that every Spring Boot developer should know before calling themselves a senior engineer. 1. Never Assume an API Will Be Called Only Once One of the most common mistakes is assuming a client sends exactly one request. In reality: Users refresh the page. Mobile apps retry automatically. API gateways retry requests. Kafka consumers may reprocess events. Network failures cause duplicate submissions. If your endpoint creates an order, payment, or booking every time it receives a request, duplicates are almost guaranteed. Better Approach Design APIs to be idempotent . For example: Use an Idempotency-Key. Store processed request IDs. Ignore duplicate requests safely. Production systems should always expect duplicate requests. 2. Database Transactions Are Not Enough Many developers believe this solves everything: @Transactional public void createOrder () { ... } It doesn't. A transaction protects changes inside a single database . It does not protect: Kafka publishing Email sending External REST APIs Redis updates File uploads If your database commits successfully but Kafka publishing fails, your system is already inconsistent. Better Approach Use patterns such as: Transactional Outbox Saga Pattern Event-driven architecture Retry with dead-letter queues 3. Don't Trust External APIs Every external service will eventually fail. Your payment provider. Your authentication service. Your notification service. Even your own internal microservices. Never assume another service is always available. Add Protection Timeouts Retries Circuit Breakers Fallback logic Monitoring Failing fast is usually better than waiting forever. 4. Logging Is More Valuable Than You Think When production goes down, nobody asks: "Was th

It was 2 AM, and the on-call chat was on fire again: the order service was healthy on every dashboard, but throughput had flatlined at ~800 req/s while P99 climbed past 4 seconds. The usual suspect? A thread pool sized by guesswork during a late-night deploy, six months earlier. We'd hand-tuned maxThreads to "something that felt right," and it wasn't right anymore. That's the moment I started appreciating a different default: in Solon, all of those knobs ship as 0 — meaning auto , derived from your machine's actual CPU cores at runtime. You can go months without thinking about a single thread-pool property. This post walks through the five knobs that exist, how the auto-tuning math works, and the three failure modes that tell you it's time to touch them. The five knobs under the hood Solon exposes these on app.yml (all values are the documented defaults): # Minimum threads for the http server (0 = auto; also accepts fixed values like 2, or core multiples like x2) server.http.coreThreads : 0 # Maximum threads for the http server (0 = auto; also accepts fixed values like 32, or core multiples like x32) server.http.maxThreads : 0 # Idle thread timeout in ms (0 = auto) # supported since v1.10.13 server.http.idleTimeout : 0 # Is this an IO-bound service? (default true) # supported since v1.12.2 server.http.ioBound : true # Enable the virtual thread pool (default false) # supported since v2.7.3 solon.threads.virtual.enabled : false Notice what's missing: no hard-coded defaults for coreThreads or maxThreads . 0 means "figure it out from the hardware." That single decision removes a whole class of "copy-pasted tuning values" problems — the ones that were right for someone else's 32-core box and wrong for your 2-core container. CPU-bound or IO-bound: the one question that matters The auto-tuner only needs you to answer one question: is your workload CPU-bound or IO-bound? CPU-bound : the work happens entirely in CPU and memory — think a "hello world" handler that returns a s

Introduction Imagine you arrive at a hotel. At the reception, you show your ID and prove who you are. The receptionist then gives you a room key card. You don't need to show your ID every time you enter your room. Instead, you simply present the key card. The hotel doesn't need to ask your name again because the card itself proves that you already authenticated. JWT (JSON Web Token) works exactly like that. Username and password = Your ID JWT = Hotel key card Server = Receptionist What is JWT? JWT stands for JSON Web Token . It is a compact string that proves a user has already logged in successfully. Instead of storing login sessions on the server, the server gives the client a signed token. The client sends this token with every request. Example: Authorization: Bearer eyJhbGciOiJIUzI1NiIs... The server verifies the token and allows access. Why Do We Need JWT? Without JWT, every request would require sending the username and password repeatedly. Browser | Username Password | Server That would be inefficient and insecure. Instead: Login once ↓ Receive JWT ↓ Reuse JWT for every request Stateless Authentication JWT enables stateless authentication . Stateful Authentication Server | |-- Session #12345 |-- Session #91821 |-- Session #44211 The server stores every user's session. Stateless Authentication (JWT) Server (No session storage) ↓ Only verifies token signature The server doesn't remember users. The token remembers. JWT Structure A JWT consists of three parts separated by periods. Header.Payload.Signature Example eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9 . eyJzdWIiOiIxMjMiLCJuYW1lIjoiRXZhbnMiLCJyb2xlIjoiYWRtaW4ifQ . K6L6GQX.... Think of it like Envelope Letter Wax Seal Part 1 — Header Example { "alg" : "HS256" , "typ" : "JWT" } The header tells us: Which algorithm signed the token. What type of token it is. Fields: alg → Signing algorithm typ → JWT Common algorithms: HS256 RS256 ES256 Part 2 — Payload The payload contains claims . Example: { "user_id" : 42 , "name" :

In Parts 1-4, we built a transactional key-value store. It has WAL for durability, memtables and SSTables for storage, compaction to control file growth, and transactions for atomic multi-key writes. Now we move to the query layer where users can actually fire SQL queries like: CREATE TABLE payments ( amount INT , id STRING , status STRING , captured BOOL , PRIMARY KEY ( id )) INSERT INTO payments VALUES ( 500 , payment_1 , pending , 1 ) SELECT * FROM payments WHERE id = payment_1 In this blog and the next one we answer: How do we translate SQL strings into operations our key-value store already understands? This post focuses on the first half of that bridge, which is parsing SQL into structured commands. In the next post, we will take those commands and turn CREATE TABLE and INSERT into bytes on disk. The Core Idea: SQL Becomes Structured Data The storage engine does not understand SQL. It understands keys, values, WAL entries, memtables, SSTables, and transactions. So the SQL layer has two jobs: Parse a human-readable SQL string into a structured object. Translate that structured object into key-value operations. For example: CREATE TABLE payments (...) -> CreateTable{TableName: "payments", ColumnDetails: ...} INSERT INTO payments VALUES (...) -> InsertIntoTable{TableName: "payments", ColumnValues: ...} SELECT * FROM payments WHERE id = payment_1 -> SelectFromTable{TableName: "payments", QueryConditions: ...} Once we have these structs, the rest of the database can stop dealing with raw strings. Why Not Parse SQL Directly in the DB Layer? Imagine if db.CreateTable() directly walked through the SQL string and also updated storage. That would mix two very different responsibilities: parsing grammar, executing database operations. Keeping them separate makes the system easier to reason about. The parser validates syntax and builds an AST. The DB layer receives that AST and decides what to store. An AST, or Abstract Syntax Tree, is just a structured representation of

Introduction Managing content efficiently becomes more difficult as Drupal websites grow. Traditional content management structures information in discrete entities but often does not expose meaningful relationships between them. A knowledge graph solves this, connecting content like nodes, taxonomy terms, users and media into an intelligent network. This facilitates semantic search, personalized recommendations and AI-driven applications to enhance how users discover and engage with content. Knowledge Graphs Explained A knowledge graph is a structured representation of linked information. It links related entities by relationships instead of storing content as distinct records. For instance, a Drupal article on cybersecurity might be linked to its author, relevant taxonomy terms, media files, and related articles. These links allow applications to understand not just the content itself, but how it relates to other information across the website. Drupal content mapping Drupal already stores content in a structured way, so it is a good fit for knowledge graphs. Typical entities are: Nodes Taxonomy terms Individuals Media files Custom-entities These entities become nodes in the graph and relationships such as created by, belongs to or references become the edges between them. This provides a richer picture of the content of the website. Selecting a Graph Database Drupal can link to graph databases like Neo4j or Amazon Neptune, where these relationships can be stored and queried efficiently. The typical workflow would be to extract Drupal content, transform it into graph nodes and relationships and synchronize updates through APIs or event-driven processes whenever content is created or modified. Improving Search and Recommendations Knowledge graphs greatly enhance the discovery of content by understanding relationships rather than relying just on keywords. For example, a user looking for Drupal security might also find relevant articles on authentication, access contr

Every portfolio site has the same problem: it's static. A grid of projects, a bio, a contact form — nothing on the page ever changes, which means nothing on the page proves you can work with live data. Recruiters and reviewers skim past it because there's nothing to skim. The fastest fix isn't building your own API — it's embedding someone else's, and picking one that's actually interesting to look at. Here's how to drop a live, auto-updating tech news feed into any site with a single script tag, using NewTqnia , a bilingual (English/Arabic) tech newsroom with a free embeddable widget. Why this is a good portfolio move, not just decoration A static "About Me" page tells someone you can write HTML. A page with a live-updating feed tells them you can integrate a third-party service, handle async content, and think about internationalization (this one supports English and Arabic out of the box) — all real, hireable skills, for the cost of one script tag. Step 1: Build your embed Go to newtqnia.com/en/widget . It's a live configurator, not a docs page — every option you touch updates a preview instantly: Content: custom heading, number of articles, language (English or Arabic), category filter (Artificial Intelligence, Robotics, Space, Health, and others), and ordering (latest first or most popular) Appearance: card / list / compact layout, horizontal or vertical orientation, light / dark / automatic theme, accent color, and toggles for images and summaries Pick settings that match your site — a compact, dark-themed, "Artificial Intelligence"-filtered list looks noticeably more intentional than the default. Step 2: Copy the generated snippet Once you're happy with the preview, the page generates a ready-to-paste embed code block for you — copy it as-is. It'll look roughly like a single <script> tag referencing your chosen configuration, something like: <script src= "https://newtqnia.com/embed/widget.js" data-lang= "en" data-category= "artificial-intelligence" data-count

From the outside, social features seem like something straightforward: follow a user, like a post, see a feed, but when you attempt to implement them at any real scale you find that every single one is something you've got to be aware of, a pattern, a well-documented query pattern with all its failure modes. Here's a step-by-step look at four of them: adjacency list, fan-out feed, mutual-friends join, and some graph traversal that SQL wasn't really designed for. The adjacency list and why "who do I follow" isn't free A follow relationship is usually modeled as a plain adjacency table: follows(follower_id, followee_id, created_at) . That's the whole schema, and it's deceptively adequate for a long time. The first place it breaks is when you need "who do I follow that also follows them", mutual connections, because that's a self-join against the same table: SELECT f2 . followee_id FROM follows f1 JOIN follows f2 ON f1 . followee_id = f2 . follower_id WHERE f1 . follower_id = : user_id AND f2 . followee_id != : user_id ; This query is ok if the fan-out is low. It feels like it's no longer fine when a few accounts have a few hundred thousand joins, because join now needs to walk a correspondingly large intermediate result set per request. The typical solution isn't a better query, it's a better index and a cap. composite indexes on (follower_id, followee_id) and (followee_id, follower_id) so both directions of the join hit an index-only scan, plus a LIMIT applied early so the planner doesn't materialize more rows than the response will ever use. The fan-out feed problem The “social systems” hard problem is the timeline: display to me my feed of what people I follow posted recently, in order. There are two ways to construct it and they are both right - anything else and outages happen. Fan-out on write means that when a user posts, you insert a row into every follower's feed table immediately. Reads are then a trivial SELECT * FROM feed WHERE user_id = :id ORDER BY creat

Three bugs we found and fixed in our own pipeline this week Journeymen grades developer work against GitHub's server-side history. That only means something if the grading pipeline itself is reliable — so here's the honest engineering update, not the highlight reel. 1. Silent progress loss on connect-repo analysis runs A connect-repo analysis run could sit in processing status with no visibility into what stage it was actually at, or whether it had stalled. From a dev's dashboard, a slow run and a stuck run looked identical. We added explicit progress-stage tracking so a stuck run is visibly stuck, not silently pending. 2. A background worker timing out without a clear signal The Lambda-based worker handling asynchronous analysis jobs was hitting its timeout under certain repo sizes, and the failure mode wasn't obvious from the outside — a run would just never complete. We root-caused the timeout and fixed the underlying slow path. 3. Dead-letter queue with no observability Jobs that failed enough times to land in the SQS dead-letter queue were, until this week, invisible — no alerting, no in-product surfacing. We wired up observability so a DLQ arrival is now a visible signal instead of a silent dead end. Why post about our own bugs The entire pitch of Journeymen is "don't trust the self-reported version, trust the verified one." That standard has to apply to us too. All three issues: found, fixed, and shipped this week. journeymen.in

I've spent years working with backend systems. APIs, databases, caching, integrations, performance, authentication, cloud infrastructure — these are the kinds of problems that become familiar after you've been building software for a while. Recently, I've been spending more time learning another side of software engineering: LLMs, RAG, AI Agents, tool calling, and MCP. At first, it felt like entering a completely different world. New terminology. New frameworks. New architectural patterns. And honestly, it made me feel like a beginner again. But the deeper I went, the more I noticed something interesting: AI engineering has a lot more backend engineering in it than I initially expected. The Traditional Backend Mental Model A simplified backend architecture might look like: Client ↓ API ↓ Business Logic ↓ Database ↓ Cache / External Services Of course, production systems have much more around this: Authentication Authorization Logging Monitoring Queues Caching Load balancing Rate limiting Distributed systems Cloud infrastructure But the general flow is deterministic. The application receives a request. Our code decides what happens. The application returns a response. Then we add AI. Adding an LLM Looks Easy The first architecture is surprisingly simple: User ↓ API ↓ LLM ↓ Response Send a prompt. Get a response. Done. For a prototype, this can be enough. But production applications rarely stay this simple. Suppose we're building an assistant that answers questions using internal company documents. Now we need retrieval. Enter RAG A simplified Retrieval-Augmented Generation pipeline might look like: Documents ↓ Chunking ↓ Embeddings ↓ Vector Database Then, when the user asks a question: User Question ↓ Embedding ↓ Vector Search ↓ Relevant Documents ↓ Context ↓ LLM ↓ Answer Conceptually, this is easy to understand. But implementing it properly raises a lot of questions. How should we chunk documents? A fixed number of characters? Tokens? Paragraphs? Sections? Semantic

Manticore Search 28.6.6 has been released. The headline additions are UUID document IDs for real-time tables and ordered, limited GROUP_CONCAT() for grouped queries. The release also includes 16 fixes for backups, replication, query processing, SQL compatibility, and secondary indexes. This post covers everything shipped from 28.4.5 through 28.6.6 . Upgrade notes There are no new mandatory data migrations in this release. UUID IDs are an opt-in table definition: existing numeric-ID tables keep working as they are. If you want UUID identifiers, create a real-time table with id uuid ; ALTER TABLE cannot convert an existing table between numeric and UUID IDs. Two fixes are particularly useful for production installations. Successful backups now always unfreeze real-time tables when they finish (previously in rare cases they didn't), rather than leaving writes blocked. And authenticated replication can again add an existing populated RT table with ALTER CLUSTER ... ADD . UUID document IDs for real-time tables Applications often already have UUID identifiers from the system of record. Until now, using them with Manticore Search meant maintaining a separate numeric ID mapping. Real-time tables can now use UUID document IDs directly: CREATE TABLE products_uuid ( id uuid , title text , price int ); Manticore accepts an explicit UUID string, or generates one when id is omitted from an insert or replace. UUID equality and IN filters work in queries, and UUID IDs can be used with REPLACE , UPDATE , and DELETE . This is currently a real-time-table capability, including columnar and replicated RT tables. Plain, percolate, and sharded tables continue to use their existing ID models. Ordered and limited GROUP_CONCAT() Grouped results often need a compact preview of the most relevant values in each group. GROUP_CONCAT() can now sort values and retain only the requested number of them in explicit SQL GROUP BY queries: SELECT category , GROUP_CONCAT ( title ORDER BY price DESC SEPARA

"We use Redis Cluster" can mean two very different things: Our dataset is distributed across Redis nodes. Every individual data structure is distributed across Redis nodes. The first can be true while the second is false. That distinction matters for leaderboards. In Podium , each leaderboard uses several Redis keys and atomic Lua scripts. Redis Cluster helps us scale a large fleet of independent leaderboards, but it cannot split one giant sorted set across primaries. We are sharing this architecture because "Redis Cluster scales horizontally" is true only after you define what the system actually shards. TeneficGames / podium High-performance, Redis-backed leaderboards for games and competitive applications. Podium High-performance, Redis-backed leaderboards for games and competitive applications. Podium provides ready-to-run HTTP and gRPC APIs for scores, ranks, seasons, and player-relative views. It is designed for backend teams operating large fleets of independent leaderboards without provisioning each leaderboard in advance. Fair, deterministic ordering when scores are equal. Single and bulk score updates, including multi-leaderboard fan-out. Standalone Redis and real Redis Cluster integration coverage. Deploy one multi-architecture OCI image with Docker, containerd, Kubernetes or another OCI-compatible runtime. Quickstart · Performance · API · Documentation · Helm chart · Docker Hub · GHCR Quickstart Start Redis 8.2 and the latest stable Podium image: docker network create podium docker run --detach --name podium-redis --network podium redis:8.2-alpine docker run --detach --rm --name podium \ --network podium \ --publish 8880:8880 \ --publish 8881:8881 \ --env PODIUM_REDIS_HOST=podium-redis \ --env PODIUM_REDIS_PORT=6379 \ trungdlp/podium:latest start Verify the service: curl http://localhost:8880/healthcheck WORKING Submit two equal scores: curl --request … View on GitHub Here is how the design works, why hash tags are necessary, and where the scaling bounda