标签：#architecture

The Problem We Were Actually Solving During load testing at 50k concurrent hunters hitting the hunt endpoints, p99 latencies stayed under 200ms. But at 270k concurrent users in production, the hunt page suddenly took 1.8 seconds to load, triggering cascading 502s from our CDN. The error surfaced in Datadog as hunt_page_render_time_bucket{le=2.0} = 42% while le=0.5 dropped to 18%. The fingerprints were identical across three regions: high latency correlated exactly with Redis cache miss rate spiking from 12% to 48% during the hunt start window. Our cache-aside pattern with a 30-second TTL was amplifying miss storms. We discovered that the treasure hunt start time was synchronised by marketing campaigns. When the clock struck 10:00:00 UTC, 270k users hit the endpoint within 30 seconds. Each request would check the cache (miss), fetch from PostgreSQL, render the page, and write the cache entry. But PostgreSQL couldnt keep up with 9k queries per second during that window, causing query queueing and connection exhaustion. The Redis layer, designed for 150k ops/sec, was not the bottleneck. The database was. What We Tried First (And Why It Failed) Our first attempt was to increase Redis TTL from 30 seconds to 5 minutes. This reduced cache misses from 48% to 24%, and p99 latency improved to 650ms. But at 320k concurrent users, the latency still spiked to 1.4s because the underlying database queries were still hitting the same table with the same indexes. The Redis layer was masking symptoms, not solving the root cause. Next, we tried database read replicas. We spun up three read replicas and routed hunt queries to them using a weighted service mesh. This worked for a few minutes, but then we hit replication lag. The replicas fell 800ms behind primary, causing hunt pages to display stale treasure locations. Our operators started getting customer complaints about seeing the wrong treasure coordinates. We rolled back within 15 minutes. We even tried increasing PostgreSQL share

The Problem We Were Actually Solving Treasure hunts in Hytale arent just about generating loot. Theyre about generating simultaneous loot across thousands of players while keeping the world state consistent. We started with the assumption that events are stateless notifications: a hunt starts, we fire an event, clients react. That model worked fine when we had 200 concurrent players. At 2,000 players, the event bus turned into a 40 MB/s firehose of JSON blobs. Each loot drop required serializing the entire chunk state—blocks, entities, metadata—so clients could render the drop in real time. The JVMs G1GC couldnt handle the allocation rate. Every 47 minutes, a GC cycle would pause for 4.2 seconds, the chunk cache would fragment, and the server would hard crash with an OutOfMemoryError in net.minecraft.server.MinecraftServer#processQueue. The real problem wasnt the hunt logic. It was the architectural laziness of treating events as a catch-all glue layer instead of a boundary layer with explicit interfaces. What We Tried First (And Why It Failed) We tried Kafka as the event bus. The plan was to shard hunts by region and stream loot drops as compacted topics. The first run worked for about 6 hours before the compacted topics started to bloat. Each hunt was generating 700 KB of serialized chunk state per drop. At 30 drops per hunt per minute, thats 21 MB per hunt per minute. With 400 active hunts, the brokers couldnt keep up. The lag grew to 12 seconds, clients started rubber-banding, and we got a flood of Discord reports: You sank my boat! The event stream was now the bottleneck, not the event source. Next, we tried Redis Streams with a Lua script to aggregate loot drops per chunk. Within 30 minutes, we hit the 4 GB maxmemory limit because Lua scripts were stacking dropped items in memory while waiting for the next batch. The script was elegant—O(1) per drop—but the memory footprint made it unusable in production. Finally, we tried a sidecar service: a small Go process

Why Does Using an ORM Decrease Database Performance? While trying to optimize the shipping module in a production ERP, I noticed that database queries were incredibly slow. At first, I examined the SQL queries and checked the indexes. However, I couldn't get the performance boost I expected. The problem lay in the Object-Relational Mapper (ORM) library, which was the cornerstone of our application. ORMs make things easier for software developers by providing an abstract layer for database operations, but this convenience often comes at a performance cost. In this post, I will explain why using an ORM decreases database performance, using concrete examples from my own field experience. The core promise of ORMs is to keep developers away from SQL and allow them to interact with databases in a way that is more aligned with the object-oriented paradigm. This is a huge advantage, especially in small and medium-sized projects or rapid prototyping processes. However, when things get complex and performance becomes critical, the efficiency of the queries generated by ORMs starts to be questioned. In many cases I have encountered, especially in enterprise software development processes, the default behaviors of ORMs created an unexpected load on database servers. Query Inefficiencies Generated by ORMs ORMs usually manage database relationships using mechanisms like "eager loading" or "lazy loading". Depending on the developer's preference, these mechanisms either fetch all related data at once (eager loading) or fetch it in pieces as needed (lazy loading). However, ORMs may not always perform these loads in the most optimized way. For example, while only a few fields like ID and name are sufficient in a list view, the ORM might query the entire table or all related tables. This situation causes unnecessary data transfer and unnecessarily overloads the database server. To give an example, on the order list screen of an e-commerce site, we needed to display the customer inform

Microsoft added sandboxed code interpreters to Azure Logic Apps, enabling agents within integration workflows to generate and execute Python, JavaScript, C#, and PowerShell in Hyper-V isolated sessions. Architects get full control over model selection per workflow. The capability positions Logic Apps as an agent platform for integration alongside Foundry and Copilot Studio. By Steef-Jan Wiggers

InfoQ expands its online certification portfolio with new AI Engineering and Organizational Architecture cohorts, giving senior practitioners a confidential peer group to pressure-test production AI, platform, team design, and architecture decisions. By Artenisa Chatziou

The call to action “to make 2026 the year of Java in the terminal” was quickly responded to by the launch of TamboUI. Inspired by Ratatui, the library used in Claude CLI, it promises support ranging from low-level terminal drawing to high-level APIs such as components and event handling. Currently at version 0.3.0, it has already been adopted by major projects such as Maven and Spring. By Olimpiu Pop

How the GitHub Issues team used client-side caching, smart prefetching, and service workers to make navigation feel instant. The post From latency to instant: Modernizing GitHub Issues navigation performance appeared first on The GitHub Blog .