标签：#data

Introduction Fixing N+1 queries with select_related / prefetch_related or selectinload (see the previous post ) gets you down to a small, sane number of queries per request. The next bottleneck is what each query costs once the table has millions of rows — and that is almost always about indexing. An index turns "scan every row" into "look it up directly." Skip it, and a query that's instant in development takes seconds once real data volume shows up in production. How Indexes Work: The B-Tree Intuition Without an index, a WHERE clause forces a sequential scan : the database reads every row and checks the condition — O(n) , cost grows linearly with table size. An index is a separate, sorted structure (almost always a B-tree ) mapping column values to row locations. Because it's sorted and balanced, finding a value is a tree walk: O(log n) . On a 10-million-row table, that's the difference between reading 10 million rows and roughly 23 tree nodes. This isn't free: Writes get slower — every INSERT / UPDATE / DELETE on an indexed column also updates the index. Storage grows — each index is a sorted copy of (part of) the data. An index trades write cost and storage for read speed. Indexing a column you rarely filter or sort on is pure cost, no benefit. Reading Query Plans: EXPLAIN ANALYZE Postgres' EXPLAIN ANALYZE shows what the planner actually did, not an estimate. Before an index , filtering orders by customer_id : EXPLAIN ANALYZE SELECT * FROM orders WHERE customer_id = 48291 ; Seq Scan on orders (cost=0.00..21453.00 rows=42 width=96) (actual time=0.021..118.442 rows=41 loops=1) Filter: (customer_id = 48291) Rows Removed by Filter: 1199959 Planning Time: 0.112 ms Execution Time: 118.471 ms Seq Scan means Postgres read all ~1.2 million rows and discarded all but 41. actual time is real elapsed time — 118ms for one lookup. After CREATE INDEX idx_orders_customer_id ON orders (customer_id); : Index Scan using idx_orders_customer_id on orders (cost=0.42..8.53 rows=42 wid

Introduction Fixing N+1 queries (see the previous post ) gets your Hibernate app down to a handful of queries per request. The next bottleneck is what each of those queries costs once your tables have millions of rows — and that is almost always a question of indexing. An index turns "scan every row" into "look it up directly." Get the index wrong — or skip it — and a query that took 2ms in development takes 4 seconds in production once real data volume shows up. How Indexes Work: The B-Tree Intuition Without an index, a WHERE clause forces a sequential scan : the database reads every row and checks the condition. That's O(n) — cost grows linearly with table size. An index is a separate, sorted data structure (almost always a B-tree ) that maps column values to row locations. Because it's sorted and balanced, finding a value is a tree walk: O(log n) . On a 10-million-row table, that's the difference between reading 10 million rows and reading roughly 23 tree nodes. The cost is not free: Writes get slower. Every INSERT / UPDATE / DELETE on an indexed column must also update the index structure. Storage grows. Each index is a copy of (part of) the data, sorted differently. An index is a trade: you pay on every write so that specific reads become fast. Indexing a column you rarely filter or sort on is pure cost with no benefit. Reading Query Plans: EXPLAIN ANALYZE Postgres' EXPLAIN ANALYZE shows what the planner actually did — not what you hope it did. Before an index , filtering orders by customer_id : EXPLAIN ANALYZE SELECT * FROM orders WHERE customer_id = 48291 ; Seq Scan on orders (cost=0.00..21453.00 rows=42 width=96) (actual time=0.021..118.442 rows=41 loops=1) Filter: (customer_id = 48291) Rows Removed by Filter: 1199959 Planning Time: 0.112 ms Execution Time: 118.471 ms Seq Scan means Postgres read all ~1.2 million rows and threw away all but 41 of them. actual time is the real elapsed time, not an estimate — 118ms for one lookup. After CREATE INDEX idx_orders

Cycle recently introduced a separate EU-based control plane, allowing European customers to keep platform management data and telemetry within Europe. The new offering is designed to improve compliance, operational isolation, and responsiveness for European organizations. By Renato Losio

Cloudflare details Town Lake, an internal unified data platform, and Skipper, an AI analytics agent unifying access to operational, billing, security, and business data. The platform processed ~91K billing queries, with billing forming majority usage. Built on a lakehouse architecture using Trino, Iceberg, R2, and DataHub, it enables governed cross-system analytics and natural language access. By Leela Kumili

Hardwood, the project Gunnar Morling kick-started handling of Parquet files in Java, reached version 1. Its multi-threaded approach and zero mandatory external dependencies promise a simpler, more efficient alternative to the Apache Parquet Java implementation. For now, the library supports just reading; writing support is expected in the upcoming versions. By Olimpiu Pop

Introduction As data volumes continue to grow, running aggregation queries directly on raw datasets becomes increasingly expensive. Business dashboards, analytics platforms, and reporting systems often execute the same calculations repeatedly—such as total sales, daily active users, page views, or revenue trends. While ClickHouse® is designed to process analytical workloads at remarkable speed, repeatedly scanning billions of records still consumes valuable CPU, memory, and storage resources. This is where AggregatingMergeTree proves its value. Rather than calculating aggregates every time a query is executed, AggregatingMergeTree stores intermediate aggregation states that are merged automatically in the background. This approach allows analytical queries to read compact, pre-aggregated datasets, resulting in dramatically faster response times and reduced infrastructure costs. In this guide, you'll learn how AggregatingMergeTree works, why aggregate states matter, how to build an automated aggregation pipeline using Materialized Views, and when this engine is the right choice for your ClickHouse® workloads. What is AggregatingMergeTree? AggregatingMergeTree is a specialized ClickHouse® table engine designed to store aggregate function states instead of raw records. Unlike the standard MergeTree engine, which stores every inserted row, AggregatingMergeTree keeps partially aggregated values that ClickHouse combines during background merge operations. This significantly reduces the amount of data that must be processed when generating analytical reports. Because much of the computational work happens during data ingestion, dashboards and reporting applications can retrieve summarized information much more efficiently. Typical scenarios include: Sales reporting Website traffic analytics Financial summaries IoT sensor monitoring Business KPI dashboards Application observability metrics Why Use AggregatingMergeTree? Imagine an online marketplace processing millions of tr

The speakers discuss Agent RFT, OpenAI’s platform for fine-tuning reasoning models via real-time tool interactions and custom reward signals. They explain how reinforcement learning solves complex credit assignment challenges within the context window. They share enterprise success stories, showing how Agent RFT eliminates long-tail token loops and drives extreme efficiency. By Wenjie Zi, Will Hang

If you ask any senior backend engineer or database administrator how to instantly make a slow, disk-bound application faster, their first answer is almost always: "Put it in memory." But why is memory so preferred, and how do modern architectures utilize RAM to achieve sub-millisecond latencies? We're seeing a massive revival of RAM disks and in-memory architectures. Let's explore why computer experts are increasingly treating RAM like a hard drive. 1. The Physics of Storage: Why RAM Wins To understand the shift towards in-memory architectures, we have to look at the numbers. Hard Disk Drives (HDDs): Mechanical spinning disks. Seek times are around 2-5 milliseconds . Solid State Drives (SSDs): Flash memory. Seek times are around 0.1 milliseconds (100 microseconds) . RAM (Random Access Memory): Volatile silicon. Access times are around 100 nanoseconds . RAM is roughly 1,000 times faster than an SSD and 10,000 to 50,000 times faster than an HDD. When you have a high-throughput system serving millions of requests per second, waiting for a disk to seek is an eternity. 2. In-Memory Databases: Redis and Memcached The most common implementation of this principle in modern backends is the In-Memory Database . How They Work Instead of writing every transaction to an SSD, systems like Redis and Memcached store the entire dataset directly in RAM. This bypasses the OS file system cache, disk I/O bottlenecks, and complex B-tree traversals required by traditional relational databases like PostgreSQL or MySQL. The Trade-off: Durability RAM is volatile. If the server loses power, all data is gone. So how do in-memory databases survive crashes? Snapshots (RDB in Redis): Periodically dumping the entire memory state to disk. Append-Only Files (AOF in Redis): Logging every write operation to a disk sequentially. Sequential writes to disk are significantly faster than random writes. This hybrid approach gives you the read/write speed of RAM with a "good enough" durability guarantee for

The Quest Begins (The "Why") I still remember the first time I faced a problem that asked for the sum of numbers in a sub‑array, over and over again, with updates sprinkled in between. It felt like I was stuck in a never‑ending loop of for i in range(l, r+1): total += arr[i] – O(n) per query, and with up to 10⁵ queries the solution timed out every single time. I was staring at the screen, thinking, “There has to be a smarter way to answer these range questions without scanning the whole array each time.” That moment was my dragon: a seemingly simple problem that kept biting me because I kept reaching for the brute‑force sword. I needed a data structure that could give me the answer in logarithmic time while still supporting point updates. Enter the segment tree – the tool that turned my O(n·q) nightmare into an O((n+q)·log n) victory. The Revelation (The Insight) So why does a segment tree work? Imagine you have an array and you want to know the sum of any interval [l, r] . If you could break that interval into a handful of pre‑computed chunks, you’d only need to add those chunk values together instead of touching every element. A segment tree is exactly that: a binary tree where each node stores the aggregate (sum, min, max, etc.) of a segment of the original array. The root covers the whole array [0, n‑1] . Its two children cover the left half and the right half, and this keeps splitting until the leaves represent single elements. The magic lies in two facts: Every node’s value is a function of its children. If you know the sum of the left child and the sum of the right child, the parent’s sum is just their addition. This means we can build the tree bottom‑up in O(n) time. Any interval can be represented as O(log n) disjoint nodes. When you walk down the tree to answer [l, r] , you either take a whole node (if its segment lies completely inside the query) or you recurse further. Because the tree’s height is log₂n, you’ll visit at most 2·log₂n nodes. Thus, building

Here is a question I could not answer from the headlines: which European countries are actually losing people the fastest, in absolute terms or per capita? Those are two different questions, and they give two different answers. So I pulled the open data and ran the numbers. The headline figure Across the 19 European countries in the 2024 dataset, 17 recorded a net loss of native-born residents . Only two were net positive. So the "brain drain" story is not a handful of outliers, it is the default state of the continent. But the interesting part is who tops the ranking, because it depends entirely on how you measure. Load the data yourself The dataset is public on GitHub (CC BY 4.0). Every number below is reproducible with a few lines of pandas. No download, no API key, it reads the raw CSV straight from the repo: import pandas as pd url = ( " https://raw.githubusercontent.com/DatapulseResearch/ " " brain-drain-eu/main/data/net_migration_native_born_2024.csv " ) df = pd . read_csv ( url ) print ( df . shape ) # (19, 3) print ( df . columns . tolist ()) # ['country', 'net_migration', 'per_1000_residents'] # How many countries lost native-born residents? losers = ( df [ " net_migration " ] < 0 ). sum () print ( f " { losers } of { len ( df ) } countries had a net loss " ) # 17 of 19 net_migration is the raw count for 2024 (negative means a net loss of native-born residents). per_1000_residents is the same flow normalized by population size. The absolute ranking: Germany runs away with it Sort by the raw count and one country dominates: worst_absolute = df . sort_values ( " net_migration " ). head ( 5 ) print ( worst_absolute [[ " country " , " net_migration " ]]) country net _ migration 0 Germany - 91067 ... Germany loses -91,067 native-born residents, far more than anyone else in absolute terms. If you stop reading here, the story writes itself: "Germany, Europe's biggest brain drain." Plenty of coverage did exactly that. The counterintuitive finding: the ranking inve

Author: Tobie Morgan Hitchcock Today we're launching Scale, a new tier of SurrealDB Cloud built for the workloads you can't afford to have go down. Our first tier, Start, was designed for building and shipping fast. Scale is designed for what happens next: production traffic, uptime commitments, and the kind of resilience your users never notice because there are always available nodes. It's the tier for teams running SurrealDB as the scalable context layer behind real applications and AI agents in production. What you get with Scale Scale is about one thing: keeping your database available and consistent under real-world conditions. Highly-available, fault-tolerant clusters. Scale runs your database as a multi-node cluster designed to survive node and infrastructure failures without dropping writes or losing consistency. A single point of failure is no longer a single point of downtime. Multiple availability-zone deployment. Your cluster is distributed across multiple availability zones, so even the loss of an entire zone doesn't take your database with it. Traffic keeps flowing while the cluster recovers in the background. Horizontal scale. As demand grows, Scale grows with it. Add capacity by scaling out across nodes rather than being capped by the size of a single machine. Start with three nodes, and keep adding to scale your application or agent's needs. See more information about SurrealDB Cloud Scale architecture here . Built on SurrealDS Scale is powered by SurrealDS , SurrealDB's distributed storage engine and the foundation that makes all of this possible. SurrealDS is a new generation distributed storage architecture, rethought from first principles. Instead of coupling storage to compute on a single box or to a proprietary cloud tier, SurrealDS embeds consensus directly in SurrealDB nodes and separates the two layers cleanly. Here's what that architecture gives you. Architecture overview Compute and storage separation. Scale compute for QPS and storage f

A top Democrat on the Senate's Intelligence Committee warned that the information accessed on a Homeland Security intelligence-sharing network may risk national security.

Google tries balancing AI data center emissions with clean energy efforts.

Apple chose Google Cloud to run Private Cloud Compute outside its own data centers for the first time, using NVIDIA Blackwell GPUs, Intel TDX, and Google's Titan chip. Apple maintains an independent append-only hardware ledger and dual-vendor attestation roots. AWS and Azure are not part of the collaboration. By Steef-Jan Wiggers

📚 Series: Databricks on AWS (Part 0, prologue) The Target Architecture ← you are here Building a Databricks AI Platform on AWS RBAC with Function-Role Groups Compute Governance: Pools, Policies, Clusters The BOOTSTRAP_TIMEOUT Mystery Fixing It with AWS PrivateLink How We Structure the Terraform Before the build story, here's the destination. This is the target-state data architecture we designed the whole platform toward — the three principles that shaped every later decision, and the Unity Catalog governance model that keeps production data safe from human hands. The rest of this series is a build log: workspaces, RBAC, compute, the networking rabbit hole, the Terraform layout. But every one of those decisions was made in service of a target picture we drew first . This post is that picture — the "to-be" architecture, not the scaffolding we happened to have up on any given week. It's built on three things Databricks basically hands you if you lean into them: the Lakehouse (one store, ACID tables, no separate warehouse to sync), the Medallion architecture (raw → cleaned → integrated → business, each layer a promotion), and Unity Catalog as the single governance plane across all of it. The interesting part isn't reciting those three buzzwords — it's the specific way we wire them so that prod, dev, and analyst sandboxes never step on each other. Three principles, and everything follows Almost every concrete rule later in this series is a consequence of one of these three. 1. Nobody touches production by hand. Create, update, delete in prod data happens only through an automated, code-reviewed pipeline running as a service principal. Human accounts don't get write on prod — not analysts, not engineers, not admins. The blast radius of a bad afternoon is capped at whatever a person can do with read-only. This one principle is why the whole "promote" flow later exists. 2. Never copy production to look at it. If an analyst wants to explore the gold layer, they read it in p

1. The Problem It Solves Logistic Regression is used when the outcome is a category rather than a number . Most commonly, it's used for binary classification , where the answer is either Yes or No , True or False , or 1 or 0 . Typical business problems include: Will a customer churn? Is this transaction fraudulent? Will a customer click an ad? Will a loan default? Is an email spam? Will a machine fail in the next 24 hours? Unlike Linear Regression, we're not trying to predict a continuous value. Instead, we're predicting the probability that an event belongs to a particular class. For example: A customer may have an 82% probability of churning . The business can then decide whether that probability is high enough to trigger an intervention. 2. Core Intuition Imagine you're trying to predict whether a customer will cancel their subscription. Suppose the only feature you have is how many times they opened your app this month. If you use a straight line like Linear Regression, the predictions quickly become unrealistic. A very active customer might end up with a -20% chance of churn . A completely inactive customer could end up with 140% . Probabilities obviously can't work like that. To fix this, Logistic Regression takes the linear equation and passes it through a mathematical function called the Sigmoid Function . Instead of producing a straight line, it creates an S-shaped curve . No matter how large or small the input becomes, the output always stays between 0 and 1 . That makes it perfect for probability estimation. 3. The Mathematical Model The model first calculates a linear score. Instead of using that score directly, it passes it through the Sigmoid function. Where: z = linear score p̂ = predicted probability The final output is always between 0 and 1 . For example: 0.08 → Very unlikely 0.32 → Low risk 0.65 → Moderate risk 0.94 → Very high probability Businesses can then choose a decision threshold. For example: Probability ≥ 0.50 → Predict Churn Probability

In the rush to build AI agents, we defaulted to complex vector databases. But high-traffic platforms are converging on a simpler, more robust foundation: plain files. Most long-term agent memory setups are massively over-engineered. When developers start building LLM applications, the default prescription is almost always: "Spin up a managed vector database and build a RAG pipeline." But if you look at the highest-traffic production agent platforms (like Claude Code, Manus, and OpenClaw), a quieter trend has emerged. They are bypassing the enterprise embeddings store and using plain markdown files as their primary memory substrate. This is not a regression to simplicity. Done well, it is a stronger engineering foundation because files are inspectable, diffable, portable, and git-native. But a folder of plain text notes with no structure is just a slow, poorly indexing database. To make a file-first architecture work at scale, you must follow a fundamental system design principle: separate storage from search . The Core Invariant: Storage vs. Search The single highest-leverage decision you can make in agent memory design is treating your storage layer and search indexes as completely separate systems. Storage (Canonical Source of Truth): Versioned, human-readable files (Markdown + YAML frontmatter). Search (Derived Index): Derived search structures (vector databases, full-text BM25 indexes, entity graphs, keyword indexes). In this architecture, every search index is treated as a disposable artifact. You can delete your vector embeddings database or rebuild your entity graph at any time, with zero loss of underlying memory. This buys you three advantages: Auditability for free: By storing memories in text files, you can version-control them using Git. Every memory update, supersession, or correction is diffable, attributable, and reversible without any custom database versioning logic. Algorithmic freedom: Swap your embedding models, adjust your chunking strategies, o

Record home battery installations unlock options for grids—and AI data centers.

Instacart redesigned its personalized marketing system using a configuration-driven multi-tenant architecture on Storefront Pro. The system replaces retailer-specific implementations with a shared execution engine, enabling scalable personalization, faster configuration propagation in under a minute, and 99.9% delivery success across hundreds of retail banners through a unified campaign platform. By Leela Kumili

Cassie Shum discusses the architectural evolution of GraphRAG and why data foundations are critical for advanced AI workflows. She explains how traditional vector RAG falls short when addressing global context, multi-hop reasoning, and provenance. She shares enterprise strategies for building semantically structured knowledge graphs that shift raw orchestrating logic down to the data layer. By Cassie Shum