标签：#Data

The speakers discuss Agent RFT, OpenAI’s platform for fine-tuning reasoning models via real-time tool interactions and custom reward signals. They explain how reinforcement learning solves complex credit assignment challenges within the context window. They share enterprise success stories, showing how Agent RFT eliminates long-tail token loops and drives extreme efficiency. By Wenjie Zi, Will Hang

If you ask any senior backend engineer or database administrator how to instantly make a slow, disk-bound application faster, their first answer is almost always: "Put it in memory." But why is memory so preferred, and how do modern architectures utilize RAM to achieve sub-millisecond latencies? We're seeing a massive revival of RAM disks and in-memory architectures. Let's explore why computer experts are increasingly treating RAM like a hard drive. 1. The Physics of Storage: Why RAM Wins To understand the shift towards in-memory architectures, we have to look at the numbers. Hard Disk Drives (HDDs): Mechanical spinning disks. Seek times are around 2-5 milliseconds . Solid State Drives (SSDs): Flash memory. Seek times are around 0.1 milliseconds (100 microseconds) . RAM (Random Access Memory): Volatile silicon. Access times are around 100 nanoseconds . RAM is roughly 1,000 times faster than an SSD and 10,000 to 50,000 times faster than an HDD. When you have a high-throughput system serving millions of requests per second, waiting for a disk to seek is an eternity. 2. In-Memory Databases: Redis and Memcached The most common implementation of this principle in modern backends is the In-Memory Database . How They Work Instead of writing every transaction to an SSD, systems like Redis and Memcached store the entire dataset directly in RAM. This bypasses the OS file system cache, disk I/O bottlenecks, and complex B-tree traversals required by traditional relational databases like PostgreSQL or MySQL. The Trade-off: Durability RAM is volatile. If the server loses power, all data is gone. So how do in-memory databases survive crashes? Snapshots (RDB in Redis): Periodically dumping the entire memory state to disk. Append-Only Files (AOF in Redis): Logging every write operation to a disk sequentially. Sequential writes to disk are significantly faster than random writes. This hybrid approach gives you the read/write speed of RAM with a "good enough" durability guarantee for

The Quest Begins (The "Why") I still remember the first time I faced a problem that asked for the sum of numbers in a sub‑array, over and over again, with updates sprinkled in between. It felt like I was stuck in a never‑ending loop of for i in range(l, r+1): total += arr[i] – O(n) per query, and with up to 10⁵ queries the solution timed out every single time. I was staring at the screen, thinking, “There has to be a smarter way to answer these range questions without scanning the whole array each time.” That moment was my dragon: a seemingly simple problem that kept biting me because I kept reaching for the brute‑force sword. I needed a data structure that could give me the answer in logarithmic time while still supporting point updates. Enter the segment tree – the tool that turned my O(n·q) nightmare into an O((n+q)·log n) victory. The Revelation (The Insight) So why does a segment tree work? Imagine you have an array and you want to know the sum of any interval [l, r] . If you could break that interval into a handful of pre‑computed chunks, you’d only need to add those chunk values together instead of touching every element. A segment tree is exactly that: a binary tree where each node stores the aggregate (sum, min, max, etc.) of a segment of the original array. The root covers the whole array [0, n‑1] . Its two children cover the left half and the right half, and this keeps splitting until the leaves represent single elements. The magic lies in two facts: Every node’s value is a function of its children. If you know the sum of the left child and the sum of the right child, the parent’s sum is just their addition. This means we can build the tree bottom‑up in O(n) time. Any interval can be represented as O(log n) disjoint nodes. When you walk down the tree to answer [l, r] , you either take a whole node (if its segment lies completely inside the query) or you recurse further. Because the tree’s height is log₂n, you’ll visit at most 2·log₂n nodes. Thus, building

Here is a question I could not answer from the headlines: which European countries are actually losing people the fastest, in absolute terms or per capita? Those are two different questions, and they give two different answers. So I pulled the open data and ran the numbers. The headline figure Across the 19 European countries in the 2024 dataset, 17 recorded a net loss of native-born residents . Only two were net positive. So the "brain drain" story is not a handful of outliers, it is the default state of the continent. But the interesting part is who tops the ranking, because it depends entirely on how you measure. Load the data yourself The dataset is public on GitHub (CC BY 4.0). Every number below is reproducible with a few lines of pandas. No download, no API key, it reads the raw CSV straight from the repo: import pandas as pd url = ( " https://raw.githubusercontent.com/DatapulseResearch/ " " brain-drain-eu/main/data/net_migration_native_born_2024.csv " ) df = pd . read_csv ( url ) print ( df . shape ) # (19, 3) print ( df . columns . tolist ()) # ['country', 'net_migration', 'per_1000_residents'] # How many countries lost native-born residents? losers = ( df [ " net_migration " ] < 0 ). sum () print ( f " { losers } of { len ( df ) } countries had a net loss " ) # 17 of 19 net_migration is the raw count for 2024 (negative means a net loss of native-born residents). per_1000_residents is the same flow normalized by population size. The absolute ranking: Germany runs away with it Sort by the raw count and one country dominates: worst_absolute = df . sort_values ( " net_migration " ). head ( 5 ) print ( worst_absolute [[ " country " , " net_migration " ]]) country net _ migration 0 Germany - 91067 ... Germany loses -91,067 native-born residents, far more than anyone else in absolute terms. If you stop reading here, the story writes itself: "Germany, Europe's biggest brain drain." Plenty of coverage did exactly that. The counterintuitive finding: the ranking inve

Author: Tobie Morgan Hitchcock Today we're launching Scale, a new tier of SurrealDB Cloud built for the workloads you can't afford to have go down. Our first tier, Start, was designed for building and shipping fast. Scale is designed for what happens next: production traffic, uptime commitments, and the kind of resilience your users never notice because there are always available nodes. It's the tier for teams running SurrealDB as the scalable context layer behind real applications and AI agents in production. What you get with Scale Scale is about one thing: keeping your database available and consistent under real-world conditions. Highly-available, fault-tolerant clusters. Scale runs your database as a multi-node cluster designed to survive node and infrastructure failures without dropping writes or losing consistency. A single point of failure is no longer a single point of downtime. Multiple availability-zone deployment. Your cluster is distributed across multiple availability zones, so even the loss of an entire zone doesn't take your database with it. Traffic keeps flowing while the cluster recovers in the background. Horizontal scale. As demand grows, Scale grows with it. Add capacity by scaling out across nodes rather than being capped by the size of a single machine. Start with three nodes, and keep adding to scale your application or agent's needs. See more information about SurrealDB Cloud Scale architecture here . Built on SurrealDS Scale is powered by SurrealDS , SurrealDB's distributed storage engine and the foundation that makes all of this possible. SurrealDS is a new generation distributed storage architecture, rethought from first principles. Instead of coupling storage to compute on a single box or to a proprietary cloud tier, SurrealDS embeds consensus directly in SurrealDB nodes and separates the two layers cleanly. Here's what that architecture gives you. Architecture overview Compute and storage separation. Scale compute for QPS and storage f

A top Democrat on the Senate's Intelligence Committee warned that the information accessed on a Homeland Security intelligence-sharing network may risk national security.

Google tries balancing AI data center emissions with clean energy efforts.

Apple chose Google Cloud to run Private Cloud Compute outside its own data centers for the first time, using NVIDIA Blackwell GPUs, Intel TDX, and Google's Titan chip. Apple maintains an independent append-only hardware ledger and dual-vendor attestation roots. AWS and Azure are not part of the collaboration. By Steef-Jan Wiggers

📚 Series: Databricks on AWS (Part 0, prologue) The Target Architecture ← you are here Building a Databricks AI Platform on AWS RBAC with Function-Role Groups Compute Governance: Pools, Policies, Clusters The BOOTSTRAP_TIMEOUT Mystery Fixing It with AWS PrivateLink How We Structure the Terraform Before the build story, here's the destination. This is the target-state data architecture we designed the whole platform toward — the three principles that shaped every later decision, and the Unity Catalog governance model that keeps production data safe from human hands. The rest of this series is a build log: workspaces, RBAC, compute, the networking rabbit hole, the Terraform layout. But every one of those decisions was made in service of a target picture we drew first . This post is that picture — the "to-be" architecture, not the scaffolding we happened to have up on any given week. It's built on three things Databricks basically hands you if you lean into them: the Lakehouse (one store, ACID tables, no separate warehouse to sync), the Medallion architecture (raw → cleaned → integrated → business, each layer a promotion), and Unity Catalog as the single governance plane across all of it. The interesting part isn't reciting those three buzzwords — it's the specific way we wire them so that prod, dev, and analyst sandboxes never step on each other. Three principles, and everything follows Almost every concrete rule later in this series is a consequence of one of these three. 1. Nobody touches production by hand. Create, update, delete in prod data happens only through an automated, code-reviewed pipeline running as a service principal. Human accounts don't get write on prod — not analysts, not engineers, not admins. The blast radius of a bad afternoon is capped at whatever a person can do with read-only. This one principle is why the whole "promote" flow later exists. 2. Never copy production to look at it. If an analyst wants to explore the gold layer, they read it in p

1. The Problem It Solves Logistic Regression is used when the outcome is a category rather than a number . Most commonly, it's used for binary classification , where the answer is either Yes or No , True or False , or 1 or 0 . Typical business problems include: Will a customer churn? Is this transaction fraudulent? Will a customer click an ad? Will a loan default? Is an email spam? Will a machine fail in the next 24 hours? Unlike Linear Regression, we're not trying to predict a continuous value. Instead, we're predicting the probability that an event belongs to a particular class. For example: A customer may have an 82% probability of churning . The business can then decide whether that probability is high enough to trigger an intervention. 2. Core Intuition Imagine you're trying to predict whether a customer will cancel their subscription. Suppose the only feature you have is how many times they opened your app this month. If you use a straight line like Linear Regression, the predictions quickly become unrealistic. A very active customer might end up with a -20% chance of churn . A completely inactive customer could end up with 140% . Probabilities obviously can't work like that. To fix this, Logistic Regression takes the linear equation and passes it through a mathematical function called the Sigmoid Function . Instead of producing a straight line, it creates an S-shaped curve . No matter how large or small the input becomes, the output always stays between 0 and 1 . That makes it perfect for probability estimation. 3. The Mathematical Model The model first calculates a linear score. Instead of using that score directly, it passes it through the Sigmoid function. Where: z = linear score p̂ = predicted probability The final output is always between 0 and 1 . For example: 0.08 → Very unlikely 0.32 → Low risk 0.65 → Moderate risk 0.94 → Very high probability Businesses can then choose a decision threshold. For example: Probability ≥ 0.50 → Predict Churn Probability

In the rush to build AI agents, we defaulted to complex vector databases. But high-traffic platforms are converging on a simpler, more robust foundation: plain files. Most long-term agent memory setups are massively over-engineered. When developers start building LLM applications, the default prescription is almost always: "Spin up a managed vector database and build a RAG pipeline." But if you look at the highest-traffic production agent platforms (like Claude Code, Manus, and OpenClaw), a quieter trend has emerged. They are bypassing the enterprise embeddings store and using plain markdown files as their primary memory substrate. This is not a regression to simplicity. Done well, it is a stronger engineering foundation because files are inspectable, diffable, portable, and git-native. But a folder of plain text notes with no structure is just a slow, poorly indexing database. To make a file-first architecture work at scale, you must follow a fundamental system design principle: separate storage from search . The Core Invariant: Storage vs. Search The single highest-leverage decision you can make in agent memory design is treating your storage layer and search indexes as completely separate systems. Storage (Canonical Source of Truth): Versioned, human-readable files (Markdown + YAML frontmatter). Search (Derived Index): Derived search structures (vector databases, full-text BM25 indexes, entity graphs, keyword indexes). In this architecture, every search index is treated as a disposable artifact. You can delete your vector embeddings database or rebuild your entity graph at any time, with zero loss of underlying memory. This buys you three advantages: Auditability for free: By storing memories in text files, you can version-control them using Git. Every memory update, supersession, or correction is diffable, attributable, and reversible without any custom database versioning logic. Algorithmic freedom: Swap your embedding models, adjust your chunking strategies, o

Record home battery installations unlock options for grids—and AI data centers.

Instacart redesigned its personalized marketing system using a configuration-driven multi-tenant architecture on Storefront Pro. The system replaces retailer-specific implementations with a shared execution engine, enabling scalable personalization, faster configuration propagation in under a minute, and 99.9% delivery success across hundreds of retail banners through a unified campaign platform. By Leela Kumili

Cassie Shum discusses the architectural evolution of GraphRAG and why data foundations are critical for advanced AI workflows. She explains how traditional vector RAG falls short when addressing global context, multi-hop reasoning, and provenance. She shares enterprise strategies for building semantically structured knowledge graphs that shift raw orchestrating logic down to the data layer. By Cassie Shum

By the team at MailTester Ninja — a real-time email verification API that stores nothing. We verify a lot of email for a living. So we pointed our infrastructure at a representative panel of 50,000 of the world's most-linked domains and measured how email is actually configured in 2026 — MX providers, SPF and DMARC. Pure DNS, aggregate only, no personal data . Here's what the internet's mail setup looks like right now. Email is still (almost) everywhere 79.9% of these domains are mail-enabled (they publish MX records). Email isn't going anywhere. Authentication: adopted, but not enforced 75.8% publish an SPF record 64% publish a DMARC record …but only 22.6% actually enforce it with p=reject That last number is the real story. Of the domains that bother to publish DMARC, only 35.2% are on p=reject — the rest sit on p=none (37.2%, monitoring only) or quarantine (27.6%). Most of the web announces a policy it doesn't enforce. That's a deliverability and spoofing gap hiding in plain sight. Who runs the world's inboxes? Other / self-hosted — 32.6% Google Workspace / Gmail — 28.2% Microsoft 365 / Outlook — 22.5% Proofpoint — 5.5% Mimecast — 3.1% Tencent QQ — 2% Namecheap — 1.3% Cisco IronPort — 0.9% Self-hosted and the two hyperscalers (Google Workspace and Microsoft 365) dominate, but the long tail of providers is very real — which is exactly why deliverability is hard: every provider blocks, greylists and reputation-scores differently. Why we publish this We built an open, daily-updated dataset and a live dashboard because deliverability decisions should be based on data, not folklore. It's CC BY 4.0 — use it, cite it, build on it. Want to check a specific domain? Our free analyzer shows any domain's MX / SPF / DMARC in one click — no signup, nothing stored. Methodology: Live DNS scan (MX/SPF/DMARC). Aggregate only — no email sent, no personal data. Sample updated Wed, 01 Jul 2026 12:31:00 GMT.

Quick Answer (TL;DR) Modifying an RDS instance class in place causes 5 to 15 minutes of downtime while AWS reboots the database. To right-size without downtime, use RDS Blue/Green Deployments (fastest, cleanest), a read-replica promotion (works on older engines), or a Multi-AZ failover to a resized standby. Blue/Green is the 2026 default for most workloads on MySQL, MariaDB, Postgres, and now SQL Server. Why this happens RDS instances are Managed EC2 hosts running the DB engine, and a class change (say db.m6i.large to db.m6i.xlarge ) requires stopping the process, migrating the EBS volumes to a new host, and restarting. AWS's default "modify" workflow does this in place and warns you about downtime. The workarounds exist because that reboot is unacceptable for user-facing services, so you build the new instance alongside the old one and cut over. Fix #1: Use RDS Blue/Green Deployments The 2026 default. Available for RDS MySQL, MariaDB, PostgreSQL, and SQL Server (added mid-2025). Steps: In the RDS console, select the instance and choose Actions → Create Blue/Green Deployment . Set the Green instance to your target instance class. AWS creates a full standby using logical replication, keeps it in sync, and validates health. When ready, click Switch over . Cutover typically takes under 60 seconds. Applications reconnect using the same endpoint. Command-line equivalent: aws rds create-blue-green-deployment --blue-green-deployment-name resize-prod --source arn:aws:rds:... --target-db-instance-class db.m6i.xlarge Best when: your engine supports it and you can tolerate the extra cost of running two instances for the sync window. Fix #2: Read-replica promotion For engines or versions that do not yet support Blue/Green, or for cross-region resizing. Steps: Create a read replica with the desired new instance class. Wait for the replica to catch up (near-zero lag). Point application writes to the read replica endpoint (requires connection-string change or DNS switch). Promote

Data Modeling, Relationships, and Schemas in Data Analytics In the fields of data analytics, data warehousing, and database management, modeling and schema design are the fundamental pillars used to organize and query information efficiently. This article provides a comprehensive guide to these core concepts. 1. Data Modeling Data modeling is the architectural process of designing how data is stored, interconnected, and accessed within a system. Core Questions Addressed: Storage: What specific data points need to be captured? Structure: How should individual tables be organized? Connectivity: How do these tables interact with one another? Levels of Data Models: Conceptual Model: A high-level business perspective focusing on entities and their relationships, devoid of technical specifications. Logical Model: Defines specific attributes, keys, and relationships. It is independent of the Database Management System (DBMS). Physical Model: The actual implementation within a database, including technical details like indexes, partitions, and storage requirements. 2. Relationships Relationships define the logic of how data in one table corresponds to data in another. One-to-One (1:1): A single record in Table A relates to exactly one record in Table B. One-to-Many (1:M): The most common relationship; for example, one Customer can place many Orders . Many-to-Many (M:M): Multiple records in one table relate to multiple records in another. This requires a Junction Table (Bridge Table) to function. Example: One Student can enroll in many Courses, and one Course contains many Students. 3. SQL Joins Joins are used to combine rows from two or more tables based on a related column. Join Type Description Inner Join Returns only the records that have matching values in both tables. Left Join Returns all records from the left table and the matched records from the right. Right Join Returns all records from the right table and the matched records from the left. Full Outer Join Returns

I've been building on DynamoDB since around 2015, and these days I build tools for it: dynoxide , a DynamoDB engine, and Nubo , a native client. So I'm not neutral about it. It's the first database I reach for, and with reason - the operational overhead is close to nil, no connections to pool or instances to size, and it holds the same single-digit-millisecond reads whether the table has a thousand items or a billion. The data modelling is a craft, and a satisfying one. It's also one of the harder databases to learn, and that's the part I keep coming back to. DynamoDB punishes the instincts you bring from SQL. You don't normalise and join at read time; you work out the questions your app will ask first, and shape the data around those access patterns, until one table answers all of them. It's a real shift in how you think, and it's where a lot of people bounce off - it feels backwards right up until it clicks. The people who teach it best all teach it the same way. Alex DeBrie's The DynamoDB Book , the arc.codes team's examples, Rick Houlihan's re:Invent talks - the legendary ones, where he models half a dozen access patterns onto a single table at a hundred miles an hour - none of them hand you rules to memorise. They show you patterns and make you run them. I learned a lot of my DynamoDB from all three, and it stuck because I was building as I went. That last part is the bit that's hard to come by on your own. Reading about an access pattern and having it in your fingers are different things, and to run one - build the table, write the items, fire the query and see what comes back - you need an AWS account, or a local emulator installed and seeded. Enough friction that plenty of people read about single-table design without ever building one. There was a second thing pulling the same way. dynoxide had learned to run in the browser only last month, compiled to WebAssembly with no server behind it, and it was a preview I didn't fully trust. What it needed was a real

INTRODUCTION When I started using Power BI, I only thought of visuals like charts and graphs. However, as I progressed, I discovered a great data dashboard is built on great data models. Data Modelling is the process of organizing your data tables and defining how they relate to each other so Power BI can combine them into meaningful reports and dashboards. Good, designed data makes it easier and faster to maintain. Why is data Modelling Important Well-organized data makes it easier to manage data. Reduction of the duplicates. Ensures data consistency. Understanding Relationships Relationships allow the data table to give communication using fields. For example, Customer Table stores all information about a customer. Product Table store product details Sales Table stores all information about the transactions. Power BI connects the information between the customer’s name and Customer Id rather than repeating them it connects the information using joins. Going through relationships I discovered schemes. Scheme is the way tables are organized in databases. There are different types of schemes e.g. Star Schema, snowflake schema and Flat table. Star Schema A star schema is a data model with one central fact table and dimension table surrounding it. Fact table A table that stores events, transactions of what happened. • Total sales • average sales • quantity sold Dimension table A dimension table describes the items in the fact table. The table contains descriptive information. • The customer table describes the customer • How much sales were made The fact table sits in the center, while the dimension tables surround it—forming a star. Dashboard designs A good dashboard has to fit one page. A dashboard should show critical information. Update automatically when data changes. Focus on data understanding and decision making. Conclusion Power BI taught me that a great report are built from a a great dashboard which is achieved by having great models. Structuring a data into

My dashboard had a useMemo doing arithmetic it had no business doing. It was a Pokémon TCG Pocket collection tracker, but that part doesn't matter. What matters is that the home page needed to show three things: overall completion, completion per set, and which set you were closest to finishing. The way I'd built it, the browser was fetching every card and every owned record, then grinding through the math on each render to figure all of that out. It worked. It also got slower and harder to read every time the data grew or I added a metric. So I moved the aggregation out of React and into Postgres, and the surprising result was that my frontend got boring . Fewer hooks, less state, almost nothing left to break. That's the whole argument of this post: aggregation belongs in the database, and when you put it there, the React code that's left over is the kind of boring you actually want in the layer your users touch. What "fetching everything into React" actually looks like Here's the shape of the original dashboard. Load all the cards, load the user's owned rows, then derive everything on the client. const [ cards , setCards ] = useState < Card [] > ([]); const [ owned , setOwned ] = useState < OwnedCard [] > ([]); useEffect (() => { ( async () => { const { data : allCards } = await supabase . from ( " cards " ). select ( " * " ); const { data : ownedRows } = await supabase . from ( " user_cards " ) . select ( " card_id " ); setCards ( allCards ?? []); setOwned ( ownedRows ?? []); })(); }, []); const ownedIds = useMemo (() => new Set ( owned . map (( o ) => o . card_id )), [ owned ]); const perSet = useMemo (() => { const groups : Record < string , { total : number ; have : number } > = {}; for ( const card of cards ) { const g = ( groups [ card . set_id ] ??= { total : 0 , have : 0 }); g . total += 1 ; if ( ownedIds . has ( card . id )) g . have += 1 ; } return groups ; }, [ cards , ownedIds ]); const overall = useMemo (() => { const total = cards . length ; const ha