标签：#Data

There is a comforting story about how AI models handle the enormous quantities of text and images they are trained on: they do not store any of it, they merely learn general patterns, and once training is done the original data is gone in any meaningful sense. It is a reassuring account, and it is not quite true. Large models memorise fragments of their training data — verbatim, recoverable fragments — and a decade of research has produced reliable ways to detect and extract them. The answer-first version: if your data was in a model’s training set, the model may have memorised identifiable pieces of it, and those pieces can leak. Two families of attack make this concrete. Membership inference works out whether a specific record was in the training data at all. Data extraction pulls memorised content back out word-for-word. Neither is exotic; both are well documented against production systems. This is the mechanism underneath both the newspaper lawsuits alleging near-verbatim reproduction of their articles and the quieter privacy research showing that models leak the people in their training sets. Understanding it is the difference between trusting the comforting story and knowing its limits. Memorisation is a feature of the maths, not a bug Start with why models memorise at all. A large neural network has an enormous number of parameters — enough capacity to do more than compress general patterns. During training it is rewarded for predicting its training data accurately, and one very effective way to predict a specific example accurately is to memorise it. For data that appears once in an unusual form, or many times in an identical form, memorisation is often the path of least resistance for the optimiser. This is measurable. Researchers can show that a model assigns systematically higher confidence, and lower prediction error, to examples it was trained on than to otherwise-similar examples it has never seen. The size of that gap grows with the size of the model

Some Americans fear the FTC may be thinking about personalized pricing all wrong.

There's about to be a big fight to secure access to space.

Mark (aka Winsaucerer) here to show you how you can test your PostgreSQL database like a sorcerer. We are going to be using Spawn, a SQL build system supporting migrations and testing. You do not need to be using Spawn for migrations in order to use it for testing. Spawn does not require any extension installed. All you need is the spawn CLI and a psql connection to the database for Spawn to connect through. Spawn was built to solve some migration pains I've experienced, but I happily discovered that when used for testing, it is very powerful. To show you some of that power, we're going to use a contrived database example. It uses golden file testing to determine success. When the test runs, we capture the stdout and stderr output from psql, and compare that to expected output. Testing with Spawn involves these steps: Create a new test with spawn test new <name> and fill out the test steps Check test outputs with spawn test run <name> (or view the SQL that will be sent to psql via spawn test build <name> ) When outputs are as expected, create the golden file with spawn test expect <name> Run the test and compare to expected output with spawn test compare <name> For now, Spawn only supports connecting via psql, which means that you have access to all the features that psql provides. To get started, follow the Spawn install instructions: Install Spawn And then create a new folder on your system, and initialise a new project with a docker compose config ready for us to play with: # inside your new folder: spawn init --docker docker compose up -d You now have a running docker based PostgreSQL database and a spawn.toml file configured to connect to it. We are not assuming that you are using Spawn or any other tool for migrations, so you can manually create and update the database by connecting directly using psql: docker exec -ti postgres-db psql -U postgres Create the database ⚠️ Caution This post is not intended as an example of how to build an orders database. The des

Five states, one query language, and an "active licenses only" checkbox that only actually filters two of them. That's the trap in scraping US childcare-license open-data registries: Socrata SODA makes every state's API look identical, but "active" is defined — or not defined at all — differently in every dataset. Quick answer New York, Connecticut, Colorado, Delaware, and Texas all publish their childcare-facility registries through Socrata, and all five accept the same $where query syntax. But only NY and CT ship a server-side status filter this Actor can apply. Colorado and Delaware have no status column in the dataset at all — there's nothing to filter on. Texas does have a status column ( operation_status ), it's just not wired into the active-only filter, so toggling activeOnly doesn't touch Texas rows either way. Treating "active only" as a global switch that behaves the same everywhere will silently hand you closed and revoked facilities in three of the five states while you believe you filtered them out. STATE_CONFIGS : dict [ str , StateConfig ] = { " NY " : StateConfig (..., col_status = " facility_status " , active_where = " facility_status= ' Active '" ), " CT " : StateConfig (..., col_status = " status " , active_where = " status= ' ACTIVE '" ), " CO " : StateConfig (..., col_status = None ), # no status column to filter on " DE " : StateConfig (..., col_status = None ), # no status column to filter on " TX " : StateConfig (..., col_status = " operation_status " ), # status exists, filter isn't wired } Why does "active only" do nothing in three states? Because the filter is applied per-state, not globally, and only two states have both a status column and a configured $where fragment for it: async def _fetch_page (...): params = { " $limit " : str ( page_limit ), " $offset " : str ( offset ), " $order " : config . order_key } if active_only and config . active_where : params [ " $where " ] = config . active_where return await _get_with_retry ( session

"Just query the Socrata API" is true and also useless advice. Socrata SODA is a real open standard — New York, Connecticut, Colorado, and Texas all expose their professional-license registries through the same $limit / $offset / $where query language. The standard ends there. What each state puts inside that standard is four unrelated data models wearing the same protocol. Quick answer Every state's cosmetology/barber/salon registry is one giant multi-profession table with its own column names, its own beauty-credential filter, and its own idea of what "active" means — and one state (Texas) doesn't expose a status column at all, so an activeOnly toggle is a silent no-op there. A generic Socrata client that assumes one schema will either miss most of the data or crash on the first state whose columns don't match. The fix is a per-state config object that maps each state's real column names to one canonical output row, with the active-license filter applied only where the underlying data supports it. @dataclass ( frozen = True ) class StateConfig : state : str endpoint : str order_key : str col_business_name : str | None col_licensee_name : str | None col_status : str | None base_where : str | None = None active_where : str | None = None Why does the same query return different professions per state? Cosmetology licenses don't get their own dataset — they're rows buried inside each state's entire professional-licensing table, next to electricians, dentists, and notaries. Filtering has to happen server-side, in SoQL, before pagination even starts, or you're downloading (and paying to store) irrelevant rows. Texas needs a starts_with() match across three license-type prefixes plus an Establishment wildcard; Connecticut needs an exact in() list of six credential names; Colorado needs a four-code in() list: TX_BEAUTY_WHERE = ( " starts_with(license_type, ' Cosmetology ' ) " " OR starts_with(license_type, ' Class A Barber ' ) " " OR starts_with(license_type, ' Barber ' ) "

Top Vector Databases for AI Agents in 2026: Qdrant vs Pinecone vs Weaviate vs PgVector vs Milvus Persistent memory is the foundation that turns a stateless LLM into a continuously improving, autonomous agent. In 2026, selecting a vector database is no longer just about raw Approximate Nearest Neighbor (ANN) speed. For AI agents, the critical requirements have shifted to: Payload & Metadata Filtering : Can you filter by tenant_id , user_id , and timestamp during vector graph traversal without sacrificing recall? Hybrid Search (BM25 + Dense Vectors + Sparse SPLADE) : Combining exact keyword matching (for code symbols and error codes) with semantic understanding. Multi-Tenancy & Memory Namespacing : Safely isolating memory blocks across thousands of users and sessions. Billion-Scale Quantization (Product Quantization & Scalar Quantization) : Slashing RAM costs by 75–90% in production. This guide provides a comprehensive architectural comparison of the top 5 vector databases for AI agents in 2026. Head-to-Head Comparison Matrix Feature / Metric Qdrant Pinecone (Serverless) Weaviate PgVector (PostgreSQL) Milvus Primary Architecture Rust-native, disk-backed Fully managed serverless Go-native, modular RAG PostgreSQL extension Distributed cloud-native Open Source Yes (Apache 2.0) Proprietary SaaS Yes (BSD-3) Yes (Open Source) Yes (Apache 2.0) Payload Filtering Exceptional (HNSW custom payload indexing) Good (Metadata filtering) Strong (Inverted index + HNSW) SQL WHERE clause Strong (Partition keys) Hybrid Search Native (Dense + Sparse vectors) Native hybrid Native BM25 + Vector SQL text search + pgvector Native multi-vector Quantization Scalar & Product Quantization (Binary) Automatic serverless compression PQ, BQ, SQ Halfvec, Binary Quantization Scalar / Product Quantization Best Fit High-performance agent memory & self-hosted RAG Zero-maintenance cloud SaaS GraphQL & multi-modal search Unified relational + vector apps Ultra-large enterprise (100M+ vectors) 1. Qdrant: The

Disclosure: I build Schemity , a desktop ERD tool - this post is from our blog and uses it for the examples. TL;DR: The database has a built-in place to document a column - COMMENT ON COLUMN in PostgreSQL, the COMMENT attribute in MySQL - and almost nobody fills it in, because a sentence of prose has to travel the same path as a schema change: a migration file, a review, a deploy. Schemity keeps field descriptions in the diagram instead, where editing one generates no SQL, reads existing database comments in on import, and exports the result as a data dictionary. You can document a database column without touching the database: write the description in the model rather than in the schema. That sounds like a dodge until you price the alternative. The database's own mechanism for column documentation, COMMENT ON COLUMN in PostgreSQL and the COMMENT attribute in MySQL, sends a sentence of prose down exactly the same path as a change to how data is stored - a migration file, a code review, an approval, a deploy window - and on MySQL it does something worse than that. Schemity keeps field descriptions in the diagram, where editing one produces no SQL at all. This is why so many production schemas have thousands of columns and almost no comments. Not because nobody wanted to write them. Because writing one costs a deploy. How do I document a database column without running a migration? Keep the description in the model rather than in the storage engine. A field description is a fact about what the column means to your team; it changes no type, no constraint, no index, and nothing about what the database will accept. When it lives in the diagram, editing it is like editing a comment in a code file: you change it, review it in the same pull request as everything else, and nothing has to run against production for it to take effect. The moment that description is a column comment, it stops being prose and becomes DDL. Now it needs a migration file, and the migration needs a

Surging demand for AI training data is driving rapid growth for the startup and its rivals.

In the current epoch of Artificial Intelligence, the industry remains singularly preoccupied with the "Model" — obsessing over the raw parameter scales of the latest LLMs or the specific benchmark performance of a new transformer variant. However, at Informatiqs, we shift the lens. We recognize that sustainable enterprise value is rarely derived from the model in isolation; instead, it emerges from the high-stakes architectural decisions and systemic orchestration that define its environment. As we launch our inaugural edition, we dissect a critical technological nexus: the convergence of Graph Neural Networks (GNNs), Generative AI, and the industrial-grade infrastructure of Google Cloud Platform (GCP). We argue that for complex enterprise datasets, the transition from flat vector embeddings in latent space toward non-Euclidean, graph-based relational intelligence is the primary differentiator for the next generation of resilient AI applications. 1. The Scientific Foundation: Exploiting Relational Inductive Bias Traditional Deep Learning architectures, such as Convolutional Neural Networks (CNNs) for images or Transformers for text, primarily operate on data structured as sequences (Euclidean space). While exceptionally powerful, these structures often fail to capture the topological nuances of real-world systems like supply chains, molecular structures, or fraudulent transaction webs where data is inherently non-Euclidean. Graph Neural Networks (GNNs) provide a framework for learning from data represented as nodes and edges. Unlike standard neural networks that process inputs in isolation, GNNs utilize a Message Passing paradigm. In this process, a node's internal representation is iteratively updated by aggregating information from its immediate neighbors. Instead of looking at a data point as a single row in a database, the GNN looks at who that data point "talks to" and how those connections define its identity. By utilizing Graph Attention mechanisms, we can fu

Jason Kelce joked that people should cool data centers with their pee, rather than potable water -- but his suggestion is not completely ludicrous.

I Gave Five Graph Databases 256MB of RAM Each. Here's What Broke. CognoDB Cloud's free tier gives you a graph database instance with half a CPU core and 256MB of RAM. That's not a lot. It's also, honestly, a pretty realistic starting point a lot of real side projects and early-stage products live exactly there, on whatever the free tier happens to give them, and find out the hard way what their database does under pressure. So I decided to actually find out. I took CognoDB and lined it up against four other graph databases Neo4j AuraDB, FalkorDB, and ArangoDB gave every single one of them the same tiny resource budget, threw the same 198,050-edge dataset at all of them, and ran the same queries. No cherry-picking, no "best case" numbers. Just: here's a small VM's worth of resources, go. One of the databases I originally planned to include never even made it into the results. It crashed on startup. Not "slow to start" a full segfault, reproducibly, across two different versions, with nothing I threw at it fixing it. More on that below, because it's honestly one of the more interesting parts of this whole thing. The setup, quickly Five candidates going in: CognoDB (mandatory, since that's the actual point of this), Neo4j AuraDB Free, Memgraph, FalkorDB, and ArangoDB. Same dataset for all of them a real social-graph-shaped dataset from Stanford's SNAP collection, ~18.7k nodes and ~198k edges, sized specifically to fit inside every platform's free tier without anyone getting an unfair advantage. Same queries too: I wrote every single query 1-hop, 2-hop, 3-hop traversals, point lookups, filtered lookups, aggregations exactly once, then translated each one into whatever query language a given platform actually speaks. No platform ever got a "friendlier" version of a query than another. And everyone ran under the same 0.5 vCPU / 256MB RAM ceiling, whether that was their real cloud free tier or a Docker container I capped by hand to match. The one that didn't survive Memgra

A small release, not a launch. The LAB tab on gex.live has a new rightmost rail called IDEA FEED . It is a stream of short, rule-shaped trading ideas about SPX dealer positioning — "fade the first touch of the call wall after a gap up", that kind of thing — collected daily by a scanner from what people actually discuss, rewritten into something the Lab compiler can parse, and published untested . That last word is the point. Why untested is the feature Every feed of trading ideas on the internet comes with a verdict attached: "this works", "78% win rate", a screenshot of a good month. The feed here refuses to do that. Each card says exactly two things about its idea: compiles clean (our compiler turned the text into a runnable rule without complaint) and untested (nobody has run it against the archive yet). The honest test is yours to run. One click drops the idea into the Lab conveyor. The compiler has already done the translation, so the first message in your session is the rule itself, stamped ↳ from IDEA FEED · compiles, untested . Running the backtest costs one Lab credit; a failed job refunds itself. If your balance is zero the button does not go dead — it turns into 0 CREDITS · BUY → , remembers the idea you picked, and comes back to it after. What you will not find No source attribution on the cards. The idea is the unit, not the poster. No win rates, no "rated", no thumbs. The archive is 1,000+ finished SPX sessions; the Lab tests against all of it with an out-of-sample split and tells you what survived, which so far is: very little. That verdict is worth more than a badge on a card. No approval gate. The scanner's finds ship directly every day, so the feed stays fresh by itself. "NEW" is personal — it means new since you last opened the rail, not new for everyone. Why build a feed that mostly produces "no" Because the alternative is pretending. The whole site is built on measuring dealer positioning from the tape instead of assuming it from yesterday's ope

gex.live has two halves. The terminal measures where SPX options dealers are positioned, every second, from the tape. The Lab is the half that asks the uncomfortable question: does any of that predict anything? What it is A browser-side conveyor with three stages and a credit meter. Compile. You describe a rule in plain text — "short the first touch of the put wall when net gamma is below the 20th percentile" — and the compiler turns it into a deterministic rule over the archive's fields: flip, walls, hold band, gamma percentile, DEX/VEX/vanna/charm per strike, time of day. Compiling is free. If the text is ambiguous the compiler says which part, instead of guessing. Backtest. The rule runs against the full session archive — 1,000+ finished SPX days, every one of them public at gex.live/sessions — with a fixed out-of-sample split. One credit per job; a job that fails refunds itself. Quant optimize. Optional. A LightGBM pass over the same feature store to see whether there is structure the hand-written rule missed, reported as out-of-sample AUC plus feature importance, not as a new "signal". The heavy part (DuckDB + LightGBM) runs in a scale-to-zero container that reads snapshots over HTTPS from the public archive. It depends on no machine and on no private data, which is the point: you are testing against the same files anyone can download. The honest-stats rule Every verdict comes with its baseline. "Your rule made 3% in-sample" means nothing next to "the unconditional drift over the same days was 2.8%". The report shows both, shows the out-of-sample half separately, and refuses to produce a headline number from the in-sample half. Most rules do not survive this. That includes our own: the site's own directional levels were tested three separate ways across the whole archive and none held out of sample — which is why the terminal sells measurement and not signals, and why the Lab exists at all. The free Idea Feed Next to the conveyor sits a rail of rule-shaped idea

The yeetcycling math resembles asteroid mining in reverse.

There's a moment every developer hits the first time they connect an AI assistant to a real database: it works beautifully, the model writes a clean SELECT , you get your answer in seconds — and then a small, cold thought arrives. What if it had written DELETE instead? That worry is healthy. An AI agent that can query your production database is also, by default, an AI agent that can UPDATE , DROP , and TRUNCATE it. Large language models are probabilistic. They hallucinate. They misread a vague prompt like "clean up the test users" as an instruction to actually delete rows. You don't want the only thing standing between a confused model and your orders table to be good intentions. The fix isn't to keep AI away from your data. It's to make write operations structurally impossible — read-only by design, enforced at layers the model can't talk its way past. This post walks through how to do that properly, from the database grant all the way up to query-level guardrails. Why "just prompt it to be careful" fails The tempting shortcut is to add "only run SELECT queries, never modify data" to your system prompt and call it a day. Don't rely on this. Prompt instructions are suggestions, not enforcement. A cleverly worded user request, an injected instruction hidden in some data the model reads, or a plain misunderstanding can all lead the model to generate a destructive statement anyway. Real read-only access is enforced below the model — in places where no amount of clever text can override it. Think of it as defense in depth, with at least three independent layers: Layer What it stops Enforced by Database permissions Any write reaching the engine SQL GRANT / REVOKE Connection / replica Writes even being routed to a writable node Read replica, read-only transaction Query parser / broker Non-SELECT statements before they run SQL parsing, allowlists Any one of these is decent. All three together mean a write has to defeat your database engine, your routing, and your parser s

The database platform Harper advocates for a single-runtime architecture that keeps application code and data together, with its benchmark against a Vercel-based stack reporting significantly better performance on live, personalized-data workloads. Harper recently released version 5.2, with a new record cache and more throughput per node. By Renato Losio

Last week I turned my data audit into a build step : a check that runs before anything else and fails the build when the database and any static copy of my travel site's legal-status data disagree. It ended the era of the site contradicting itself. It did nothing about the site agreeing with itself on something false. That's not a hypothetical. The most expensive error the whole project found was a country whose law changed in January while every copy on my site — database, data files, search index — kept saying the old thing in perfect unison. Internal consistency was the camouflage . No diff between my own sources could ever have caught it, because every internal source was equally behind the world. A build gate proves agreement. Agreement is not truth. Something has to look outside. You can't diff against the world, but you can sample it The naive version of "look outside" is another audit — a human session checking primary sources jurisdiction by jurisdiction. I've done three of those now, and I know exactly what they're worth: they're correct the day they ship and they decay from that morning on. Laws don't change on my audit schedule. So the outside check became what the inside check became: a scheduled job. Once a week, a script asks a web-connected model — one that searches and cites, not one answering from training memory — for the current legal status of about fourteen jurisdictions, and compares each answer to the corresponding database row. Fourteen, not all 271, because the selection is doing the real work: A hot list is checked every single run: the highest-traffic pages plus the jurisdictions with active legislative motion — the places where being a month stale costs the most. Everything else sits on a rotating cursor : eight per run, round-robin, so every row on the site gets sampled roughly twice a year without any run costing more than a few cents. The whole thing runs on about seven cents a week. Two rules were non-negotiable, both inherited from

@Service public class SimpleUserService { @Autowired private JooqTemplate jt ; public List < user > selectUserInDept ( UserParam param ) { //If deptIDs==null or deptIDs. isEmpty automatically ignores this query condition // SELECT * FROM user_table WHERE name LIKE '%?%' AND dept_id IN (?,?...); return jt . queryv ( "user_table" , User . class , "name%" , param . getName (), "dept_id:in" , param . getDeptIds ()); } public List < user > selectUserNotInDept ( UserParam param ) { // SELECT * FROM user_table WHERE name LIKE '%?%' AND dept_id NOT IN (?,?...); return jt . queryv ( "user_table" , User . class , "name%" , param . getName (), "dept_id:notin" , param . getDeptIds ()); } }

I have a set of React components that render live telemetry: an attitude indicator, a moving map, tapes and gauges, a scrolling event log. They take a data source, subscribe to it, and paint whatever numbers arrive. That works for a live feed. The obvious next thing you want is replay: load a recorded session, scrub a timeline, watch the same instruments play it back. The naive version of this is a trap, and it took me a wrong turn to see why. My first instinct was that replay is a data problem, load the samples, push them into the components in order, done. It compiled, it ran, and the charts were empty. Not broken, not erroring. Empty. The instruments that show a single current value worked fine. The time-series charts sat blank while correct data flowed into them. That empty chart is the whole story of this post, because the reason it's empty is the reason replay is more interesting than it looks. The components are watching a clock you forgot about Here's the data source interface these components consume. It's small on purpose: interface TelemetryValue { timestamp : number ; // wall-clock, unix ms value : number ; channel ?: string ; } interface AltaraDataSource { subscribe ( callback : ( value : TelemetryValue ) => void ): () => void ; getHistory (): TelemetryValue []; readonly status : ConnectionStatus ; destroy (): void ; } A live source stamps each sample with Date.now() as it arrives. A time-series chart, reasonably, assumes that's what timestamps mean: it anchors its x-axis to Date.now() and draws a moving window of the last few seconds, discarding anything older than windowMs because that's off the left edge of the view. Now replay a session recorded an hour ago. Every sample carries its original timestamp, an hour in the past. The chart buffers them correctly, then asks "is this within the last few seconds of now?", the answer is no for every single sample, and it draws nothing. The data is all there. It's just an hour to the left of the visible window,