标签：#Data

A generated image looks finished until review starts. Someone approves the first version. Someone else crops it. A branded copy goes out. Another edit changes the prompt. A week later, the useful question is simple: which prompt, model, seed, size, parent image, and publishing settings produced the version on screen? In a content studio, I put those answers in the PostgreSQL row that stores the image. Logs explain what happened during a run, then rotate away. Object storage keeps the bytes and forgets why they exist. The row is the only one of the three that survives edits, review, and publishing. 1. The row is the receipt The table in apps/api/src/database/init-ai-images-table.js treats generated and edited images as one record type. An original image gets its own row. An edit gets another row, with original_image_id pointing back to the parent. CREATE TABLE IF NOT EXISTS ai_generated_images ( id SERIAL PRIMARY KEY , image_url TEXT NOT NULL , -- what produced it prompt TEXT NOT NULL , model VARCHAR ( 100 ) DEFAULT 'fal-ai/imagen4' , model_version VARCHAR ( 100 ), seed BIGINT , width INTEGER , height INTEGER , -- how it derives from another row is_edited BOOLEAN DEFAULT FALSE , original_image_id INTEGER REFERENCES ai_generated_images ( id ), edit_prompt TEXT , edit_strength DECIMAL ( 3 , 2 ), -- what actually shipped branded_url TEXT , branding_options JSONB , metadata JSONB , tags TEXT [], created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP ); That self-reference is the design choice. It makes the image table append-only-ish: new variants are inserted as new rows instead of overwriting the earlier state. The cost is more rows and more discipline at write time. The benefit is editable history that product screens and debugging queries can follow. flowchart TD original["Original row: prompt, model, seed, dimensions"] editA["Edited child: edit_prompt, edit_strength, edit_steps"] editB["Edited child: edit_prompt, edit_guidance_scale"] brandedA["Branded output: branded_url,

100 城时区页给跨区调度当速查，DST 自动算 作者是 数据管道 / 跨时区调度 方向的开发者。这篇不是广告，是踩坑记录 + 顺手做的工具。 背景 做 数据管道 / 跨时区调度 时，时间戳转换是最常被低估的雷区。16 个时间戳工具(Unix 转换/时区/ISO8601/Cron/Duration…) 已覆盖日常；但每个语言/框架的坑都不一样，所以又补了 30 个语言/框架时间戳页(python/javascript/java/sql/…)，每页含 6 个真实坑。 我踩过的坑（举几个） 秒 vs 毫秒：前端 Date.now() 是毫秒，后端常存秒，混用差 1000 倍。 时区不是字符串：存 UTC、展示本地，别把本地时间当 UTC 落库。 2038 问题：32 位系统 time_t 在 2038-01-19 溢出，老系统要提前查。 夏令时：一年有两次重复/缺失的本地时间，跨区调度尤其坑。 我顺手做的东西 转换速查页： https://gotimestamp.com/timezone/new-york 相关语言页： https://gotimestamp.com/timezone/london 开源 MCP： https://github.com/caresotin/tsforge-mcp —— 把时间戳转换/校验直接接进 LLM 工作流，不用手算。 小结 时间戳没那么简单，但工具到位就省心。上面都是免费、开源、可直接用的，希望对同样踩坑的人有帮助。

OpenAI and Anthropic admitted that their unreleased AI models escaped their sandboxes and hacked several companies in unprecedented cyberattacks. Who is legally to blame? Should prosecutors charge the two AI frontier labs? Can victims sue them? We spoke to lawyers who specialize in computer hacking laws to find out.

In a recent Azure Architecture blog article, Azure lead engineer Kishorekumar Pattabiraman outlines practical criteria for choosing between skills, sub-agents, and other approaches when building AI systems, emphasizing reusability, simplicity, and long-term maintainability. By Sergio De Simone

Apple has appealed a new legal demand by the U.K. government, which critics say could threaten the privacy rights of users all over the world.

Half the market is arguing about whether RAG or a semantic layer is the right foundation for enterprise AI. They are not competing. They answer different questions, and most teams need both. Two shapes of question Every question an agent receives breaks into one of two forms: "What did we say about X?" — lives in contracts, policies, tickets, docs. Unstructured. RAG was built for this. "What is true about X?" — lives in your warehouse and governed metrics. Structured. A semantic layer was built for this. Treating them as rivals is how teams end up with a system that can quote the pricing policy but cannot tell you this quarter's realised price. Where each one breaks RAG Semantic layer Good at Retrieving relevant prose Resolving definitions and joins Fails on Aggregation, math, current state Anything not modelled as data Permissions Flattened at ingest, rebuilt at query time Compiled per person, per query Answer stability Varies with retrieval ranking Identical by construction Audit story Cites a chunk Reproduces the exact SQL The permissions row is the one that ends pilots. A retrieval index that ingested everything has, by construction, assembled your most sensitive object — and reconstructing entitlement at query time is guesswork. The layer that actually decides Neither a document chunk nor a metric definition is worth much until something compiles it into a governed query and runs it. That is the piece most architectures are missing: intent → context resolution → constrained planning → governed execution . RAG can feed the first step. It cannot perform the last three. Point an agent at raw tables and the best models score in the low teens on real enterprise data. Give the same model compiled, governed context and it clears the high nineties. The retrieval quality was never the bottleneck. The full breakdown — the precise division of labour, why hybrid architectures win, and how compile-time governance closes the gap RAG cannot — is here: 👉 RAG vs. Semantic Layer

Data pipeline automation has moved from a technical aspiration to a business imperative. As organisations deploy more AI agents that need fresh, reliable data, the volume and complexity of data pipelines has grown beyond what manual management can sustain. AI-powered pipeline automation — using AI to build, monitor, and repair data pipelines — is the emerging solution. Key Insight: AI-automated data pipelines reduce pipeline development time by 60%, decrease pipeline failures by 45%, and enable data teams to manage 3x more pipelines per engineer compared to manual approaches. The Pipeline Scaling Challenge The average enterprise now maintains 1,500+ data pipelines, according to research by Barracuda Networks, and this number is growing 25% annually as organisations add new data sources, new AI use cases, and new reporting requirements. Each pipeline has an average of 4.2 transformation steps, 2.1 data quality checks, and connects an average of 2.8 systems. The total pipeline infrastructure is complex, fragile, and increasingly beyond the capacity of manual management. The scaling challenge manifests in three ways. First, pipeline development backlog — the average data team has a 3-6 month backlog of pipeline requests from business users. Second, pipeline failures — the average enterprise experiences 15-20 pipeline failures per week, each requiring manual investigation and repair that consumes 30-40% of the data engineering team's capacity. Third, pipeline maintenance — as source systems change (schema updates, API modifications, deprecated fields), pipelines break silently and produce incorrect results until someone notices. This 'silent failure' problem is particularly dangerous because it erodes trust in data without anyone being aware that anything is wrong. The root cause of these challenges is that data pipelines have been built as static, manually-maintained infrastructure. A pipeline is coded, tested, and deployed. When the source or destination changes, a hu

Vector databases have become the backbone of AI applications, powering semantic search, RAG systems, recommendation engines, and multi-modal AI. With the market maturing rapidly in 2026, choosing the right vector database impacts everything from query latency to operational costs. We ranked the 8 best options based on performance, scalability, ease of use, and enterprise readiness. TL;DR: Ranked 8 best vector databases: Pinecone leads for managed simplicity, Weaviate for flexibility, Milvus for open-source scale, and pgvector for PostgreSQL-native teams. Selection depends on your scale, latency requirements, and existing infrastructure. Key Evaluation Criteria for Vector Databases Modern vector databases must excel across multiple dimensions. We evaluated each option on: query performance (latency at various scales, index types supported), scalability (horizontal scaling, multi-tenancy), data type support (dense vectors, sparse vectors, multi-modal embeddings), integration ecosystem (SDKs, LangChain/LlamaIndex support), and operational maturity (hosting options, backup, monitoring). Query performance: P95 latency at 1M, 10M, and 100M vector scales Scalability: Horizontal scaling, sharding, multi-tenancy Data type support: Dense vectors, sparse vectors, binary vectors, multi-modal Integration: SDK availability, LLM framework support, MCP compatibility Operations: Self-hosted vs. managed, backup, monitoring, compliance Ranking: The 8 Best Vector Databases for AI Applications ### 1. Pinecone Pinecone remains the most popular fully-managed vector database, known for its simplicity and reliability. The 2026 release adds sparse-dense hybrid search, serverless tier with sub-millisecond P99 latency, and namespace-based multi-tenancy. Its serverless pricing model makes it cost-effective for variable workloads. * **Best for:** Teams wanting fully managed vector search without operational overhead * **Pros:** Zero operations, excellent performance, simple API, strong ecosystem

Continuando com a saga de siglas, encontrei de maneira simplista a versão oposta do ACID, o BASE....

User segmentation requirement Imagine you need to send a push notification to users who satisfy all of the following conditions: Push notification is enabled User is a VIP Active within the last 30 days Following the Voucher Hot category The traditional approach is to query multiple tables: SELECT DISTINCT u . id FROM users u JOIN user_configs c ON c . user_id = u . id JOIN devices d ON d . user_id = u . id JOIN follows f ON f . user_id = u . id WHERE c . push_optin = 1 AND c . mute = 0 AND d . fcm_token IS NOT NULL AND f . category = 'voucher_hot' AND u . last_active >= NOW () - INTERVAL 30 DAY ; As your user base grows into the millions, every campaign requires joining multiple large tables, filtering millions of records, and repeatedly computing the same audience. Query latency increases significantly, making real-time segmentation increasingly difficult. A Different Approach Instead of querying the database every time, we precompute each boolean attribute as a bitmap. Think of a bitmap as a huge array containing only 0 and 1, where the index corresponds to the user ID. For example, a bitmap representing whether a user has enabled push notifications: User ID : 0 1 2 3 4 5 6 7 Bitmap : 1 0 1 1 0 0 1 1 To check whether user 123 has enabled notifications, simply read bit 123. 1 → enabled 0 → disabled Each bitmap represents exactly one boolean property: bitmap:push_optin bitmap:vip bitmap:active30 bitmap:follow:voucher_hot Memory Usage Bitmap is extremely memory efficient. Each user requires only one bit. For 1 million users: 1.000.000 bits ~ 125.000 bytes ~ 122 KB That means every segment only consumes about 122 KB of Redis memory. Even 100 different segments require only around 12 MB . Finding Intersections Suppose you want all users that are: VIP AND Push Opt-in AND Active30 AND Following Voucher Hot Redis can calculate the result with a single command: BITOP AND result vip push_optin active30 follow_voucher_hot Need the number of matched users? BITCOUNT result No

If you've ever tried to buy a portable power station, you know the problem: every brand claims to be the best, the spec sheets are a wall of numbers, and the forums are full of confident but contradictory advice. "What size do I actually need?" is the most-asked question and the least-clearly-answered. So I did the boring thing. I built a structured database of 102 portable power stations from 24 brands — capacity, output, chemistry, cycle life, solar input, weight, price — and started running the numbers. A few findings were genuinely surprising. 1. Prices quietly collapsed The median portable power station now sits at $0.61 per watt-hour . The cheapest in the dataset is $0.39/Wh (the GRECELL T1000). A few years ago, ~$1/Wh was normal and anything under $0.70 felt like a deal. The practical takeaway: if you're paying much more than ~$0.70/Wh in 2026, you're mostly paying a brand premium. That premium sometimes buys you a better app, ecosystem, or support — but it's worth knowing you're paying it. 2. LiFePO4 basically won 94% of the models I track now use LiFePO4 (lithium iron phosphate) instead of the older lithium-ion (NMC) chemistry. This matters more than any marketing bullet point: LiFePO4: ~3,000–4,000+ charge cycles Older Li-ion (NMC): ~500 cycles At daily use — say you cycle it every day in a van or for backup — that's roughly 8 years vs 18 months before the battery is meaningfully degraded. If a listing still uses NMC to hit a lower price, that "deal" can cost you far more over its life. 3. The fridge myth costs people hundreds of dollars This is the single most common sizing mistake. People size a giant, expensive battery to run a fridge because they do the math like this: Fridge nameplate (150W) × 24 hours = 3,600 Wh/day But a fridge's compressor only runs about 40% of the time . Its real average draw is closer to 60W, so: 150W × 40% × 24h ≈ 1,440 Wh/day In runtime terms: a 1,000Wh power station runs a full-size fridge for about 14 hours , not the ~6 hour

Arun Joseph shares real-world insights on scaling enterprise agentic platforms like Deutsche Telekom’s LMOS. He discusses bridging organizational fault lines, replacing tool sprawl with core platform abstractions, and moving beyond basic chatbots to operational intelligence systems through ephemeral agents and an Agent Definition Language (ADL). By Arun Joseph

Embabel has reached its 1.0 release, providing a framework for AI agents on Java It allows Java and Kotlin developers to define agents as typed domain objects. Built on Spring AI, Embabel supports multiple model providers and combines planning with predefined state machines, offering flexibility for agent workflows. By Erik Costlow

In Parts 1-4, we built a transactional key-value store. It has WAL for durability, memtables and SSTables for storage, compaction to control file growth, and transactions for atomic multi-key writes. Now we move to the query layer where users can actually fire SQL queries like: CREATE TABLE payments ( amount INT , id STRING , status STRING , captured BOOL , PRIMARY KEY ( id )) INSERT INTO payments VALUES ( 500 , payment_1 , pending , 1 ) SELECT * FROM payments WHERE id = payment_1 In this blog and the next one we answer: How do we translate SQL strings into operations our key-value store already understands? This post focuses on the first half of that bridge, which is parsing SQL into structured commands. In the next post, we will take those commands and turn CREATE TABLE and INSERT into bytes on disk. The Core Idea: SQL Becomes Structured Data The storage engine does not understand SQL. It understands keys, values, WAL entries, memtables, SSTables, and transactions. So the SQL layer has two jobs: Parse a human-readable SQL string into a structured object. Translate that structured object into key-value operations. For example: CREATE TABLE payments (...) -> CreateTable{TableName: "payments", ColumnDetails: ...} INSERT INTO payments VALUES (...) -> InsertIntoTable{TableName: "payments", ColumnValues: ...} SELECT * FROM payments WHERE id = payment_1 -> SelectFromTable{TableName: "payments", QueryConditions: ...} Once we have these structs, the rest of the database can stop dealing with raw strings. Why Not Parse SQL Directly in the DB Layer? Imagine if db.CreateTable() directly walked through the SQL string and also updated storage. That would mix two very different responsibilities: parsing grammar, executing database operations. Keeping them separate makes the system easier to reason about. The parser validates syntax and builds an AST. The DB layer receives that AST and decides what to store. An AST, or Abstract Syntax Tree, is just a structured representation of

The lost-update bug that quietly corrupts homegrown wallet balances — and the five disciplines we used across PayWithToken to make money movement correct under concurrency. There is a bug that lives in a large share of the world's homegrown wallet systems. It doesn't throw an error. It doesn't show up in tests. It surfaces months later as a balance that is quietly, inexplicably wrong — and in a payments system, a wrong balance is either a customer who has lost money or a company that has given it away. This is the story of that bug, why the "obvious" wallet code causes it, and the handful of disciplines we used across PayWithToken to make money movement correct under concurrency. The bug: lost updates Here is wallet code almost everyone writes first. Credit a user's balance: // DON'T do this $row = $db->query("SELECT balance FROM users WHERE id = $id")->fetch(); $new = $row['balance'] + $amount; $db->exec("UPDATE users SET balance = $new WHERE id = $id"); Read the balance, add to it in PHP, write it back. It works perfectly — until two things happen at the same time. Picture a wallet at ₦1,000. Two credits of ₦500 arrive simultaneously — say a bank webhook and the user tapping "confirm" on their phone: Request A reads balance = 1000. Request B reads balance = 1000 (A hasn't written yet). A computes 1500, writes 1500. B computes 1500, writes 1500. Two credits landed; the balance rose by ₦500. ₦500 vanished. This is a lost update, and it is a race condition, which means it is invisible until you have real concurrent traffic — exactly when you can least afford it. The debit version of the same bug lets a balance go negative or double-spends a token. Fix #1: let the database do the arithmetic The read-modify-write happened in PHP, across three round trips, with a gap where another request could interleave. The fix is to make the update a single atomic statement and let the database's row lock serialise it: // DO this — one atomic statement $db->prepare("UPDATE users SET

That's a rebuild I shipped. The system: a RAG assistant for fraud analysts — ask it "how do we handle card testing followed by a successful auth?" and it should answer from the team's own SOPs and case history. The complaint: the answers were wrong, therefore the model must be dumb, therefore procurement should buy a bigger model. The model was fine. It was answering perfectly — from garbage context. Walk the forensic trail with me, because every step is checkable on your own system this week. Exhibit A: the chunking was destroying meaning before anything was embedded The ingestion split SOP documents every N characters, mid-sentence. Which means half the vectors in the index encoded fragments like this: chunk_147 = " ...ing to a freight forwarder. In these cases, do NOT " chunk_148 = " cancel the order immediately. First verify the customer via " The policy — don't cancel, verify first — exists in no single chunk. An embedding can't encode a meaning that isn't in its input. Retrieval was being asked to find semantics the pipeline had already shredded. Fix one: chunk on structure (sections, paragraphs), never on character counts, with enough overlap that no rule straddles a boundary. Exhibit B: dense-only retrieval, bimodal queries Fraud analyst queries split into two populations: pattern questions ("high-value order, new account, rushed shipping") and identifier questions ("what's the SOP for decline code 4863?", "rule VEL-013 rationale"). The system was dense-only — and embeddings treat a rare token like 4863 as noise, so identifier queries retrieved similar-feeling chunks instead of the literal match. Half the query population was structurally doomed regardless of model quality. Fix two: hybrid retrieval — BM25 for the identifiers, embeddings for the patterns, reciprocal rank fusion to merge. Exhibit C: nobody could see any of this, because quality was a rumor No golden dataset. No retrieval metric. The system's accuracy was whatever the loudest anecdote said it

We live in an era where our wrists track every heartbeat, step, and sleep cycle. Yet, most of this "Quantified Self" data sits rotting in massive .xml or .json export files that are impossible to read. What if you could simply ask your AI, "How did my resting heart rate trend during the week I was stressed about the product launch?" In this tutorial, we are building a Quantified Self RAG (Retrieval-Augmented Generation) pipeline . We will take fragmented health data from Apple HealthKit and Google Health Connect, process it using DuckDB , and vectorize it into Pinecone using LangChain . By the end of this guide, you’ll have a production-grade Health Data RAG system capable of high-performance natural language queries over your personal biometrics. The Architecture: From Raw Logs to Vector Insights Handling health data at scale requires a robust ETL (Extract, Transform, Load) process. Vectorizing every single heart rate measurement (which can occur every few seconds) is inefficient and expensive. We need to downsample and summarize before embedding. graph TD A[Apple Health/Google Health] -->|Export XML/JSON| B[Raw Data Storage] B --> C{DuckDB Processing} C -->|Cleaning & Downsampling| D[Structured Parquet/JSON] D --> E[LangChain Document Loader] E --> F[OpenAI Embeddings] F --> G[Pinecone Vector Database] H[User: 'Why was my sleep poor last Tuesday?'] --> I[LangChain RAG Chain] G --> I I --> J[LLM Contextual Answer] Prerequisites 🛠️ To follow along, you'll need: Python 3.10+ Tech Stack : Pinecone , LangChain , DuckDB , OpenAI , and Pandas . An export of your health data (Apple Health export.xml or Google Takeout). Step 1: Efficient Data Crunching with DuckDB Apple Health exports are notoriously large XML files. Loading them directly into memory with standard Python is a recipe for a crash. We use DuckDB for its blazing-fast analytical capabilities to filter and downsample our data. import duckdb # Load and parse the XML (simplified logic) # Note: In a real scenario,

From the outside, social features seem like something straightforward: follow a user, like a post, see a feed, but when you attempt to implement them at any real scale you find that every single one is something you've got to be aware of, a pattern, a well-documented query pattern with all its failure modes. Here's a step-by-step look at four of them: adjacency list, fan-out feed, mutual-friends join, and some graph traversal that SQL wasn't really designed for. The adjacency list and why "who do I follow" isn't free A follow relationship is usually modeled as a plain adjacency table: follows(follower_id, followee_id, created_at) . That's the whole schema, and it's deceptively adequate for a long time. The first place it breaks is when you need "who do I follow that also follows them", mutual connections, because that's a self-join against the same table: SELECT f2 . followee_id FROM follows f1 JOIN follows f2 ON f1 . followee_id = f2 . follower_id WHERE f1 . follower_id = : user_id AND f2 . followee_id != : user_id ; This query is ok if the fan-out is low. It feels like it's no longer fine when a few accounts have a few hundred thousand joins, because join now needs to walk a correspondingly large intermediate result set per request. The typical solution isn't a better query, it's a better index and a cap. composite indexes on (follower_id, followee_id) and (followee_id, follower_id) so both directions of the join hit an index-only scan, plus a LIMIT applied early so the planner doesn't materialize more rows than the response will ever use. The fan-out feed problem The “social systems” hard problem is the timeline: display to me my feed of what people I follow posted recently, in order. There are two ways to construct it and they are both right - anything else and outages happen. Fan-out on write means that when a user posts, you insert a row into every follower's feed table immediately. Reads are then a trivial SELECT * FROM feed WHERE user_id = :id ORDER BY creat

Olá, comunidade DEV! Meu nome é Beth Rodrigues e este é o meu primeiro post por aqui. Durante muito tempo, minha relação com tecnologia esteve ligada ao uso das ferramentas no dia a dia de trabalho. Planilhas, sistemas, relatórios e processos faziam parte da minha rotina, mas em algum momento comecei a enxergar algo diferente: por trás dos dados existiam histórias, oportunidades de melhoria e decisões que poderiam ser mais inteligentes. Foi assim que comecei minha transição para a área de Dados. Minha jornada está sendo construída passo a passo: estudando SQL, Excel, Power BI, Python e conceitos de análise de dados. Também tenho explorado ferramentas de inteligência artificial, automações e formas de transformar problemas reais em soluções. Uma das coisas que mais me motivam é perceber que dados não são apenas números em uma tabela. Eles podem ajudar uma empresa a entender seus clientes, melhorar processos e encontrar caminhos mais eficientes. Atualmente estou criando pequenos projetos para praticar, aprender e construir meu portfólio. Quero compartilhar por aqui meus aprendizados, erros, descobertas e experimentos nessa caminhada. Acredito muito que aprender tecnologia é como montar um grande quebra-cabeça: no começo algumas peças parecem não fazer sentido, mas aos poucos a imagem começa a aparecer. Espero trocar experiências com pessoas da comunidade, aprender com quem já está nessa estrada e também contribuir compartilhando minha evolução. Que venha essa nova fase! 🚀

Originally published at nlqdb.com/blog The standard agent-memory build is an afternoon of work: embed every fact worth keeping, upsert it into a vector store, and before each reply pull the top-k most similar memories back into context. And for what it's built for, it works. Ask "what did this user say about the Berlin migration" and the right snippets come back, ranked by cosine distance. Recall is solved enough that it feels like memory is solved. Then the agent has been running for a month, and you ask its memory a different kind of question: "how many users asked about pricing this month?" "Average deal size per stage?" "Top 10 topics I logged, ranked by count?" The store dutifully returns the twenty memories most similar to the question text , the LLM eyeballs them, and you get a confident, specific, wrong number. Recall is similarity. Reporting is aggregation. Nothing malfunctioned — the two questions want different machines. A vector store's primitive is nearest-neighbour search: embed the query, rank stored vectors by distance, return the top-k, optionally narrowed by a metadata filter. That is the whole contract. There is no COUNT , no GROUP BY , no JOIN , no HAVING — a similarity engine ships no query planner, and even the metadata filter only narrows candidates around the approximate search, so what comes back is still a ranking of similar items, never a computed result set. "How many" has to touch every matching row . If the agent logged 4,000 memories and top-k is 20, the context the LLM sees is structurally incapable of producing the count — and an LLM doing arithmetic over a retrieved sample is a hallucination generator, not a query engine. The failure is quiet, too: the answer arrives fluent and plausible, and nothing flags that it was computed from half a percent of the data. -- "top topics this month, ranked by count" is not a similarity query. -- It's this — and it must scan every matching row, not the top-k: SELECT topic , count ( * ) AS mentions