标签：#Data

General Intuition has raised $320 million to scale AI trained on millions of hours of gameplay, betting action data can help AI develop something closer to human intuition.

Market research company Klue told customers that it believes the hacking group that stole their data is now deleting it. The company, however, warned about a second group of hackers wanting ransom.

We have tools for checking whether a query is injectable. We have linters, scanners, ORMs, parameterized queries, and database policies. But after the database returns rows, most applications simply trust that the result set matches the operation that asked for it. queryguard starts there. The query may be safe. The result may still be wrong. SQL injection taught us to distrust query construction. Parameterized queries answered the question: Did the user control the query structure? That question is well understood. The tooling is mature. But it is a different question from the one queryguard asks: Did this operation receive only the rows and fields it was allowed to receive? Those two questions are not the same. A perfectly safe parameterized query can still return the wrong row — because a predicate was dropped, a join widened the result, a developer selected a column they shouldn't have, or a query was rewritten without updating its scope contract. queryguard is not a database firewall. It is not a SQL injection scanner. It is not an ORM plugin. It is a contract check for observed result sets. Where it sits The hook position is the core design decision. queryguard sits immediately after cursor execution — before any result shaping, filtering, serialization, or response mapping. cursor = conn . execute ( sql , bindings ) rows = [ dict ( row ) for row in cursor . fetchall ()] evidence = queryguard . run_check ( contract , { " contract_id " : " user_profile_lookup " , " contract_version " : " 0.1.0 " , " params " : { " user_id " : user_id }, " session " : { " tenant_id " : tenant_id }, " result " : rows , }) if evidence [ " verdict " ] != " PASS " : raise QueryguardViolation ( evidence ) return rows Not at the HTTP layer. Not inside the ORM. Not at the API gateway. Immediately after the cursor returns rows — while the result is still raw, before anything shapes or discards it. This is intentional. If rows are shaped before queryguard sees them, queryguard cannot det

For three months, I weighed myself every morning and took body measurements every Sunday. I used a caliper, a tape measure, and a scale that probably lies to me about hydration levels. The goal wasn't to get ripped. It was to understand whether any of these measurements actually mean something day to day. The Problem With Most Health Calculators Most body fat calculators fall into one of two camps: Too simple — plug in height and weight, get a BMI number that tells you nothing about your actual composition. Too complicated — requires measurements you need a degree to take correctly, plus an email signup and a paid subscription. Neither is useful for someone who just wants to know "am I making progress?" Building Something Practical I put together a calculator that uses the Navy Method — it takes neck, waist, and hip measurements and estimates body fat percentage. The math has been around since the 80s and correlates reasonably well with DEXA scans for most people: function navyBodyFat ( gender , neck , waist , hip , height ) { if ( gender === ' male ' ) { return 86.010 * Math . log10 ( waist - neck ) - 70.041 * Math . log10 ( height ) + 36.76 } return 163.205 * Math . log10 ( waist + hip - neck ) - 97.684 * Math . log10 ( height ) - 78.387 } The inputs are simple enough that anyone can take them with a tape measure. The output gives you a ballpark number that's consistent enough to track trends over time. What 90 Days of Data Taught Me Three things stood out: Daily weight is useless; weekly trend is everything. My weight would swing 2-3 pounds daily due to water, food, and sleep. The weekly moving average was the only signal worth watching. Body fat percentage changes slowly. Like, frustratingly slowly. In 90 days of consistent training, I moved maybe 2%. But that's real — if a calculator tells you you dropped 5% body fat in a month, it's broken. Consistency beats precision. Taking measurements at the same time, under the same conditions, with the same method matter

IBM’s nanostack transistors could boost chip performance or energy efficiency.

Most developers think password hashing is about authentication. It's not. Authentication is just a side effect. Password hashing exists for a much darker reason: because databases get stolen. Every year, companies invest millions in firewalls, monitoring systems, cloud security, and access controls. Yet breach after breach continues to make headlines. The uncomfortable truth is that security teams don't assume a breach will never happen. They assume it eventually will. And when that day comes, one question determines whether the incident becomes a minor security event or a full-scale disaster: Did you hash the passwords? The Difference Between an Incident and a Catastrophe Imagine an attacker gains read access to your production database. Not a far-fetched scenario. A leaked backup. A vulnerable API. A compromised employee account. A misconfigured cloud bucket. The attacker runs a simple query: SELECT email , password FROM users ; If your system stores passwords in plain text, the breach is over. The attacker already won. No advanced techniques. No brute force. No expensive infrastructure. They now possess something more valuable than your database itself: your users' digital identities. Because users rarely reuse databases. They reuse passwords. The same password protecting an account on your platform might also unlock their Gmail, GitHub, LinkedIn, banking app, or company VPN. What started as your security problem instantly becomes everyone else's. This Is Not Theoretical History has repeatedly shown what happens when passwords are handled incorrectly. The RockYou breach exposed more than 30 million passwords stored in plain text. Attackers didn't need to crack anything. They simply read the data. Years later, those leaked passwords were still appearing in credential stuffing attacks across the internet. A single backend decision survived longer than the company itself. That's the thing about password leaks. They don't expire when the incident report is published.

Live demo: https://safestate.vercel.app , code: https://github.com/usv240/safestate A product recall today is basically a notice. It lives on a webpage, or a PDF, or an email that somebody is supposed to read. Say the problem out loud and it gets uncomfortable fast. A recalled crib can be listed and sold to another family, and nobody in that sale ever sees the recall. Reselling recalled goods is actually illegal, and recalled infant products have killed kids. I spent this hackathon building something to close that gap. I called it SafeState, and the idea is small: make the recall do something. When a second-hand item is listed or sold, the marketplace checks SafeState first, and recalled units get blocked right at checkout. It is precise down to the serial number, so safe units still sell. It runs on the stack this hackathon is about. A Next.js front end on Vercel, with Amazon Aurora DSQL behind it. Why DSQL is the whole point here The promise SafeState has to keep is this: the moment a recall lands in any region, no marketplace anywhere should ever read that product as "safe" again. That is a strong consistency problem, not a nice-to-have. If there is any window where a recalled product still looks safe, that is exactly when it gets sold. An eventually consistent store or a nightly sync leaves that window open. DSQL's active-active, multi-region setup with strong consistency is what closes it. I set up a real peered cluster across us-east-1 and us-east-2, with us-west-2 as the witness. Write a recall through one region's endpoint and you can read it back from the other region right away. There is a page in the app that lets you run that yourself. The one trick that makes it work DSQL runs on snapshot isolation (PostgreSQL REPEATABLE READ) with optimistic concurrency. It catches write-write conflicts at commit time. Snapshot isolation will not protect you from write skew, so I had to design around that. To guarantee that a recall and a sale of the same product actua

Today I launched Beach Day API , a developer API for real-time beach, ocean, water quality, advisory, amenity, access, and condition data. The goal is simple: make it easier for developers to build apps and tools around beach conditions without having to manually gather data from scattered sources. Beach Day API currently supports beaches across the United States and Australia , and returns structured JSON that can be used in travel apps, weather apps, surf tools, tourism websites, hotel and resort platforms, map-based search experiences, local discovery apps, and coastal safety dashboards. What the API includes Beach Day API can provide data such as: Beach profiles GPS and location data Ocean and weather conditions Water quality grades Advisories and closures Amenities Access details Beach-specific safety and visitor information A proprietary Beach Day Score The Beach Day Score is designed to give developers a fast way to surface whether a beach looks like a good choice for visitors on a given day. Why I built it Most weather APIs are broad. They can tell you temperature, wind, rain, or general conditions, but they usually do not answer the real user question: “Is this a good beach day?” That question depends on more than weather. It can involve water quality, advisories, closures, ocean conditions, amenities, beach access, and the actual visitor experience. Beach Day API is built around that more specific use case. Example use cases Some things developers could build with it: A beach finder app A surf or coastal conditions app A hotel or resort beach conditions widget A local tourism guide A travel planning tool A map-based beach discovery experience A safety dashboard for advisories and closures A recommendation engine for nearby beaches Built for simple integration The API uses API-key authentication and returns clean JSON responses. I wanted it to be straightforward enough that a developer could start testing quickly and then build it into a real product withou

Part 4 of the Building Enterprise AI Automation Systems Series Introduction Most Named Entity Recognition (NER) tutorials end with a prediction. The model successfully extracts: COMPANY INVOICE CONTRACT PURCHASE_ORDER The article ends. The notebook prints a beautiful JSON response. Mission accomplished. Or so it seems. In real enterprise systems, extracting entities is only the beginning. Consider the following prediction: { "COMPANY" : "ALPHABRIDGE" , "INVOICE" : "MFG-INV-000157" } At first glance, everything looks correct. But from a business perspective, the system still knows almost nothing. Questions remain unanswered. Which ALPHABRIDGE? Which customer record? Which contract? Which invoice? Which business relationship? These questions belong to a completely different problem known as Entity Resolution. Entity Resolution transforms extracted text into business knowledge. Without it, AI understands words but not businesses. NER Finds Text Named Entity Recognition answers one question: "What pieces of text represent meaningful entities?" For example: PAYMENT FROM ALPHABRIDGE SOLUTIONS MFG-INV-000157 becomes { "COMPANY" : "ALPHABRIDGE SOLUTIONS" , "INVOICE" : "MFG-INV-000157" } This is extraction. Nothing more. The model has no idea whether: the company exists, the invoice exists, the invoice belongs to the company, the invoice has already been paid, the contract is still active. Extraction is syntax. Enterprise automation requires semantics. The Hidden Problem Imagine the following customer master. CUS-00001 ALPHABRIDGE SOLUTIONS Now imagine receiving these transaction narratives. PAYMENT FROM ALPHABRIDGE PAYMENT FROM ALPHABRIDGE LTD PAYMENT FROM ABS PAYMENT FROM ALPHA BRIDGE Humans immediately recognize these as the same customer. Machines do not. To a computer, every string is different. Without resolution, automation immediately breaks. What Entity Resolution Actually Does Entity Resolution answers a different question. Instead of asking: "What entity is this?"

Apache Iceberg looked like the answer to everything when we first adopted it. Open format, ACID transactions, time travel, schema evolution. We migrated our Hive tables, ran a few queries, and felt good about life. Three months later, our S3 costs doubled. Queries that used to take 10 seconds were taking 4 minutes. Metadata operations were timing out. Nobody on the team could explain why. That was the beginning of a real education in how Iceberg actually behaves in production. This post covers what I wish someone had told us before we went all-in. The Small Files Problem Is Not Optional Iceberg is append-friendly by design. Every micro-batch write, every streaming insert, every incremental load creates new Parquet files. Each file also gets its own metadata entry. After a week of hourly loads, you might have 10,000 files in a single partition where you wanted 20. The result: Iceberg's metadata layer has to plan queries across thousands of file manifests. Planning takes longer than execution. Your 10-second query becomes a 4-minute query, and your users start filing tickets. Fix: automate compaction from day one. In Spark, compaction is called rewrite_data_files . The basic call looks like this: -- Run this on a schedule, not on-demand CALL iceberg_catalog . system . rewrite_data_files ( table => 'analytics.events' , strategy => 'binpack' , options => map ( 'target-file-size-bytes' , '134217728' , -- 128MB target per file 'min-input-files' , '5' -- only compact if 5+ small files exist ) ) Target file size of 128MB to 512MB is the practical sweet spot. Smaller than that, you still have too many files. Larger, and your query engines cannot parallelize reads efficiently. If you are not using Spark, PyIceberg exposes compaction through the table maintenance API (as of 0.7.x). For Flink or Trino-only shops, schedule compaction as a separate Spark job. Yes, it is annoying, but it is the right call. Hidden Partitioning Is the Feature You Are Probably Ignoring Old Hive parti

Part 3 of the Building Enterprise AI Automation Systems Series Introduction Named Entity Recognition (NER) is one of the oldest problems in Natural Language Processing. Most tutorials introduce NER using examples like: Person Organization Location Date A sentence such as: Elon Musk founded SpaceX in California. becomes PERSON ORGANIZATION LOCATION While this is useful for learning NLP fundamentals, it has very little relevance to enterprise software. Businesses do not automate biographies. They automate operations. Enterprise documents contain an entirely different language. Invoices. Contracts. Purchase Orders. Bank Statements. Remittance Advice. Payment Narratives. ERP Exports. The entities that matter inside these documents are not "PERSON" or "LOCATION". Instead, they are business concepts such as: Customer Contract Invoice Purchase Order Payment Type Understanding these entities is the first step toward intelligent automation. In this article, we'll build a Financial Named Entity Recognition pipeline capable of transforming raw enterprise transaction narratives into structured business knowledge. The Difference Between Generic NER and Enterprise NER Traditional NER focuses on linguistic entities. Enterprise NER focuses on operational entities. Consider the following sentence. PART PMT ALPHABRIDGE SOLUTIONS MFG-INV-000157 A generic language model may identify: Organization and ignore everything else. From a business perspective, this is almost useless. What we actually need is: PAYMENT_TYPE COMPANY INVOICE The objective is not language understanding. The objective is business understanding. Step 1 — Designing the Business Taxonomy Before training any model, define what the model should learn. This is one of the most overlooked stages in machine learning projects. Many teams immediately begin annotation without first defining a taxonomy. As a result, annotations become inconsistent. Models become confused. Evaluation becomes unreliable. For our transaction intell

Part 2 of the Building Enterprise AI Automation Systems Series Introduction One of the biggest obstacles in enterprise AI is not choosing a model. It is finding data. Most tutorials assume that training data already exists. Reality is very different. Large organizations rarely share operational datasets. Financial transactions contain confidential information. Contracts contain sensitive agreements. Invoices reveal commercial relationships. Bank statements expose customer activity. For legal, regulatory, and competitive reasons, these datasets almost never become public. This creates a difficult problem for AI engineers. How do you build intelligent systems when the data you need cannot be accessed? The answer is synthetic data. Unfortunately, most synthetic datasets found online are little more than randomly generated CSV files. They contain names. Numbers. Dates. But they completely ignore something far more important: Business relationships. In this article, we'll explore how to design synthetic enterprise datasets that preserve real business logic and can be used for machine learning, automation, benchmarking, and AI engineering. Random Data Is Not Synthetic Data Many developers believe synthetic data simply means generating fake values. For example: Customer,Invoice,Amount John,INV001,500 Alice,INV002,1200 Bob,INV003,900 Technically, this is synthetic. Practically, it is useless. Why? Because enterprise systems are built around relationships. Invoices belong to contracts. Contracts belong to customers. Payments reference invoices. Purchase orders authorize invoices. Bank transactions settle invoices. Without these relationships, there is nothing meaningful to learn. A machine learning model trained on isolated records learns isolated patterns. Real enterprise automation requires connected data. Thinking Like an Enterprise System Before writing a single line of Python, ask one question: "How does the business actually operate?" Imagine a manufacturing company. A

Google’s Search history update stores media uploads from your interactions, like images used in reverse image searches, for training its AI models.

Recently, we encountered a case on a RHEL 9 machine where an RPM query did not show any information about installed package even though the package files were present in the filesystem. Although RPM reported that the package was not installed, I was still able to run tcpdumpcommand and use it without any issues. The binary was present there in the filesystem and I was able to run it successfully. Fixing RPM Database Desynchronization on RHEL - bidhankhatri.com.np Recently, we encountered a case on a RHEL 9 machine where an RPM query did not show any information about installed package even though the package files were present in the filesystem. bidhankhatri.com.np

The 2026 FIFA World Cup is reshaping competitive balance in ways that traditional 32-team analysis cannot predict. With 16 groups of 3 teams instead of 8 groups of 4, the mathematical probability of upsets—and the consequences of single matches—has fundamentally shifted. Early tournament data already shows this pattern emerging. The Format Change: A Statistical Earthquake The move from 32 to 48 teams introduces a critical structural change: Format Groups Teams/Group Matches/Team Elimination Threshold 2022 (Qatar) 8 4 3 Top 2 of 4 2026 (USA/CAN/MEX) 16 3 2 Top 2 of 3 This seemingly small difference creates massive implications. In a 3-team group, each team plays only 2 matches to determine their fate . Compare this to the 2022 format where teams had 3 chances to secure advancement. The upset probability multiplier: With two fewer matches per group, variance compounds. A single bad result—or a fortunate one—carries exponentially more weight. Early Tournament Evidence: The Data Doesn't Lie Let's examine the first week of actual 2026 results: Match Expected Result Actual Result Upset Indicator Portugal 5-0 Uzbekistan Portugal W Portugal W (5-0) Expected England 0-0 Ghana England W Draw Minor Upset France 3-0 Iraq France W France W (3-0) Expected Argentina 2-0 Austria Argentina W Argentina W (2-0) Expected Norway 3-2 Senegal Senegal slight favorite Norway W Major Upset Jordan 1-2 Algeria Algeria strong favorite Competitive Closer than xG Panama 0-1 Croatia Croatia W Croatia W Expected Colombia 1-0 Congo DR Colombia W Colombia W Expected Three critical takeaways: Norway's 3-2 victory over Senegal is statistically significant. Pre-tournament models favored Senegal slightly (ranked 18th globally vs Norway's 22nd). In a 4-team group, this result matters less; in a 3-team group, Norway essentially secures qualification with one match remaining. England's 0-0 with Ghana represents draw probability explosion. With only 2 group matches, a draw consumes 50% of your advancement op

Jump Tables, TLV Footers, and the Real Cost of Reading What You Don't Need You're paying for bytes you never read. A data engineer on a busy pipeline touches dozens of Parquet files a day: schema discovery, predicate pushdown, column pruning, metadata scrapes for a data catalog sync. In each case, the application needs maybe 200 KB of context from a file that is 4 GB on disk. Without a seekable archive format and a jump table to find the right frame, your HTTP client fetches the whole thing, and your cloud egress invoice reflects every unnecessary gigabyte. This post quantifies the problem, then walks through how HuskHoard uses seekable Zstd frames, a per-volume jump table, and TLV-encoded footer metadata to make partial egress a first-class citizen across multi-volume archives — disk, cloud, and LTO tape alike. The Problem, In Dollars S3 standard egress runs $0.09/GB. GCS is $0.08/GB. Even Cloudflare R2, which is free for egress from R2 to the internet , still costs you in latency and API call count when you cannot bound the range of bytes you need. Here is a representative read pattern for a cold analytics archive: Operation Bytes Needed Bytes Fetched (naïve) Ratio Schema discovery ~50 KB (Parquet footer) 1–8 GB (full file) ~1:16,000 Single column scan ~200 MB (one column chunk) 4 GB (full row group) 1:20 Data catalog sync (1M files) ~50 GB (footers only) ~4 PB (full files) 1:80,000 Selective restore (1 row group) ~128 MB 4 GB 1:32 On 100 TB of cold Parquet data with $0.09/GB egress: Full read for schema sync : 100 TB × $0.09 = $9,216 Partial read (footers only, avg 100 KB/file, 1M files) : ~100 GB × $0.09 = $9.00 Savings per catalog sync: $9,207 — 99.9% reduction Even a conservative column-scan scenario (pulling 15% of each file's bytes) cuts a $9,216 monthly read bill to $1,382 . The ceiling on savings is determined entirely by how precisely you can address the bytes you actually need. That precision is what frames and jump tables buy you. Zstd Frames: What They

Naomi Saphra discusses 5 rules governing language model behavior, breaking down why LLMs act like populations rather than individuals. She explains how tokenization creates strange semantic blind spots and highlights the mechanics of sycophancy, showing how models leverage subtle data associations to match user biases and demographics - even guessing political views based on favorite sports teams. By Naomi Saphra

Building point-in-time correct, production-grade feature pipelines — from raw Kafka events to online feature serving in milliseconds, using Spark Structured Streaming and the Databricks Feature Store. Table of Contents The Feature Engineering Problem Architecture Overview Feature Store Concepts: ERD Environment Setup Streaming Feature Pipeline Point-in-Time Correct Training Dataset Generation Writing Features to the Online Store Serving Features at Inference Time Feature Table Reference References The Feature Engineering Problem Feature engineering is where most ML projects silently fail in production. Not because the model is wrong — but because the features the model sees at training time are different from the features it sees at inference time . This is called training-serving skew , and it's the #1 silent killer of ML systems. Three specific failure modes cause it: Online/offline inconsistency — the batch pipeline that computes training features uses different logic than the real-time service that computes inference features Data leakage — training features accidentally include information from the future (e.g. joining on a label that was created after the event) Feature staleness — a model trained on 30-day rolling averages is served features that are 6 hours stale because the pipeline backfills are slow The Databricks Feature Store — now part of Unity Catalog as Feature Engineering in Unity Catalog — solves all three by: Storing feature computation logic alongside the data (no drift between training and serving) Enforcing point-in-time lookups during training dataset creation Providing a unified API for both batch offline reads and low-latency online reads Architecture Overview Feature Store Concepts: ERD Understanding the data model behind the Feature Store is essential for designing correct pipelines. Here's how the entities relate: The critical relationship: a Model Version is bound to a Training Set , which records exactly which feature tables and which p

While working on a MERN project, I came across these indexes: transactionSchema . index ({ user : 1 , date : - 1 }); transactionSchema . index ({ user : 1 , type : - 1 }); transactionSchema . index ({ user : 1 , category : - 1 }); My first reaction was: "Why are we creating 3 different indexes for the same schema? Isn't one index enough?" At that time, my understanding was: "Indexes help MongoDB find records faster." Which is true, but it wasn't enough to explain why multiple indexes existed for the same collection. That simple doubt led me down a rabbit hole of learning about indexes, compound indexes, how MongoDB stores them, and the famous Prefix Rule. Here's what I learned. What is an Index? Imagine a collection with millions of transactions. db . transactions . find ({ user : " Aarthi " }); Without an index, MongoDB may need to inspect every document until it finds the matching records. This is called a Collection Scan . Think of it like searching for a chapter in a book without a table of contents. You'd have to flip through page after page until you find it. An index works like a book's table of contents. Instead of scanning every document, MongoDB can jump directly to the relevant records. Example: db . transactions . createIndex ({ user : 1 }); Now MongoDB can quickly locate all transactions belonging to a specific user. What is a Compound Index? A compound index contains multiple fields. Example: db . transactions . createIndex ({ user : 1 , date : - 1 }); This means MongoDB organizes the index by: user └── date Conceptually, it looks something like: Aarthi 2025-08-10 2025-08-09 2025-08-08 John 2025-08-10 2025-08-05 The data is first grouped by user , and within each user, it is ordered by date . Now queries like: db . transactions . find ({ user : " Aarthi " }). sort ({ date : - 1 }); become very efficient. MongoDB can jump directly to Aarthi's records and retrieve them in date order. The Prefix Rule: The Concept That Finally Made It Click Consider this i

A deep dive into how Spark transforms your SQL into a physical execution plan — and how Databricks layers Adaptive Query Execution and the Photon vectorized engine on top to squeeze out maximum performance. Table of Contents Why Query Optimization Matters The Catalyst Optimizer Pipeline Stage 1: Parsing — From SQL to Unresolved Logical Plan Stage 2: Analysis — Binding to the Catalog Stage 3: Logical Optimization — Rule-Based Rewrites Stage 4: Physical Planning — Strategies and Cost Models Adaptive Query Execution (AQE) The Photon Engine Reading Explain Plans Tuning Reference Table References Why Query Optimization Matters A Spark query written by a human and a Spark query executed by the engine are often very different things. The gap between them — the optimization — is what separates a job that runs in 3 minutes from one that runs in 3 hours on identical hardware. Databricks compounds Spark's native Catalyst optimizer with two additional layers: Adaptive Query Execution (AQE) — re-optimizes the query at runtime using actual statistics collected mid-job Photon — a C++ vectorized execution engine that replaces the JVM-based Spark executor for eligible operators Understanding all three lets you write queries that cooperate with the engine rather than fight it. The Catalyst Optimizer Pipeline Catalyst is Spark's rule-based and cost-based query optimizer. Every query — whether written in SQL, DataFrame API, or Dataset API — passes through the same four-stage pipeline before a single byte of data is read. Stage 1: Parsing — From SQL to Unresolved Logical Plan # ── Catalyst Stage 1: Parsing ───────────────────────────────────────────────── # Spark uses ANTLR4 to parse SQL into an Abstract Syntax Tree (AST). # At this point column names are NOT validated — the plan is "unresolved". from pyspark.sql import SparkSession spark = SparkSession . builder . appName ( " catalyst-demo " ). getOrCreate () # Both of these produce identical internal representations df_api = ( spark .