标签：#data

What is nbwipers? nbwipers is a CLI tool that strips outputs and metadata from Jupyter notebooks before git commit. Written in Rust - faster than nbstripout Supports git clean filter Works with .ipynb files Why use it? Jupyter notebooks store cell outputs inside the .ipynb file (JSON). This causes problems: Noisy diffs - output changes pollute every commit Repo size - images and large outputs bloat the repo Security - sensitive data can leak in outputs (API keys, query results) The solution: strip outputs automatically on git add via a clean filter. Why not nbstripout? nbstripout is written in Python. It is slow - git status , git diff , and git add all became noticeably slow on this repo because nbstripout was invoked for every .ipynb file. The main cause is Python startup time. With 100+ notebooks, nbstripout can take 40+ seconds where a Rust-based tool takes ~1 second. Faster alternatives: Tool Language Notes nbstripout-fast Rust Up to 200x faster; no git filter install support nbwipers Rust Inspired by nbstripout-fast; adds git filter + pyproject.toml config nbwipers is essentially nbstripout-fast with better git integration. Switching to nbwipers fixed the slowness. Setup 1. Install felixgwilliams/nbwipers is now in the aqua registry as of v4.517.0 . Using aqua , add to aqua.yaml : packages : - name : felixgwilliams/nbwipers@v0.6.2 Then run: aqua install 2. Configure git filter Run once per repo (writes to .git/config ): git config filter.nbwipers.clean "nbwipers clean -" git config filter.nbwipers.smudge cat git config filter.nbwipers.required true Or edit .git/config directly: [filter "nbwipers"] clean = nbwipers clean - smudge = cat required = true required = true makes the commit fail if nbwipers is not installed. This prevents accidentally committing outputs. 3. Add .gitattributes In the repo root, add .gitattributes : *.ipynb filter=nbwipers **/.ipynb_checkpoints/*.ipynb !filter **/.virtual_documents/*.ipynb !filter The !filter lines exclude checkpoint an

DuckDB 1.5.3 Iceberg updates, PostgreSQL TDE extension & AI index tuning Today's Highlights Today's highlights include DuckDB's enhanced Iceberg integration with new DML and schema evolution features, alongside a deep dive into PostgreSQL's new open-source Transparent Data Encryption. Additionally, we explore AI-driven strategies for automating PostgreSQL index tuning, offering practical performance improvements. New DuckDB-Iceberg Features in v1.5.3 (DuckDB Blog) Source: https://duckdb.org/2026/05/29/new-iceberg-features.html The latest DuckDB v1.5.3 release introduces significant enhancements for working with Apache Iceberg tables, a critical component in modern data lake architectures. Key additions include full MERGE INTO support, allowing users to efficiently update, insert, and delete rows in Iceberg tables based on a source query. This release also brings ALTER TABLE commands for schema evolution, enabling operations like adding, renaming, or dropping columns, crucial for adapting to changing data requirements. Furthermore, DuckDB now supports partition transforms within Iceberg, providing more control over data organization and query optimization. Compatibility has been extended to Iceberg V3, ensuring support for the latest table format specifications, and improved handling for Iceberg REST Catalogs streamlines metadata management. These features position DuckDB as an even more powerful embedded analytical database for processing large-scale, evolving datasets directly in a data lake environment, making complex ETL/ELT operations more accessible and performant. Comment: The MERGE INTO and ALTER TABLE additions are game-changers for using DuckDB in production data pipelines with Iceberg, enabling robust upserts and schema changes. Open-Source TDE for PostgreSQL: What pg_tde Is, and Whether You Need It (Planet PostgreSQL) Source: https://postgr.es/p/9kM This article introduces pg_tde , PostgreSQL's new open-source Transparent Data Encryption (TDE) option, a l

The latest twist in paying humans to wear head cameras for robot training data.

Mallika Rao discusses the hidden risk of evaluation debt in production AI systems, drawing on her experience at Twitter, Walmart, and Netflix. She explains why traditional metrics fail modern architectures, breaks down a five-layer evaluation stack spanning infrastructure and UX, and shares a diagnostic maturity model to help engineering leaders eliminate silent semantic failures. By Mallika Rao

When I first connected to a precious metals WebSocket API, I expected to get a clean stream of prices. What I actually got was a firehose of mixed ticks—gold, silver, platinum—all arriving through the same callback. If you’ve ever tried to build a trading bot or a custom chart, you know this is a recipe for disaster. In this post, I’ll share how I solved the problem with a few lines of Python and a clear mapping strategy. The scenario: You have one WebSocket URL that pushes quotes for multiple metals. You need to separate them so you can update different UI components, run independent strategies, or store them in distinct database tables. The data pain point: every message uses the same JSON structure, and the only differentiator is a field like symbol . If you don’t act on it immediately, everything gets mixed up. Identify Assets via the Symbol Field Start by checking the API docs for the field that carries the instrument code. Usually it’s symbol , but instrumentId or type are also used. Here’s a typical reference table: Field Description Example symbol Asset code XAUUSD, XAGUSD instrumentId Internal platform ID 1001, 1002 type Asset class gold, silver I turn this into a dictionary mapping each symbol to a human-readable category: asset_map = { " XAUUSD " : " gold " , " XAGUSD " : " silver " , " XPTUSD " : " platinum " } Buffer Messages by Type Because these streams are high-frequency, I avoid processing every tick individually. Instead, the WebSocket callback just updates an in-memory store that is already grouped by asset type: # Keep the hot path extremely light def on_message ( msg ): symbol = msg [ ' symbol ' ] price = msg [ ' price ' ] asset_type = asset_map . get ( symbol , " unknown " ) cache [ asset_type ][ symbol ] = price Then, a background timer fetches the latest prices from cache["gold"] and cache["silver"] separately and does the actual work—like computing indicators or rendering charts. The key benefit is complete isolation: your gold logic never t

In modern application development, exposing database logic as REST APIs is a powerful way to integrate systems. Oracle REST Data Services (ORDS) makes it easy to turn PL/SQL into RESTful APIs without needing a separate backend service. In this blog, we’ll walk through how to create a simple POST API using ORDS and PL/SQL to insert data into a table. Pre-requisites A cloud-based ATP wallet (I prefer) Let's start how we create the APIs on the top of any custom table which relies on databases Create a table in the oracle SQL Developer and followed by create an ORDS Module 1.Create an ORDS Module A module is a logical container for related REST endpoints. What this Module does ?? Creates a module named nj_api Defines base URL: http://server_name/ords/table_Schema/nj_api/ 2: Define a Template (Endpoint Path) A template represents the API endpoint path. It defines how Endpoint URL:/ords/table_schema/nj_api/insert_data 3: Define the Handler (Business Logic) The handler contains the logic executed when the API is called. Key Concepts: p_method => 'POST': Defines HTTP method p_source_type => ORDS.source_type_plsql: Uses PL/SQL block Bind variables (:name, :num, etc.) map directly to JSON request body parameters 4: Testing the API Using Tools like Postman,cURL,ORDS REST Workshop I tested with Postman FYR Let's call same in Oracle VBCS in new blog. .. Try other methods like Delete, PATCH & GET

GitHub reports cutting token costs in agentic CI workflows by up to 62% by pruning unused MCP tools, swapping some MCP calls for gh CLI, and running daily “auditor” and “optimizer” agents. A token-usage.jsonl artefact and an Effective Tokens metric help track spend across models and spot regressions. By Mark Silvester

ORA-00031: Session Marked for Kill — What It Means and How to Fix It ORA-00031 occurs when a DBA issues ALTER SYSTEM KILL SESSION but Oracle cannot terminate the target session immediately. Instead, Oracle marks the session as "KILLED" and waits for it to reach a safe termination point — typically after completing a rollback or releasing OS-level resources. This is less of a hard error and more of a transitional state that every Oracle DBA will eventually encounter. Top 3 Causes 1. Large Transaction Rollback in Progress When you kill a session mid-transaction, Oracle must roll back all uncommitted changes to preserve data integrity. The larger the transaction, the longer the session stays in KILLED status. -- Check rollback progress for KILLED sessions SELECT s . sid , s . serial # , s . username , t . used_ublk AS undo_blocks , t . used_urec AS undo_records FROM v $ session s JOIN v $ transaction t ON s . taddr = t . addr WHERE s . status = 'KILLED' ; 2. Unresponsive or Disconnected Client If the client network connection is broken or the client process has hung, Oracle cannot deliver the kill signal. The session lingers in KILLED state until the OS-level connection finally times out. -- Find the OS process ID (SPID) for stuck KILLED sessions SELECT s . sid , s . serial # , s . username , s . status , p . spid AS os_pid , s . machine , s . program FROM v $ session s JOIN v $ process p ON s . paddr = p . addr WHERE s . status = 'KILLED' ; 3. OS-Level I/O or Resource Wait Sessions blocked at the OS level (disk I/O stall, memory pressure, storage issues) cannot respond to Oracle's internal kill signal. In these cases, only an OS-level process termination will resolve the problem. -- Identify what the session was waiting on before being killed SELECT sid , serial # , status , event , wait_class , seconds_in_wait FROM v $ session WHERE status = 'KILLED' ; Quick Fix Solutions Option 1 — Use the IMMEDIATE keyword (recommended first step) -- Standard kill (asynchronous) AL

Everyone can build it. Almost no one can afford to run it at scale. And the companies selling the picks and shovels are about to get undercut by the same forces they unleashed. by VEKTOR Memory — 20 min read How This Article Started: 20 Forums, 40 Headlines, and a Growing Sense That Everyone Was Confused I woke up to clear skies and the sun finally shining, and I set out to understand this idea, the truth behind it, and the nagging suspicion that the narrative around AI and software costs had become so loud, so uniform, and so confidently confusing that someone needed to sit down and actually go through it. No tweets, or are they now X's? No LinkedIn thought leader infomercials, no Substack hype, just actual research and deep thoughts. So I spent time reading, collating data. Forums, whitepapers, LinkedIn posts, Hacker News threads, VC essays, Reddit arguments. I went looking for the real signal underneath the noise. What I found instead was the full spectrum of human overconfidence, lots of moat real estate. On one end: the hype machine at full throttle. “Software is going to zero.” “A solo dev can now build what a 50-person team built in 2021.” “The era of the $500/month SaaS subscription is over.” “Vibe coding will replace your entire engineering org.” These headlines were everywhere. Breathless. Confident. Shared tens of thousands of times, this angle gets views, of course, the algorithm loves being fed claps, shares, comments, and reposts. Most were written by people who had a very good Tuesday with Codex, Windsurf, Claude and Cursor and decided that instant dev, open source to Github and getting oodles of stars, maybe even roping in a celebrity, was now the permanent condition of software development. “We are now famous on GitHub!" Very hipster, very vibes, see you on the playa.. On the other end: the backlash. Experienced engineer, people with 15 to 25 years in production systems are pushing back hard. “Show me the vibe-coded app that survived its first real

Most scraping projects start by finding a website to scrape. This one started from the opposite direction: I knew the data existed as official government downloads, and my job was to make it accessible via a clean API. The data source Ofsted (the UK school inspections body) publishes monthly management information as CSV files on GOV.UK. The file covers all 22,000+ state-funded schools in England with their latest inspection grades, local authority, postcode, phase, and size data. It's 16 MB, published under the Open Government Licence v3.0 — explicitly permitting commercial use. No scraping needed. No authentication. Just a CSV download and some parsing logic. The architecture The actor is deliberately simple: Fetch the GOV.UK stats page to find the current month's CSV URL (the URL hash changes with each release) Download the CSV (~16 MB from assets.publishing.service.gov.uk ) Parse it with csv-parse Apply the user's filters (name, local authority, region, postcode prefix) Push matching records to the Apify dataset No Crawlee. No browser. No proxy. Just fetch() and a CSV parser. const match = html . match ( /href=" ( https: \/\/ assets \. publishing \. service \. gov \. uk \/[^ " ] +latest_inspections_as_at [^ " ] + \. csv ) "/ ); That one regex does the URL discovery. The GOV.UK page lists files in reverse chronological order, so the first match is always the latest release. The interesting part: Ofsted changed their grading system mid-build I built this in May 2026. In November 2025, Ofsted scrapped their 20-year-old four-word judgement system (Outstanding / Good / Requires Improvement / Inadequate) and replaced it with a report card format — six separate grade areas, each on a five-point scale: Exceptional Strong Expected standard Needs attention Urgent improvement Plus a standalone Safeguarding verdict (Met / Not met). The April 2026 CSV reflects this change entirely. There's no "Overall effectiveness" column. Schools inspected before November 2025 have null gr

One leading privacy lawmaker said it was time to "start treating the adtech industry as a national security threat."

Enterprise AI is entering a different phase now, one where enterprises are no longer evaluating whether AI is exciting. They are evaluating whether it is safe to deploy broadly.

Advanced JSON Schema Validator Techniques for Power Users Once you're comfortable with basic validation, these advanced techniques will help you handle complex validation scenarios and integrate validation deeply into your systems. 1. Conditional Validation with if/then/else The most powerful feature in modern JSON Schema is conditional validation. Use it to enforce different rules based on the data itself: { "type" : "object" , "properties" : { "type" : { "type" : "string" , "enum" : [ "individual" , "business" ] }, "taxId" : { "type" : "string" }, "businessName" : { "type" : "string" } }, "allOf" : [ { "if" : { "properties" : { "type" : { "const" : "business" } } }, "then" : { "required" : [ "taxId" , "businessName" ] }, "else" : { "properties" : { "taxId" : { "not" : {} }, "businessName" : { "not" : {} } } } } ] } This schema makes taxId and businessName required only when type is "business". For individual accounts, those fields must not be present. 2. Custom Error Messages with Error Message Extension Enhance validation with user-friendly error messages that guide users toward correct input: { "type" : "object" , "properties" : { "password" : { "type" : "string" , "minLength" : 8 , "pattern" : "^(?=.*[A-Z])(?=.*[0-9])" , "errorMessage" : { "minLength" : "Password must be at least 8 characters" , "pattern" : "Password must contain at least one uppercase letter and one number" } } } } While not part of the core JSON Schema spec, many validators (including AJV) support the errorMessage keyword for better user-facing error reporting. 3. Schema Composition with allOf, anyOf, and oneOf Combine multiple schemas to create sophisticated validation rules: { "allOf" : [ { "$ref" : "#/$defs/baseUser" }, { "$ref" : "#/$defs/withTimestamp" }, { "if" : { "properties" : { "role" : { "const" : "admin" } } }, "then" : { "$ref" : "#/$defs/adminPrivileges" } } ] } allOf: Data must match ALL sub-schemas (intersection) anyOf: Data must match AT LEAST ONE sub-schema (union) oneOf: Da

Over the past decade, the core evolution of data engineering has been the deconstruction and reconstruction of traditional data warehouse architectures through the Modern Data Stack. We separated data ingestion from databases, forming the Data Ingestion layer, using tools like FiveTran, Airbyte, and Apache SeaTunnel to solve ELT / CDC / Reverse ETL problems; We separated compute from storage, forming cloud data warehouse and lakehouse systems such as Snowflake, Databricks, Iceberg, and Hive; We separated orchestration from scripts, leading to orchestration systems like Apache Airflow and Apache DolphinScheduler; SQL development, data modeling, lineage, data quality, BI, and AI analytics were further split into independent tools. This architecture was undoubtedly progress. It moved data engineering away from the primitive era of “a bunch of scripts + Crontab” toward cloud-native infrastructure, elastic computing, engineering governance, and open ecosystems. The greatest contribution of the Modern Data Stack was “decoupling,” and its biggest side effect was also “decoupling.” Tools became more powerful, but data engineers were forced to switch between more systems than ever before: datasources in one place, synchronization configs in another, DAGs somewhere else, logs elsewhere, SQL stored in Git, and Snowflake / Iceberg / cloud warehouse execution results living in yet another environment. As a result, many data engineers spend less time on data modeling, business understanding, metric definitions, architecture design, and cost optimization — and far more time configuring datasources, setting field mappings, dragging DAG nodes, modifying SQL, checking logs, and rerunning tasks. This is the hidden pain created by the Modern Data Stack: data engineers became trapped inside tools. The emergence of engineering-focused AI systems like Codex and Claude Code is now changing the entire software engineering workflow. But how can data engineers truly achieve Vibe Coding? That

In part 1 , the single-document case was easy. In part 2 , two documents brought Write Skew, and we saw that even a native ACID transaction — snapshot isolation — lets it through. So teams reach for the reflex fix: a distributed lock — Redis-based, often a Redlock-style implementation. Acquire a lock on a key, do your Read → Modify → Write, release. On paper, you've finally serialized the critical section — operationally, at least. In practice, you've stepped on three mines. 1. Network latency Every guarded transaction now makes extra round-trips to Redis — before and after hitting your NoSQL store. You've doubled your coordination surface and taken a hard dependency on a second system being up, reachable, and fast on the hot path of every write. The "fast" database is now gated by the lock service. And the coupling bites harder than the average latency suggests: every Redis tail-latency spike becomes your write-latency spike — your p99 inherits Redis's p99 — and if Redis fails over mid-transaction, the lock you think you're holding can effectively vanish on the new primary, dropping you straight into the corruption case below. 2. Deadlock You can dodge deadlock entirely with a single coarse lock — but then every writer serializes on it, and you've thrown away the very concurrency you reached for NoSQL to get. So to keep throughput you go fine-grained, one lock per resource — and the moment an invariant touches more than one key (across this series, it always does), deadlock is back on the table: Transaction A locks key X, then needs Y. Transaction B locks Y, then needs X. Both block until timeout or intervention. The textbook cure — real deadlock detection, maintaining a wait-for graph across every lock holder and breaking cycles as they form — is a distributed-systems project in its own right: not something you bolt onto a cache you reached for precisely to save engineering time. So nobody builds it. Instead teams impose a standing discipline: always acquire locks

Last year, I was doing something that felt increasingly absurd: manually reading AI-generated content to decide if it was "good enough." PostAll — the content automation tool I've been building — was producing hundreds of blog posts per week for clients. And I had no systematic way to evaluate quality at scale. I was spot-checking. Vibes-checking, really. That doesn't work at volume. So I built a programmatic quality analysis pipeline, ran it over 1,000 AI-generated posts, and let the numbers tell me what my gut was missing. The findings surprised me. A few of them genuinely changed how I think about AI content quality. What I Actually Measured First, a definition of terms, because "quality" is almost meaninglessly vague in this space. I broke quality into five measurable dimensions: Readability — Flesch-Kincaid grade level and reading ease score Keyword density — Target keyword frequency and distribution across the post Grammar error rate — Errors per 1,000 words, caught via LanguageTool's API Factual accuracy — Claims that could be verified programmatically (dates, statistics, named entities cross-referenced against a knowledge base) Structural consistency — Presence of expected elements: intro hook, subheadings, conclusion, CTA I used 1,000 posts across three categories: SaaS product descriptions, long-form "how-to" articles (1,200–2,000 words), and listicles (500–900 words). All were generated by PostAll using GPT-4o, with various prompting strategies. The Setup The analysis pipeline isn't complicated, but the piece that makes it useful is the batch processing layer: import anthropic import language_tool_python import textstat from dataclasses import dataclass from typing import Optional import json @dataclass class QualityReport : post_id : str flesch_reading_ease : float flesch_kincaid_grade : float grammar_errors_per_1000_words : float keyword_density : float structural_score : int # 0–5 based on element presence flagged_claims : list [ str ] overall_score :

Why Does Using an ORM Decrease Database Performance? While trying to optimize the shipping module in a production ERP, I noticed that database queries were incredibly slow. At first, I examined the SQL queries and checked the indexes. However, I couldn't get the performance boost I expected. The problem lay in the Object-Relational Mapper (ORM) library, which was the cornerstone of our application. ORMs make things easier for software developers by providing an abstract layer for database operations, but this convenience often comes at a performance cost. In this post, I will explain why using an ORM decreases database performance, using concrete examples from my own field experience. The core promise of ORMs is to keep developers away from SQL and allow them to interact with databases in a way that is more aligned with the object-oriented paradigm. This is a huge advantage, especially in small and medium-sized projects or rapid prototyping processes. However, when things get complex and performance becomes critical, the efficiency of the queries generated by ORMs starts to be questioned. In many cases I have encountered, especially in enterprise software development processes, the default behaviors of ORMs created an unexpected load on database servers. Query Inefficiencies Generated by ORMs ORMs usually manage database relationships using mechanisms like "eager loading" or "lazy loading". Depending on the developer's preference, these mechanisms either fetch all related data at once (eager loading) or fetch it in pieces as needed (lazy loading). However, ORMs may not always perform these loads in the most optimized way. For example, while only a few fields like ID and name are sufficient in a list view, the ORM might query the entire table or all related tables. This situation causes unnecessary data transfer and unnecessarily overloads the database server. To give an example, on the order list screen of an e-commerce site, we needed to display the customer inform

Author: Tobie Morgan Hitchcock Three months after 3.0 went GA, we're excited to announce that...

When regulation becomes theater and encryption becomes window dressing By Vektor Memory — 20 min read It is raining here in the Southern Hemisphere again. It has been raining for three weeks now, nonstop. I’m sitting with my chai coffee, watching out of the window, and thinking about data sovereignty. It is, genuinely, the kind of thing I think about often. The northern hemisphere is winding up for summer. Europe is getting ready for long evenings and beach holidays. I’m quietly jealous. I’ve always wanted to split the year: six months south, six months north. Endless summer. The perpetual warmth of a life lived chasing the sun. But here I am. Chai. Rain. Data. I’ve been turning over one question in particular: why is it that the moment you mention data sovereignty, people immediately reach for GDPR? It’s reflexive, especially among Europeans. Understandable. GDPR is loud, it’s enforced, it has teeth. French, German, and Dutch visitors make up a large disproportionate share of our site traffic at VEKTOR, and the interest in privacy and sovereignty from that audience is intense and genuine. Northern Europeans, by and large, take this seriously in a way that other markets don’t; they are working on ways to disassociate from the cloud around the world. And yet. How many times have we clicked “Accept All” on a cookie banner in the last week? How many times have you scrolled past a privacy policy that runs to forty-two pages? How many times have you handed over your email address, your location, your device fingerprint, your behavioral patterns not because you wanted to, but because there was no meaningful alternative? GDPR created the most sophisticated legal architecture for data rights the world has ever seen. It also created the most sophisticated ritual of consent theater the world has ever performed. That gap, between the law and the lived reality, is what this article is about. Ubiquitous data centre growth image The Reflex Problem When people think of data sovere

The third-party website exposed passports, selfies, and the location data of applicants who submitted their documents as part of the U.K. visa application process. Instead of fixing the issue, the website sent attorneys.