标签：#data

After migrating Spark pipelines to Azure Kubernetes Service, two infrastructure settings interacted destructively: spark.kubernetes.local.dirs.tmpfs=true backed shuffle spill with RAM instead of disk, and a hard podAffinity rule forced all executors onto one node. Together, they caused repeated OOM kills invisible to standard diagnostics. By Pranav Bhasker

Hand the same paired before/after dataset (n = 25) to ChatGPT five times. Same prompt: "These are the same subjects measured before and after an intervention. Did their scores change significantly?" Four of the five runs return p = 0.009 from a paired t-test. The fifth run does a Shapiro–Wilk normality check on the differences first, decides they're non-normal, switches to a Wilcoxon signed-rank test, and reports p = 0.000018 . All five reach the same conclusion (significant). But notice what happened: only one run out of five thought to check an assumption you'd want it to check. The other four skipped it. The choice of method — and the test statistic, and the p-value — depended on whether the LLM happened to run an assumption check that time. On borderline data, this is the difference between reject and don't reject. If you're using LLMs for exploratory data analysis on a weekend project, you might shrug. If you're using them for anything that gets cited, gets submitted to a regulator, or gets handed to a clinician, this is a problem. It's a known problem — Cui & Alexander (2026) documented exactly this kind of method-divergence empirically; AIRepr (Zeng et al., 2025) shows the same thing across reproducibility metrics. The current answer in the literature is to constrain the agent so its execution is replayable. But replayability fixes "did we run the same code." It doesn't fix "did we run the right analysis." I've spent the last two months building a different fix. The more interesting half is the architecture. Let me walk through it. The real problem isn't temperature The first reflex is "set temperature=0 ." It's not enough. temperature=0 doesn't make a tool-using agent deterministic across runs. Three reasons: Inference isn't bitwise deterministic, even at temperature=0. Production LLM serving batches requests dynamically, and the attention kernels aren't batch-invariant — so the same input produces different output tokens depending on what other requests it

With shiftbloom studio. I build tools and projects about a variety of experimental approaches to real-world problems. The issue for such use-case often was how most small media systems start out: one big always-on recorder that keeps costing money even when nothing is happening. For live capture you obviously need to stay ready at all times — sometimes you can’t risk losing the first minutes. But for everything else it’s complete overkill. The Core Problem Backfills, VOD downloads, clip imports, repairs and re-encodes are queue work. They can wait a few seconds, run on burst capacity, or even on a regular VPS or laptop. They don’t need the same always-hot infrastructure as the live recorder. That’s why I split the system. Instead of one large monolith, I deployed: Observer cells — only for live streams (time-critical) Harvest cells — for all queue processing (can be delayed) The Three Roles 1. Mothership A small control-plane cron job. It checks queue sizes, currently live channels and running observer tasks, then decides: how many harvest cells should exist right now which channels need an observer cell It’s intentionally simple. The database remains the single source of truth. 2. Observer Cells Each observer cell records exactly one live channel. It receives its assignment through environment variables: +++env OBSERVER_VOD_ID OBSERVER_CHANNEL_ID OBSERVER_CHANNEL_LOGIN OBSERVER_CHANNEL_NAME +++ It starts recording immediately, writes HLS segments to object storage, sends heartbeats, and waits a short standby window after the stream goes offline. This window is important because streams sometimes drop and reconnect quickly. Without it you end up with many small broken VOD fragments. 3. Harvest Cells These handle all background work: downloading VODs, re-encoding, recovering broken files, etc. They can run anywhere Docker is available — AWS tasks, a small VPS, or even a spare laptop. They only need outbound access to Postgres and object storage. What Changed Previous

Hello! I'm Satoshi Kaneyasu, DevOps engineer at Serverworks. In this article, I'll introduce the basic concepts and terminology of vector databases for those who are just starting to learn about them. Target Audience This article is aimed at beginners to vector databases. You may have heard that vector databases are related to LLMs and RAG, but aren't quite sure what they actually are. Think of this as written with that kind of reader in mind. What Is a Vector Database? A vector database is a database that stores data as vectors (arrays of numbers) and searches for data using "distance" or "similarity" between vectors. Traditional relational databases search for data using "exact match" or "partial match" (LIKE queries), but vector databases can search for things that are semantically similar . For example, searching for "weather in Tokyo" might return results like "temperature in Tokyo" or "weather conditions in Kanto" — data that differs as a string but is semantically related. Visualizing Vector Space In a vector database, all data is represented as points in a multidimensional space. When searching, the query is also converted into a vector, and data that is "close in distance" within that space is retrieved. This diagram represents it in two dimensions, but in a real vector database, proximity and distance are defined across many dimensions. Use Cases for Vector Databases Vector databases are used across a wide range of applications: Use Case Description RAG (Retrieval-Augmented Generation) Knowledge base search to provide external knowledge to LLMs. Allows internal documents and up-to-date information to be reflected in LLM responses Semantic Search Searching internal documents or FAQs by meaning rather than keywords. Handles spelling variations and synonyms Recommendation Recommending products and content whose vectors are close to a user's preference vector. Used as an alternative or complement to collaborative filtering Image Search Searching for similar im

A few months ago, I built an AI agent to automate one of the most repetitive parts of my workflow:...

A government committee says that the country’s growing dependence on the data analytics company is a serious liability.

The cybersecurity company is nearing a $300 million round led by Evolution Equity Partners.

The LedgerSync pipeline was inserting 1.5 million rows into PostgreSQL using pandas.to_sql() . It took four minutes per run. I switched to psycopg2's COPY command and it dropped to 18 seconds. Same data, same schema, same machine. That is not an optimization tip. It is the difference between a pipeline that fits in an Airflow schedule and one that does not. This article is about patterns like that: the ones that matter when you are building pipelines that run on a schedule, not when you are writing ad-hoc queries. Loading Data: to_sql vs execute_values vs COPY There are three ways to write rows from Python into PostgreSQL, and the performance gap between them is significant. pandas to_sql issues one INSERT statement per row by default, or a multi-row INSERT with method="multi" . It is the easiest to write and the slowest for any serious volume. psycopg2 execute_values batches many rows into a single multi-row INSERT VALUES statement. About 5x faster than to_sql for medium-sized loads. psycopg2 COPY streams rows directly to PostgreSQL using its native bulk-load protocol. No statement parsing, no row-by-row overhead. For LedgerSync at 1.5M rows, this was the one that mattered. import psycopg2 import io import pandas as pd conn = psycopg2 . connect ( " host=localhost dbname=proj_db user=proj_user password=proj_pass " ) def bulk_copy ( df : pd . DataFrame , table : str , columns : list [ str ]): buf = io . StringIO () df [ columns ]. to_csv ( buf , index = False , header = False ) buf . seek ( 0 ) with conn . cursor () as cur : cur . copy_from ( buf , table , sep = " , " , columns = columns ) conn . commit () print ( f " Loaded { len ( df ) } rows into { table } " ) Use COPY for initial loads and large backfills. For incremental daily writes of a few thousand rows, execute_values is fine and gives you more control over conflict handling: from psycopg2.extras import execute_values def bulk_insert ( rows : list [ dict ], table : str ): if not rows : return columns = list

T-SQL on Microsoft Fabric - Episode 1: Mastering Data Retrieval with SELECT, WHERE, and ORDER BY Learning Goals In this lesson, you will learn how to: Read data from tables using SELECT Filter rows with WHERE Sort query results with ORDER BY Get familiar with standard T-SQL syntax Practice directly in Microsoft Fabric Warehouse 1. Understanding Database and Schema In Fabric Warehouse, objects are commonly organized like this: Warehouse | |-- sales | |-- Customers | |-- Orders | |-- hr | |-- Employees | |-- finance |-- Transactions Schemas help you: Group related tables Manage permissions Organize large systems more effectively 2. Create a Schema Create a schema for the sales dataset: CREATE SCHEMA sales ; Check existing schemas: SELECT * FROM sys . schemas ; 3. Create Tables Create the Customers table: CREATE TABLE sales . Customers ( CustomerID INT , CustomerName VARCHAR ( 100 ), City VARCHAR ( 50 ), Country VARCHAR ( 50 ) ); Create the Orders table: CREATE TABLE sales . Orders ( OrderID INT , CustomerID INT , OrderDate DATE , Amount DECIMAL ( 10 , 2 ) ); 4. Insert Sample Data Customers INSERT INTO sales . Customers VALUES ( 1 , 'John Smith' , 'New York' , 'USA' ), ( 2 , 'Emma Brown' , 'Chicago' , 'USA' ), ( 3 , 'David Wilson' , 'London' , 'UK' ), ( 4 , 'Sophia Taylor' , 'Manchester' , 'UK' ), ( 5 , 'Michael Lee' , 'Singapore' , 'Singapore' ); Orders INSERT INTO sales . Orders VALUES ( 101 , 1 , '2026-01-10' , 1200 . 00 ), ( 102 , 1 , '2026-01-15' , 800 . 00 ), ( 103 , 2 , '2026-01-20' , 2500 . 00 ), ( 104 , 3 , '2026-02-01' , 500 . 00 ), ( 105 , 5 , '2026-02-05' , 3200 . 00 ); 5. SELECT Get all columns: SELECT * FROM sales . Customers ; Get specific columns: SELECT CustomerName , Country FROM sales . Customers ; 6. Alias Rename columns in the output: SELECT CustomerName AS Customer , Country AS Nation FROM sales . Customers ; 7. WHERE Filter rows using conditions. Customers in the USA: SELECT * FROM sales . Customers WHERE Country = 'USA' ; Orders greater than 100

Your API is fast. Your code is clean. Your architecture looks solid on paper. Then you hit 500,000 records and everything slows down. Queries that ran in 12ms now take 4 seconds. Your dashboards lag. Users start filing support tickets. Your on-call engineer is staring at a query plan at midnight wondering what went wrong. Nine times out of ten, the answer is indexing. Not missing indexes — wrong indexes. Indexes that exist but don't help. Indexes that actively hurt write performance without meaningfully improving reads. This is a breakdown of the most damaging database indexing mistakes in production SaaS systems — and how to fix them before they become incidents. Mistake 1: Indexing Everything "Just in Case" The most common mistake isn't under-indexing. It's over-indexing out of anxiety. New engineers especially fall into this pattern — add an index on every column that appears in a WHERE clause, just to be safe. Seems responsible. It isn't. Every index you add is a write tax. On every INSERT, UPDATE, and DELETE, PostgreSQL (or MySQL) has to update every index on that table. On a table with 8 indexes, every write touches 8 data structures. At low volume, this is invisible. At 10,000 writes per minute, it becomes your bottleneck. The fix: Audit your indexes regularly. In PostgreSQL: SELECT schemaname , tablename , indexname , idx_scan , idx_tup_read , idx_tup_fetch FROM pg_stat_user_indexes ORDER BY idx_scan ASC ; Any index with idx_scan = 0 or near zero hasn't been used since your last stats reset. That's a candidate for removal — not immediately, but after investigation. Mistake 2: Not Understanding Index Selectivity An index on a boolean column ( is_active , is_deleted ) is almost always useless. Here's why: selectivity measures how many distinct values exist relative to total rows. A boolean column has two values. If 95% of your rows have is_active = true , an index on that column tells the query planner almost nothing useful. It will often skip the index entire

AI applications are generating more data than ever before. From model telemetry and user interactions to observability events and real-time analytics, modern systems need infrastructure that can ingest, process, and query massive datasets with low latency. That's exactly the problem ClickHouse is targeting with its latest release. The update introduces improvements across query performance, memory management, Kafka integration, lakehouse support, and developer tooling. While many of these changes appear incremental on the surface, together they highlight a much larger shift happening across the industry. One of the most notable additions is improved memory management for large joins. ClickHouse can now automatically spill hash joins to disk when memory usage exceeds configured thresholds. Instead of failing due to memory pressure, queries can continue running using more efficient execution strategies. For teams working with large feature tables, event enrichment, AI telemetry, or observability data, this can significantly improve reliability. The release also expands ClickHouse's Kafka capabilities with Schema Registry integration, AvroConfluent write support, metadata mapping, and zone-aware communication. These improvements make it easier to integrate ClickHouse into real-time event pipelines while reducing latency and unnecessary cross-zone traffic in cloud environments. Another major focus is support for modern lakehouse architectures. Improvements for Apache Iceberg and Apache Paimon strengthen ClickHouse's ability to query data stored in open table formats while maintaining high analytical performance. As more organizations separate storage and compute, ClickHouse is increasingly positioning itself as a high-speed query layer on top of cloud-native data lakes. Performance optimization remains a major theme throughout the release. Improvements include faster JOIN execution, better ORDER BY LIMIT performance, enhanced JSON processing, smarter index pruning, redu

A linked list is an ordered linear data structure where elements are not stored in sequential memory locations, instead they are stored in nodes that are linked together by a pointers. Linked list are used in data intensive application because linked list offer specific benefits for high frequency data manipulation, this benefits include: Efficient insertion and deletion Adding and removing elements from a linked list is highly efficient, unlike arrays which requires shifting all subsequent elements to maintain indexing. A linked list only requires updating the pointer. Dynamic sizing: linked list can grow or shrink during runtime without needing to pre-allocate memory. Memory management: Nodes in a linked list are only allocated when needed which prevents memory wastage. Flexible Traversal: Doubly and circular list allow you to move forward or backward, which makes them helpful for complex navigation The first node in a linked list is called the head which signifies the start of the list, while the last node is called the tail and has a pointer of null except in a circular linked list. Each node in a linked list has two things which are: the actual data the pointer or reference There are three main types of linked list: Singly linked list Doubly linked list Circular linked list Singly Linked List: Singly linked list are lists where each node has a next pointer that points to the next node. Doubly Linked List: Doubly linked list are list where each node has a next and previous pointer that points to the previous and next node. Circular Linked List: Circular linked list are list where the last node points back to the first node, forming a circle. Table of Contents create node class create linked list class isEmpty and getSize Methods prepend and append Method removeHead and removeTail Methods insert and search Methods getIndex and removeIndex Methods clear and print Methods create node class First let's open our code editor and create a new file called singlyLinkedLi

In this article, author Aaditya Chauhan discusses the limitations of RAG pipelines based purely on vector search and how an internal omni-search application using Reciprocal Rank Fusion (RRF) that combines BM25 and vector results, can enhance the search solution. By Aaditya Chauhan

Switching focus from Frontend development to Data Engineering means shifting from building user interfaces to architecting robust data pipelines. It’s a completely different mindset, and the learning curve is exciting! As I dive deeper into the world of Data, these are the 4 essential tools and concepts that have become the absolute backbone of my daily learning roadmap: 1️⃣ Python (The Swiss Army Knife): Coming from JavaScript/TypeScript, picking up Python has been a breath of fresh air. From writing custom ETL scripts to data manipulation with Pandas, it's the ultimate language for data manipulation. 2️⃣ Advanced SQL (The Core): It's not just about simple SELECT statements anymore. Mastering Window Functions, CTEs (Common Table Expressions), and query optimization is where the real magic happens when interacting with Data Warehouses. 3️⃣ ETL/ELT Pipelines: Understanding how to efficiently Extract, Transform, and Load data without breaking downstream analytics. Moving from UI state management to Data state management is a game-changer. 4️⃣ Cloud Ecosystems & Modern Stack: Exploring how data flows through modern cloud environments and learning how big data tools manage scale. The transition requires patience, but applying my previous engineering background to these new tools makes the journey incredibly rewarding. 💡 To the Data Engineers in my network: What is the one tool or concept you believe is a "must-have" for someone transitioning into the field today? Drop your advice below!

PostgreSQL 19 Graph Queries & REPACK; SQLite Advanced SQL Patterns Today's Highlights PostgreSQL 19 introduces major features with SQL/PGQ for graph queries on relational data and a new REPACK command to combat table bloat. Meanwhile, the SQLite community explores advanced SQL patterns for complex 'first match' selection logic, pushing the boundaries of efficient embedded database querying. SQL/PGQ in PostgreSQL 19: Graph Queries Without the Graph Database (Planet PostgreSQL) Source: https://postgr.es/p/9kW PostgreSQL 19 is set to integrate native graph querying capabilities through the introduction of SQL/PGQ (Property Graph Queries) and the GRAPH_TABLE construct. This innovative feature allows users to perform sophisticated graph-like analytics, such as pathfinding and pattern matching, directly on their existing relational datasets without requiring data migration to a specialized graph database. The syntax employed for defining graph patterns closely resembles that of Cypher, making it intuitive for developers familiar with graph query languages. Developers will be able to define 'nodes' and 'edges' from their relational tables and then utilize the GRAPH_TABLE function within standard SQL queries to traverse and extract complex, interconnected information. This effectively transforms PostgreSQL into a multi-model database capable of handling both traditional relational workloads and demanding graph analytics side-by-side, within a single consistent environment. This significant enhancement expands PostgreSQL's utility across a wide range of applications, including social network analysis, fraud detection, and supply chain optimization, where understanding relationships between data points is critical. By embedding graph capabilities, PostgreSQL removes the need for separate graph database infrastructure, simplifying application architectures and leveraging the robustness and familiarity of the PostgreSQL ecosystem for new and existing projects. Comment: This is

This article was originally published on davidohnstad.com . I cross-post here to reach the Dev.to community. { " @context ": " https://schema.org ", " @graph ": [ { "@type": "Person", " @id ": " https://davidohnstad.com/#author ", "name": "David Ohnstad", "url": " https://davidohnstad.com ", "sameAs": [ " https://www.linkedin.com/in/davidohnstad/ ", " https://orcid.org/0009-0007-9023-7456 ", " https://davidohnstad5.mystrikingly.com/ ", " https://github.com/davidohnstad40-netizen ", " https://hashnode.com/@davidohnstad ", " https://davidohnstad.com ", " https://davidohnstad.net ", " https://davidohnstad.info ", " https://david-ohnstad.com ", " https://davidohnstadminnesota.com " ], "jobTitle": "Senior Data Product Manager", "worksFor": { "@type": "Organization", "name": "Veeam Software", "url": " https://www.veeam.com " }, "alumniOf": { "@type": "CollegeOrUniversity", "name": "College of St. Scholastica" }, "address": { "@type": "PostalAddress", "addressLocality": "Duluth", "addressRegion": "MN", "addressCountry": "US" }, "description": "Senior Data Product Manager at Veeam Software, MS and MBA from the College of St. Scholastica, based in Duluth, Minnesota. Specializes in data architecture, AI/ML integrations, and SaaS platform development." }, { "@type": "Article", " @id ": " https://davidohnstad.com/data-product-manager-org-structure-reporting#article ", "headline": "Data Product Manager Org Structure: Reporting Lines That Matter", "description": "David Ohnstad reveals where data product managers actually fit in org charts and why reporting lines determine success. Real insights from a data PM restructure.", "url": " https://davidohnstad.com/data-product-manager-org-structure-reporting ", "datePublished": "2026-05-29T14:06:18Z", "dateModified": "2026-05-29T14:06:18Z", "author": { "@type": "Person", " @id ": " https://davidohnstad.com/#author " }, "publisher": { "@type": "Organization", "name": "David Ohnstad", "url": " https://davidohnstad.com ", "logo": { "@type"

Last week I wrote about why healthcare benefit data is still trapped in PDFs . The response told me something: people in this space know the problem is real. But extraction is only half the story. The harder question is: when an AI system pulls a copay amount from a carrier document, can you prove where that number came from? Not "the AI said so." Not a confidence score with no backing. Can you point to a specific page, a specific table cell, a specific paragraph in the source PDF and say: this value came from here? That is field-level provenance. And in healthcare, it is no longer optional. The Regulatory Floor Just Rose In January 2026, two state laws went into effect that changed the baseline for AI-generated content in healthcare. California SB 942 requires AI systems to disclose when content is AI-generated and to maintain audit trails. Texas HB 149 mandates transparency about AI decision-making processes in regulated industries, with healthcare squarely in scope. These are not theoretical. They are enforceable. And they are just the beginning. CMS transparency mandates tighten every year. Gartner declared digital provenance an enterprise baseline for 2026. The industry is not moving toward provenance. It has arrived. The Problem with Self-Reported Citations Most AI extraction systems today work like this: a language model reads a document, extracts values, and reports where it found them. The model does the extraction AND the citation. It is grading its own homework. This seems fine until you look closer. A language model that hallucinates a copay amount will also hallucinate the page number it came from. The citation and the extraction fail together, silently, in the same direction. In a coverage dispute that ends up in a regulatory proceeding, "the AI told us it found this on page 3" is not evidence. It is hearsay from a statistical model. What Deposition-Grade Provenance Looks Like Field-level provenance means every extracted value carries metadata from an

The company says it needs "significant" water resources to cool its data centers, and that access to abundant, affordable water is a challenge.

Anthropic introduced Dynamic Workflows, a new capability for Claude Code designed to handle complex software engineering tasks by coordinating large numbers of AI agents within a single workflow. The feature allows Claude to dynamically create orchestration scripts, break work into subtasks, run them in parallel, and validate results before presenting a final answer. By Robert Krzaczyński

During maintenance tasks like backup or inspection, you often need to switch a gbase database cluster to READONLY mode and back to NORMAL afterwards. This post shows how to perform the switch online with gcadmin — no downtime required. Tools and Prerequisites Tool : gcadmin , located on any Coordinator node. User : Must be the cluster installation user (default gbase ). Pre‑check : Ensure gcware services are running ( gcware_services status ). Scope : The mode change applies cluster‑wide; no need to repeat per node. Step‑by‑Step (READONLY → NORMAL) Step 1: Check the Current Mode # Cluster‑wide gcadmin showcluster # Specific VC in a multi‑VC environment gcadmin showcluster vc vc1 Look at the VIRTUAL CLUSTER MODE field. Proceed only if it shows READONLY . Step 2: Switch to Normal Read/Write Mode # Single VC or whole cluster gcadmin switchmode normal # Specific VC gcadmin switchmode normal vc vc1 The switch takes effect in seconds. The cluster synchronises the new mode to all nodes automatically — no process restart is needed. Step 3: Verify the Change Run gcadmin showcluster again. When VIRTUAL CLUSTER MODE shows NORMAL , the cluster is fully writable again. Production Notes Business impact : An online switch does not interrupt running read queries. Pending write operations queued during READONLY mode are executed automatically after the switch. Node to run on : Any single Coordinator node is enough. Privileges : Only the gbase user can switch modes. Reverse switch : To return to read‑only mode, use gcadmin switchmode readonly [vc vc_name] . Troubleshooting : If the command fails, check gcware service health and node status first, then retry. Companion Commands # Check all cluster processes gcluster_services all info gcware_services all info # View detailed VC information gcadmin showvc The mode‑switch mechanism in GBase 8a is built for high availability. Following the "verify → switch → re‑verify" workflow lets you change cluster modes safely and transparently in a g