标签：#Data

We train a model on handwritten digit classification. 99% accuracy . Then we train the same model on a new task — say, fashion item recognition. We go back and test it on digits. 34% accuracy . It has completely forgotten. Not gradually, not partially — almost entirely. What Just Happened? We trained a CNN on MNIST digits — 99.2% accuracy . After fine‑tuning on Fashion MNIST, it reached 91.1% accuracy . But when re‑evaluated on MNIST, accuracy collapsed to 33.9% . This collapse is catastrophic forgetting : the model’s weights shifted to optimize for the new task, erasing the old solution. Why did training on more data make the model worse at something it already knew? MNIST is handwritten digits (0–9). Fashion MNIST is clothing items like shirts and shoes. Both are 28×28 grayscale images, but the tasks are distinct. Why Does It Happen? The core issue is that the model relies on the same set of weights for both tasks. There is no separation or dedicated memory; every parameter is shared . When training shifts from Task A ( MNIST digits ) to Task B ( Fashion MNIST ), gradient descent simply minimizes the loss on the data it sees at that moment. It has no awareness that Task A ever existed. In the loss landscape, imagine two parabolic bowls: one for Task A and one for Task B. The optimum for Task A lies at θ A ∗ ​ , while Task B's optimum is at θ B ∗ ​ . As training on Task B progresses, the weights θ move towards θ B ∗ ​ . This movement inevitably raises the loss for Task A because its minimum is left behind. The root cause is the shared weight space. Gradient descent is a stateless optimizer; it only follows the current gradient signal. Since the minima for Task A and Task B are far apart, there is no single configuration of θ that satisfies both tasks simultaneously. This is why catastrophic forgetting occurs. Weight space can be visualized as an N-dimensional space, where each axis corresponds to one parameter. Every point in this space represents a full set of wei

Hello everyone! Following up on my previous post , Day 1 of my Modern Data Stack migration was an absolute rollercoaster of refactoring and deep data auditing. I’m moving our legacy system (spreadsheets and Qlik) into a robust pipeline using Python, ClickHouse, and dbt . Here is what went down over the last 24 hours. 1. From Messy Scripts to a Single, Parameterized Extraction Engine 🛠️ In the legacy setup, each company had its own folder, its own .env file, and its own duplicated Python extraction script. It was a maintenance nightmare. Yesterday, I completely refactored this structure: Centralized Configuration: Merged all separate environments into a single, global .env file at the root level, mapping all 8+ companies and their branches. Eliminated Code Duplication (DRY): Instead of having identical extraction logic copied across folders, I built a single, unified codebase. Now, we have one universal script for Sales, one for Stock, one for Orders, etc. The behavior changes dynamically based on the company argument we pass to the CLI (e.g., python -m extract.run extract --source company1 ). To speed up this refactoring, I used Claude to generate the initial application skeleton. Since the AI already had the context of our legacy extraction logic, translating it into this new clean architecture was incredibly smooth. 2. Highs and Lows: The Data Parity Challenge With the pipeline modernized, I ran the pilot ingestion for Company #1 . To minimize friction for our downstream BI consumers, I kept the ClickHouse Bronze tables structured 1:1 with the legacy CSV schemas. The Good News: The data ingestion into the Bronze layer worked flawlessly. Moving up to the Silver layer (where we do data cleaning and domain-specific transformations), everything validated beautifully. Row counts matched perfectly. The "Fun" Part (The $2 Million Gap): When I materialized the Gold layer (our consolidated group business models), I hit a massive wall. The new pipeline reported $2 million U

Adi Polak discusses the architecture required to transition from stateless prompts to state-aware, context-rich AI agents. Drawing on 15 years in distributed systems, she shares how engineering leaders can leverage Apache Kafka and Flink for real-time stream processing, dynamic memory tiering, and tool orchestration via MCP to solve token limits, cost spikes, and latency bottlenecks. By Adi Polak

Redis is fast. But self-hosting Redis on a serverless stack is a nightmare — cold starts, connection pool exhaustion, and managing a persistent server that your serverless functions keep hammering. Upstash solves this with an HTTP-based Redis API that scales to zero, charges per request, and works natively with Next.js App Router. This guide covers the patterns that actually matter in production: cache-aside with proper TTLs, SWR (stale-while-revalidate), session storage, and pub/sub. Real code, real trade-offs. Read the full article with all code examples at stacknotice.com Why Upstash Over a Traditional Redis Instance Standard Redis uses persistent TCP connections. Serverless functions don't maintain persistent connections — every invocation potentially opens a new one. At scale, you hit ECONNREFUSED or max connection errors that are annoying to debug and expensive to fix. Upstash's @upstash/redis client talks over HTTP/REST. No connection pool, no connection limit headaches. Each request is stateless. This is exactly what Next.js Server Components and Route Handlers need. Other advantages: Pay per request — a cache that never gets hit costs $0 Global replication — low latency from any Vercel edge region Native Edge Runtime support — works in Next.js middleware Free tier — 10,000 commands/day, no credit card needed Setup npm install @upstash/redis // lib/redis.ts import { Redis } from ' @upstash/redis ' export const redis = new Redis ({ url : process . env . UPSTASH_REDIS_REST_URL ! , token : process . env . UPSTASH_REDIS_REST_TOKEN ! , }) Pattern 1: Cache-Aside // lib/cache.ts import { redis } from ' ./redis ' export async function withCache < T > ( key : string , fetcher : () => Promise < T > , options : { ttl ?: number ; prefix ?: string } = {} ): Promise < T > { const { ttl = 300 , prefix = ' cache ' } = options const cacheKey = ` ${ prefix } : ${ key } ` const cached = await redis . get < T > ( cacheKey ) if ( cached !== null ) return cached const data = awai

I recently built GeoPrizm , a free and open-source dashboard for tracking bilateral relations through global news event signals. The idea is simple: instead of reading dozens of headlines every day and trying to guess whether a relationship is improving or worsening, can we turn public news event data into a readable trend signal? GeoPrizm is my attempt at that. Website: https://www.geoprizm.com/en GitHub: https://github.com/Haullk/relationship-temperature The problem International relations are usually discussed through headlines, speeches, official statements, and expert commentary. That is valuable, but it creates a few practical problems: It is hard to compare country pairs on the same scale. A single headline can feel more important than it really is. Readers often see conclusions before they see the underlying signals. Most non-specialists do not have time to follow every event in detail. I wanted a lightweight way to answer one question: Based on public news event signals, is this bilateral relationship trending more cooperative, neutral, or tense? Data source: GDELT GeoPrizm uses the GDELT global news event database. GDELT monitors global news coverage and converts news reports into structured event records. These records include fields such as: actor countries event date CAMEO event category GoldsteinScale value number of mentions number of articles source information For GeoPrizm, the key idea is to focus on events where two countries appear as actors, then aggregate the cooperation or conflict signals over time. From event signals to an index Each bilateral pair is converted into a 0-100 relationship index. The midpoint is 50. Above 50 means the recent signal is more cooperative or favorable. Around 50 means the signal is relatively neutral or mixed. Below 50 means the recent signal is more tense or conflict-heavy. The rough process is: Select recent GDELT events for a country pair. Keep events where both actors are present and the GoldsteinScale value is

PostgreSQL Error 2201X: invalid row count in result offset clause PostgreSQL error code 2201X ( invalid_row_count_in_result_offset_clause ) is thrown when the value provided to an OFFSET clause is not a valid non-negative integer. This commonly surfaces in applications that implement pagination using dynamic queries or user-supplied parameters, where the offset value may be negative, NULL, or a non-integer type. Top 3 Causes 1. Negative OFFSET Value The most frequent cause is a miscalculated page offset. When computing (page - 1) * page_size , a bad page number can produce a negative result. -- Triggers error 2201X SELECT * FROM orders ORDER BY created_at DESC OFFSET - 5 LIMIT 20 ; -- ERROR: invalid row count in result offset clause -- Fix: Use GREATEST to clamp the value to 0 SELECT * FROM orders ORDER BY created_at DESC OFFSET GREATEST ( 0 , - 5 ) LIMIT 20 ; 2. NULL Passed as OFFSET When an application fails to initialize or bind the offset parameter, NULL gets passed to the query. This often happens with ORMs or query builders where optional parameters are not explicitly set. -- Triggers error 2201X SELECT * FROM products ORDER BY id OFFSET NULL LIMIT 10 ; -- ERROR: invalid row count in result offset clause -- Fix: Use COALESCE to provide a default value SELECT * FROM products ORDER BY id OFFSET COALESCE ( NULL , 0 ) LIMIT 10 ; -- Parameterized query with safe fallback SELECT * FROM products ORDER BY id OFFSET COALESCE ( $ 1 :: BIGINT , 0 ) LIMIT COALESCE ( $ 2 :: BIGINT , 10 ); 3. Non-Integer Type (Float or String) Passing a float or a string that cannot be cleanly cast to a whole number causes this error. This typically happens when Python float values or unvalidated user input strings are interpolated directly into a query. -- Triggers error 2201X SELECT * FROM employees ORDER BY last_name OFFSET 10 . 7 LIMIT 5 ; -- ERROR: invalid row count in result offset clause -- Fix: Use FLOOR and explicit cast to BIGINT SELECT * FROM employees ORDER BY last_name OFFSET F

SQL is arguably the most widely used language in software engineering, yet it is often the least carefully written. Most teams enforce strict linting on their application code but leave SQL queries as a free-for-all. This guide covers the formatting rules that separate maintainable, team-friendly SQL from query spaghetti that haunts on-call rotations. Why Poorly Written SQL Is a Real Engineering Problem Unformatted SQL is not just an aesthetic issue - it is a correctness risk. Dense, run-on queries make it nearly impossible to spot accidental Cartesian products, missing GROUP BY clauses, or WHERE conditions that silently bypass indexes. By the time a performance problem surfaces in production, tracing it back to the root cause becomes a painful exercise in reading someone else's stream of consciousness. Rule 1: Keyword Capitalization SQL engines treat select and SELECT identically, but human readers do not. Always uppercase reserved keywords such as SELECT, FROM, WHERE, JOIN, GROUP BY, and ORDER BY. Keep table names, column names, and aliases lowercase. This single habit immediately creates a visual boundary between the logic structure of the query and the underlying data it operates on. Rule 2: Indentation and Clause Alignment Think of SQL clauses as layers in a data pipeline. Each major clause - SELECT, FROM, WHERE, GROUP BY, ORDER BY - should start at the left margin. Columns and filter conditions beneath them should be indented by 4 spaces (or 1 tab, as long as your team is consistent). This structure lets any reviewer skim the query top-to-bottom and understand the data flow at a glance. Rule 3: Trailing vs. Leading Commas This is a genuinely debated topic on data teams. Leading commas (placing the comma at the start of each new line) make version control diffs significantly cleaner when columns are added or removed. Trailing commas look more natural for developers coming from JavaScript or Python. Neither approach is wrong - what is wrong is mixing both styles

Why Your Vector Database Is Overpriced: Lucene's 32x Compression and Serverless Economics In 2026, the boundary between "search engine" and "AI infrastructure" has dissolved. What started as text indexing has become the backbone of retrieval-augmented generation, vector databases, and serverless AI pipelines. This is the story of how the oldest search technology in the Java ecosystem became the most important infrastructure you've never noticed. The Convergence No One Saw Coming Five years ago, if you said Apache Lucene would power the next generation of AI infrastructure, you'd have been laughed out of the room. Lucene was the boring Java library that powered Elasticsearch — reliable, yes, but hardly exciting. The action was in vector databases: Pinecone, Weaviate, Qdrant. The cool kids had moved on. That narrative died in 2025. What happened was a structural inversion. While vector-native databases optimized for one thing (fast similarity search), the real production pain points were everywhere else: hybrid search, metadata filtering, provenance tracking, multi-tenant security, and — most critically — the ability to query both your documents and your vectors in a single, unified system. Lucene didn't just survive this transition. It engineered it. Through a series of aggressive, hardware-native optimizations between versions 10.0 and 10.4, Lucene transformed from a text indexer into a vector search kernel capable of outperforming specialized databases while maintaining the operational maturity that enterprises actually need. And Elasticsearch, riding on Lucene's coattails, didn't just integrate vectors — it re-architected itself into a stateless, serverless platform that happens to do search. This post examines three layers of that transformation: the engine (Lucene), the platform (Elasticsearch), and the architecture (AI-native search infrastructure). Each layer tells a different story, but they share a common thread: the future of AI infrastructure is being buil

Imagine a simple money transfer scenario. John sends money to his friend Sarah. The system successfully deducts money from John's account, but before it can credit Sarah's account, the application crashes. Without proper safeguards, John's money would disappear from the system, creating inconsistent and unreliable financial records. To prevent this type of problem, database transactions are used. Transactions ensure that a group of related database operations either complete successfully together or fail together. If any part of the process encounters an error, all changes are reverted, ensuring that the database remains consistent. Transactions make database operations atomic. Atomicity means that all operations inside a transaction are treated as a single unit of work. Either every operation succeeds and is committed to the database, or all operations fail and are rolled back. Partial updates are never permanently stored. A database transaction is a group of one or more database operations executed as a single unit. Either all operations succeed together or all operations fail together. This guarantees database consistency even if an application crashes, a network failure occurs, or an unexpected error is encountered during execution. It is important to understand that a transaction is not simply a single database query. While individual queries such as save, update, or delete interact with the database, a transaction wraps multiple queries inside a controlled all-or-nothing boundary. This prevents partial updates and ensures data integrity throughout the process. Prerequisites Before starting, make sure you have the following: Basic knowledge of NestJS Basic understanding of TypeORM Basic knowledge of PostgreSQL Understanding of basic database operations (save, update, delete) in TypeORM Project Setup In this article, we will create a simple NestJS application to demonstrate the importance of transactional queries when multiple database write or update operations

If you scrape Weibo's hot-search board you get a snapshot: ~50 trending topics, ranked, right now. That's table stakes — and on its own it's almost useless as a signal. The value isn't what is trending; it's what's moving : which topic just jumped 30 places in 20 minutes, which is decaying, which is brand-new this hour. That's velocity , and velocity is where the signal lives — for brand-crisis teams, consumer-trend desks, and anyone modelling attention in China. The catch: a single scrape can't tell you velocity. You have to diff the board against its own past, reliably, run after run. That's a stateful pipeline, and it has a few non-obvious gotchas. Here's the shape of the problem and how to handle it. Why a snapshot isn't enough Rank-right-now tells you nothing about trajectory. "#7" could be a topic on its way to #1 or one fading out of the top 50 — same row, opposite meaning. To act on a trend you need the derivative : direction, speed, and how long it's been climbing. None of that is in a single pull. The trending-delta problem Three things make "just diff the board" harder than it looks: Key by identity, not position. You can't track a topic by its rank — rank is the thing that changes. Key by the topic itself (its text/keyword) or your deltas are nonsense. State has to survive between runs. A scheduled scrape is stateless by default — each run starts cold. To compute "this rose 12 places since 30 minutes ago," you must persist the previous board and reload it next run, keyed so independent schedules don't overwrite each other. The board churns. Topics appear, peak, and fall off. You want each tagged new / rising / falling / steady / dropped , plus how long it's been on the board and its running peak — none of which exist in the raw snapshot. How to handle it (the pattern) current = pull_board () # [{topic, rank, heat}, ...] previous = load_state ( key ) # durable store that persists across runs for t in current : prev = previous . get ( t . topic ) # match o

lakehouse has two storage areas ; Files and Tables Files Store structured, queryable data by sql Supports schema definitions and ACID transactions Tables Stores Raw or semi-structured data(CSV, parquet, JSON) No schema support Flexible for data explorations Schema allows for logical ordering of data on business functions or domain (sales,marketing etc) A dbo schema is enabled by default once a lakehouse is created Schema-enabled lakehouses also support schema-level permissions and cross-workspace queries using the four-part namespace Lakehouse mode : Lakehouse Explorer and SQL analytics endpoint Lakehouse Explorer: Allows managing, Update, create, upload of data.You can switch between tables in the lakehouse SQL anlytics endpoit : Does not allow modifying of the underlying data. You can query using TSQL at read only mode. Loading data into lakehouse: Upload data into files/ folders on the explorer Load into delta tables (no code) Transform using power query in dataflow gen2 INgest into notebooks using apache spark (programmatically) Use Copy data to move data into differnt sources using data factory pipelines -Shortcuts allow you to reference external data reducing copies. Access is managed by One Lake. Schema shortcuts map an entire schema to a folder of Delta tables in another lakehouse. SQL analytics endpoint provides read-only access to lakehouse tables using T-SQL queries. SQL USE CASES : adhoc queries, BI connections to power bi or azure data studio, Data validation You can use SQL views to store reusable query logic. Views are useful when you need to apply business rules, simplify complex joins, or provide curated data for downstream consumers. You can use Spark SQL for SQL-like queries or PySpark for programmatic data manipulation in Notebooks. Spark SQL works well for familiar SQL patterns. PySpark provides greater flexibility for complex transformations and integration with Python libraries. Power BI is the business intelligence and reporting layer in Fabr

Data Visualizer Live Demo 🌐 Try it live: https://datavisualizer.urlmediainspector.dev/ What It Is Data Visualizer is a visual workspace where developers can explore, transform, execute, and understand data using interconnected nodes on an infinite canvas. Instead of jumping between API tools, JSON viewers, spreadsheets, code editors, schema inspectors, and visualization platforms, everything happens inside a single interactive environment. Each node represents a specific capability and can be connected together to create powerful workflows for data exploration, processing, automation, and analysis. Key Features Infinite Visual Workspace Work on an unlimited canvas where data, code, APIs, documents, and visualizations can be organized as connected workflows instead of isolated files and tabs. API Exploration Connect to APIs, inspect responses, analyze payloads, and build reusable visual pipelines for data processing. JSON & YAML Visualization Navigate deeply nested structures through interactive visual representations that make complex data easier to understand. JavaScript & TypeScript Execution Run JavaScript and TypeScript directly inside workflow nodes to transform, filter, and manipulate data in real time. Browser-Based Python Runtime Execute real Python entirely in the browser without requiring local installations or external servers. CSV & Dataset Analysis Import and explore tabular data visually, making it easier to inspect records, understand relationships, and process large datasets. Schema Exploration Visualize schemas and nested structures to quickly understand how data is organized and connected. PDF, Image & Video Support Work with documents and media assets directly inside the workspace without constantly switching applications. Visual Data Pipelines Create workflows by connecting nodes together, allowing data to flow naturally between APIs, transformations, code execution, schemas, and visualizations. Interactive Data Transformation Modify and reshape

DuckLake Spec, pg_background 2.0, and pgsql_tweaks 1.0.3 Enhance Database Ecosystem Today's Highlights This week's highlights include DuckDB's new DuckLake specification for simplified dataframe integration with data lakes, alongside key updates from the PostgreSQL community. We cover pg_background 2.0 for safer asynchronous SQL execution and the release of pgsql_tweaks 1.0.3 for enhanced monitoring and performance tuning. The DuckLake Spec Is so Simple, Even a Clanker Can Build One for Dataframes (DuckDB Blog) Source: https://duckdb.org/2026/05/04/ducklake-dataframe.html The DuckDB team has unveiled the DuckLake v1.0 specification, a significant step towards simplifying data lake interactions with dataframes. This specification aims to provide a robust yet straightforward framework for reading and writing dataframes directly from and to data lake storage, emphasizing ease of implementation. The announcement highlights the specification's simplicity, so much so that even AI can be leveraged to generate compatible dataframe reader/writer tools. This initiative promises to democratize data lake access, allowing developers and data engineers to integrate DuckDB's powerful analytical capabilities with their data lake architectures more seamlessly. By defining a clear standard, DuckLake facilitates the creation of a vibrant ecosystem of tools and connectors, enabling efficient data processing directly within the data lake context without complex ETL pipelines. This development positions DuckDB as an even more versatile tool for analytical workloads, bridging the gap between local data processing and large-scale data lake environments. The ability to easily build data lake connectors, potentially even with AI assistance, marks a notable shift towards more accessible and integrated data workflows. This could streamline operations for data scientists and analysts who frequently work with large datasets stored in various data lake formats, allowing them to leverage DuckDB's

Microsoft shut down dozens of GitHub code repositories for Azure and AI coding tools after a reported hack.

If you work with Brazilian companies — as an accountant, credit analyst, or anyone processing PJ clients at scale — here's a practical automation approach using free public data. What you can verify automatically For any CNPJ, public data gives you: Situação cadastral : ATIVA, BAIXADA, INAPTA, SUSPENSA — critical for invoice validation Razão social : legal name for contract matching CNAE : is this company allowed to do what they claim? QSA : who are the actual partners/directors? Data abertura : how old is the company? The data 65M+ CNPJs from Receita Federal, indexed and searchable at Jurídico Online . Free. Also available as a Python package: pip install juridico-online from juridico_online import empresa_url , buscar_url # Get company page URL for a CNPJ url = empresa_url ( " 00.000.000/0001-91 " ) print ( url ) # https://juridicoonline.com.br/empresa/00000000000191 # Search by company or partner name search = buscar_url ( " Magazine Luiza " ) print ( search ) Checks worth automating 1. Situação ATIVA before accepting any invoice INAPTA or BAIXADA companies cannot legally issue NF-e. 2. CNAE vs service being billed A company with CNAE "comércio de alimentos" billing for software development is a red flag. 3. Company age vs contract value A 3-month-old company offering a R$500k contract deserves extra scrutiny. 4. Shared partners across suppliers If two suppliers share directors, that's a conflict of interest. Search partner names at juridicoonline.com.br to see all companies they control. Integration patterns ERP/AP : validate CNPJ status before releasing payment Onboarding : auto-fill razão social when client enters CNPJ Batch audit : cross-check your vendor list quarterly Monitoring : alert if a key supplier's CNPJ changes status The data is public, free, and updated regularly. No excuse to check manually at scale.

Short answer first: use Supabase if you want Postgres plus auth, realtime, storage, and a dashboard as one managed bundle. Self-host Postgres – or use a managed Postgres – if you mostly need a database and your app already handles its own auth and logic. The choice is not really "Postgres vs Supabase". It's whether you need the extra layers Supabase puts on top of Postgres. Supabase is not a database Supabase runs on PostgreSQL, but it's a stack of services around it: Postgres – the actual database Auth – user signup, login, JWT tokens Realtime – live updates over websockets Storage – an S3-style file store Edge Functions – serverless functions Studio – dashboard + auto-generated REST/GraphQL API So "self-host Postgres or use Supabase" compares a plain database to a full backend. The honest question: do you need those extra layers, or just the database underneath them? A quick test: You use Supabase Auth, Storage, and Realtime → Supabase earns its place. You use one of them → it's replaceable. You use none and treat it as "Postgres with a nice dashboard" → you want plain Postgres. Side-by-side comparison Factor Supabase (managed) Self-hosted Supabase Plain Postgres (managed or self-hosted) Database engine PostgreSQL PostgreSQL PostgreSQL Built-in auth Yes Yes No (bring your own) Realtime / websockets Yes Yes No File storage Yes Yes No Dashboard + auto API Yes Yes No (use any SQL client) Backups Managed (limits by plan) You manage Managed or you manage Cost shape Metered, grows with usage Server cost + your time Database only Self-host effort None High (many containers) Low–medium Lock-in Medium–high Medium Very low The lock-in point decides it for many teams. Your data is standard Postgres in every option ( pg_dump portable). The lock-in is everything else: Auth tokens, Storage paths, Supabase-specific RLS policies, Edge Function code. The more Supabase-specific features you adopt, the harder the exit. When each option wins Pick managed Supabase when: You're startin

The bill is expected to blanket-ban companies and startups from selling people's precise location data across the state.

Google says Gemma 4 12B is "designed to bring agentic, multimodal intelligence directly to your laptop", further noting that the new model can be combined with Google AI Edge to "build and experiment locally, on everyday machines". This integration allows for a wide range of capabilities, from autonomous data processing to generating visual insights and even building webpages or executing tools. By Sergio De Simone

The era of passive data analytics is over. Today, the most forward-thinking data teams aren't just...

In this article, the author examines how AI is transforming phishing from a manual, targeted activity into an automated and scalable attack model. The article breaks down each stage of the phishing lifecycle, showing how AI improves reconnaissance, profiling, content generation, delivery, and interaction, while outlining layered defenses that combine controls, processes, and user awareness. By Marco Rizzi