标签：#Data

Introduction Excel is a spreadsheet application developed by Microsoft that helps users organize, analyze and visualize data. It is used by businesses, organizations, researchers and students worldwide because it makes working with data easier and more efficient. Business Decision Making One of the ways Excel is used in real-world data analysis is in supporting business decision-making. Companies collect data such as customer information, financial transactions and sales records. Excel helps in organizing and analyzing this data using tools such as formulas and PivotTables. This makes it easier to identify trends and patterns in business performance, such as which products to stock and when to restock them. For example, a supermarket can analyze the monthly sales in Excel to identify the best-selling products and ensure that they remain in stock. Marketing Performance Excel is also used to analyze marketing performance. Businesses use it to track data from marketing campaigns such as website visits, social media engagement and sales conversions. This information is organized using charts and reports, which help evaluate which strategies are producing the best results. This allows companies to allocate their resources more effectively and improve future campaigns based on data rather than assumptions. As a result, Excel plays an important role in helping businesses understand their customers and improve the effectiveness of their marketing efforts. Financial Reporting Excel is widely used in financial reporting. It helps businesses to organize and analyze financial statements such as income statements, cash flow reports and balance sheets. It is also used to record transactions, calculate totals, and generate summaries that show the financial health of the business. By using built-in formulas and functions, accountants can quickly compute profits, expenses, taxes and forecasts with a high level of accuracy. Excel also allows the creation of financial charts and dashb

Introduction In today's fast-paced business environments, data is considered the cornerstone of decision-making, policy formulation, and other organizational needs. MS Excel is a robust spreadsheet developed by Microsoft for organizing, analyzing, and visualizing data in rows and columns. In the data science and analytics domain, MS Excel is critical for analyzing and managing data to generate insights that enhance decision-making. Excel's polarity is characterized by its ease of use, flexibility, automation, and visualization. Ways Excel Is Used in Real-World Data Analysis Across the data science and analytics domain, MS Excel is frequently employed in the following ways; a) Data Cleaning and Preprocessing At the beginning of every data science and analytics project, data cleaning is required, and MS Excel is the primary tool. Typical Excel features and functions applied during data cleaning include Text to Columns, Remove Duplicates, Find and Replace, and Power Query. b) Exploratory Data Analysis Before performing data science and analytics activities, it is crucial to understand the dataset at hand, its structure, and trends. MS Excel features Pivot Tables, Pivot Charts, and Slicers that provide instant aggregation, sorting, and visualizations. c) Data Analysis and Reporting Modern organizations and businesses operate based on insights generated from data. MS Excel features such as pivot tables, charts, and conditional formatting help data analysts analyze and visualize data for clear, actionable insights that enhance decision-making. MS Excel Features or Formulas The typical MS Excel features and formulas employed in the data science and analytics domain include the following. Data Cleaning Functions Function Purpose Example Result UPPER() Converts text to uppercase =UPPER("john") JOHN LOWER() Converts text to lowercase =LOWER("JOHN") john PROPER() Capitalizes the first letter of each word =PROPER("john doe") John Doe TRIM() Removes extra spaces from text =TRIM(

I've been building a database engine from scratch in Rust, and I recently finished the lexer. The lexer itself wasn't the most interesting part. What I found more valuable was how my design evolved as I learned more about Rust and how compilers and database systems are typically implemented. My First Approach When I started, I stored the input as a Vec<char> . It felt straightforward because I could access characters directly without worrying about UTF-8 boundaries. I also represented identifiers like this: Ident ( String ) At first glance, this seems perfectly reasonable. Every identifier token carries its own text, making it easy for the parser to consume. The Problem As the lexer grew, I started asking myself a simple question: The identifier already exists in the original SQL query. Why am I allocating another string and copying the same data into every token? For a query like: SELECT username , email FROM users ; the source text already contains: username email users Creating separate String allocations for each identifier means duplicating data that already exists. I also learned an important detail about Rust enums. The size of an enum is influenced by its largest variant. Once variants start carrying additional data, every token instance becomes larger than it otherwise needs to be. Moving to a Span-Based Design Instead of storing identifier text directly inside tokens, I switched to storing only the token kind: Ident along with source location information: Span { start , end , line , column , } Now the token only answers two questions: What is this token? Where did it come from? If the parser needs the actual identifier text, it can recover it directly from the original SQL source using the stored byte range. Replacing Vec<char> with &str The second design change was moving away from: Vec < char > and operating directly on: & str using lifetimes. Instead of creating another collection containing the entire input, the lexer now walks over borrowed source tex

It's important to monitor Dataform - jobs executed by Dataform can be the primary source of BigQuery costs in a modern data platform. Forgetting to incrementalise a table, using a table instead of a view in the wrong place or performing complex window functions on a large table can all incur large costs and long run times. Using the WorkflowInvocationAction for each job we can extract its BigQuery Job ID, then extract key metadata for each BigQuery job by querying INFORMATION_SCHEMA.JOBS_BY_PROJECT , before writing the output back to BigQuery so that it can be analysed (maybe even by transforming it in Dataform). from google.cloud import dataform_v1 from google.cloud import bigquery from datetime import datetime # ------------------------------------------------------------ # CONFIG # ------------------------------------------------------------ PROJECT_ID = " my-project " REGION = " europe-west2 " REPOSITORY_ID = " analytics " WORKFLOW_INVOCATION_ID = " 123456789 " BQ_REGION = " region-europe-west2 " OUTPUT_TABLE = " my-project.raw_dataform_monitoring.raw_dataform_bigquery_metrics " # ------------------------------------------------------------ # CLIENTS # ------------------------------------------------------------ dataform = dataform_v1 . DataformClient () bq = bigquery . Client ( project = PROJECT_ID ) repository = dataform . repository_path ( PROJECT_ID , REGION , REPOSITORY_ID ) invocation_name = f " { repository } /workflowInvocations/ { WORKFLOW_INVOCATION_ID } " # ------------------------------------------------------------ # 1. GET WORKFLOW INVOCATION ACTIONS → EXTRACT JOB IDS # ------------------------------------------------------------ job_ids = set () actions = dataform . list_workflow_invocation_actions ( parent = invocation_name ) for action in actions : # only BigQuery actions contain job metadata if hasattr ( action , " bigquery_action " ) and action . bigquery_action : if action . bigquery_action . job_id : job_ids . add ( action . bigquery_action

[While preparing for interviews, I realized I had a strange habit. I would solve a problem, get stuck, open the solution, understand it, and move on feeling productive. A few days later, I couldn’t solve a similar problem on my own. The issue wasn’t lack of practice. The issue was that I was consuming solutions faster than I was developing problem-solving skills. So I changed my approach. Instead of looking for answers, I started forcing myself to think longer, write down my ideas, identify where I was stuck, and only then seek guidance. That worked much better. But I couldn’t find a tool that supported this style of learning. Most platforms either: Give you the answer. Give you the editorial. Give you AI that writes the code for you. So I started building my own. The goal was simple: An AI coach that guides the thought process instead of generating the solution. Over time I added: DSA practice System Design preparation Low-Level Design preparation Company-wise interview questions Topic-wise strength and weakness analysis Personalized revision lists The interesting part wasn’t building it. The interesting part was realizing that interview preparation is less about collecting solutions and more about training how you think. What has helped you improve more during interview prep? Reading solutions? Or struggling with the problem first? Sde vault - https://sdevaultweb.onrender.com/

Cloudflare recently described how a slowdown in its billing pipeline was traced to contention inside the query planning stage of ClickHouse. The team profiled the bottleneck and patched ClickHouse to replace an exclusive lock with a shared lock, drop the per-query copy of the parts list, and improve part filtering. By Renato Losio

TL;DR — The DIFP protocol was designed to be data-compact and geo-aware from day one. We recently discovered it maps almost perfectly onto the Nostr event format. Here's how, and why it matters for decentralized food infrastructure. Background: What Is DIFP? DIFP (Djowda Interconnected Food Protocol) is an open protocol designed to sync food product data across distributed nodes — compactly, efficiently, and with geo-location awareness built in by default. One of its core design decisions is the PAD system (Preloaded Asset Distribution): Apps ship with a preloaded asset pack — item metadata, compressed images, category structure — all bundled at install time. Only price and availability need to travel over the wire during sync. This means the data footprint per product is tiny. Very tiny. Enter Nostr Nostr is a simple, open protocol for decentralized communication. One of its key specs: events support up to 64 KB of content . When we started exploring Nostr as a potential transport layer, we ran the numbers — and the fit was surprisingly clean. The Math: Products Per Event Baseline encoding A product represented with three fields: { "id" : 500 , "available" : true , "price" : 30000 } At this level of verbosity, a single 64 KB Nostr event can hold approximately: ~1,500 – 2,000 products Already useful. But we can do better. Optimized encoding Two key optimizations: 1. Drop the availability key — If a product entry exists in the JSON, it's available. If it's absent, it's not. No boolean needed. 2. Drop the field names — Instead of {"id": 500, "price": 30000} , just store: 500,30000 Field mapping is handled at the app level, not the protocol level. The device knows position 0 is the product ID, position 1 is the price (in smallest currency unit, e.g. cents). Result ~6,000 – 7,000 products per single Nostr event Possibly more, depending on the price distribution and ID ranges in a given catalog. Geo-Discovery: MinMax99 Cells DIFP uses a geo-cell system called MinMax99 to

I've been frustrated with SQL learning resources for a while. Most are either: Dry reference docs Toy exercises with no context ("SELECT * FROM employees") Paid platforms with paywalls after level 3 So I built SQLwak — a free, browser-based SQL game where you're hired as a Graduate Analyst at Lion City Bank , a fictional Singapore bank. How it works Instead of abstract exercises, every challenge is a real business request from a colleague: "The Operations team needs all Central region branches for an upcoming audit." "Risk wants customers with credit scores below 600 who have active loans." "Finance needs vessels ranked by cargo revenue — use window functions." You write actual SQL against a realistic 9-table banking database and get immediate feedback. 57 levels across 4 tiers Tier Skills 🟢 Foundational SELECT, WHERE, ORDER BY, LIMIT 🟡 Intermediate JOINs, GROUP BY, HAVING, subqueries 🔴 Advanced CTEs, multi-table aggregations ⚫ Expert Window functions (RANK/DENSE_RANK OVER PARTITION BY), UNION ALL, compound CTEs The database schema Lion City Bank has two divisions: Retail Banking: customers, accounts, transactions, loans, branches, products Maritime Trade Finance (Advanced/Expert levels): vessels, cargo_shipments, trade_finance_facilities — covering voyages between Singapore, Port Klang, Bangkok, Jakarta, and Ho Chi Minh City. The maritime division exists because Singapore is a major trade hub. It makes the Expert levels genuinely interesting — you're ranking vessels by cargo revenue and analyzing trade finance utilisation rates, not just counting rows. Technical details Next.js 15 + TypeScript + Tailwind CSS SQLite via WebAssembly — all query execution is client-side, no backend needed Deployed on Vercel Fully open source: github.com/martinl5/sqlwak No signup. No download. Just SQL. Open the link and start writing queries: sqlwak.vercel.app Would love feedback on difficulty progression, new level ideas, or schema additions. What SQL concepts do you wish you'd pract

Introduction A traditional database. That is what many who have not really interacted with Excel to a great extent would define it as in its most basic form. Not that they are wrong, only that is the scope their utilization of Excel covers. Mostly record keeping, basic operations, and data representation. But for those whose utilization scope of Excel is broader, we definitely know better. This underestimation of Excel is a grave mistake for anyone considering themselves as tech-oriented, especially for anyone dealing with data operations, be it simple record keeping or complex concepts involving data. What is Excel A spreadsheet program or tool that facilitates data organization, analysis, and visualization through mathematical operations, chart creation, and building financial models. Real-world application of Excel in Data Analytics Reporting and visualisation Excel facilitates data representation in the form of charts(bar charts, pie charts, line graphs) and dashboards. Businesses and organisations utilize this to get an organised, more insightful, and simplified view and report of their raw data. Financial Accounting Excel's provision for mathematical operations, functions, and formulas in analysis facilitates financial accounting. Balance sheets and income statements preparation, budgeting, and expense tracking are just some of the ways Excel can be used in accounting. Decision-Making Businesses and organisations heavily rely on analysis to support their decision-making. Excel helps in the analysis through different data metrics comparisons, e.g., sales across seasons and locations, forecasting, and tracking key performance indicators. This helps businesses make the best decisions based on the insights gathered from the analysis. Beginner Excel Features and Formulas for Data Analysis Learnt so far Sort and Filter By applying the Filter feature for each column, data in specific columns can not only be sorted from newest to oldest, but also be filtered based on

DuckDB Integrates Lance Lakehouse; SQLite CVE Fix; Postgres 19 Beta on K8s Today's Highlights This week, DuckDB introduces integrated vector and hybrid search with the Lance lakehouse format, enabling advanced AI workloads directly from SQL. Meanwhile, SQLite addresses a significant security vulnerability with a recent fix, and a new guide helps users test PostgreSQL 19 beta efficiently in Kubernetes clusters. Test-Driving the Lance Lakehouse Format in DuckDB (DuckDB Blog) Source: https://duckdb.org/2026/05/21/test-driving-lance.html DuckDB has announced a new integration allowing users to test-drive the Lance lakehouse format, a design specifically geared towards AI workloads. This collaboration between LanceDB and DuckLabs brings fast vector and hybrid search capabilities directly into DuckDB SQL, significantly streamlining data analysis for machine learning applications. The integration means users can perform complex queries involving vector embeddings and traditional tabular data without needing to move data between different systems or tools. This is a crucial development for data scientists and engineers who rely on efficient data access and processing for large-scale AI projects. The ability to perform sophisticated searches within the familiar DuckDB environment enhances productivity and reduces the operational overhead typically associated with managing specialized vector databases. This new feature democratizes access to advanced data structures for AI, making it simpler to build and prototype machine learning models on massive datasets. By supporting an open lakehouse format, DuckDB continues to position itself as a versatile analytical database, bridging the gap between traditional data warehousing and emerging AI data needs. Users are encouraged to try out this integration, which promises to unlock new possibilities for data exploration and feature engineering using a unified SQL interface. It's an excellent example of how embedded analytical databases

A post titled "Learn SQL Once, Use It for 30 Years" hit the front page of r/programming this week (307 points, 48 comments). The claim sounds like the kind of thing a database vendor would put on a billboard, so I went looking for the part that holds up. It turns out the longevity is not marketing. It is a property of how the language was designed, and it is the reason SQL is one of the few skills on a developer's resume that does not quietly expire. I run a site that compares developer tools, which means I spend a lot of time watching technologies rise, peak, and get replaced. Most of what you learn in this field has a half-life measured in single-digit years. The framework you mastered in 2019 is legacy by 2024. SQL is the strange exception, and the reasons are worth understanding before you decide where to spend your next month of learning. Where the staying power comes from SQL did not start as a language. It started as a math paper. In 1970, Edgar Codd published "A Relational Model of Data for Large Shared Data Banks," which proposed organizing data into tables of rows and columns with formal rules for combining them. IBM built a query language on top of that model in the mid-1970s, called it SEQUEL, and later renamed it SQL after a trademark conflict. The important detail is the order: the model came first, the language second. SQL is a surface over a mathematical foundation that has not needed to change. That foundation is why the skill compounds instead of decaying. When you learn SQL, you are not memorizing one vendor's API. You are learning the relational model, and the model is the same whether the data sits in Postgres, MySQL, SQLite, Oracle, or SQL Server. A join is a join everywhere. Move from one database to another and the syntax shifts at the edges, but the way you think about the problem carries over intact. Compare that to a frontend framework, where moving stacks means relearning how to think, not just how to type. Declarative is the whole trick

Key Use Cases Power BI Visual Monitoring can be used for: power bi visual monitoring power bi report visual monitoring visual regression testing for Power BI power bi screenshot monitoring monitoring Power BI visuals visual monitoring for Power BI Report Server automated Power BI dashboard validation visual correctness control for BI reports Power BI Visual Monitoring: Automatically Detecting Broken Visuals in Power BI Reports In large Power BI environments, analytics teams often face the problem of silent regressions : even minor changes in data or models can break individual visuals without any obvious errors. Report owners frequently don’t notice that a visual has stopped rendering or is showing incorrect data — this can happen due to changes in data source structure, access rights, deleted fields, broken measures, or refresh failures. Manually checking hundreds of report pages across multiple dashboards in such conditions is extremely inefficient and nearly impossible. We, a team of BI developers and analysts, encountered this pain point during a large analytics implementation project and decided to create a solution for automated Power BI visual monitoring . Project Source Code: GitHub: https://github.com/svergio/Power-bi-report-visual-monitoring Documentation: https://svergio.github.io/Power-bi-report-visual-monitoring/ Wiki: https://github.com/svergio/Power-bi-report-visual-monitoring/wiki Why Standard Power BI Tools Don’t Solve the Problem Standard Power BI tools such as Usage Metrics and Performance Analyzer help analyze report usage and performance but do not detect visual issues. For example, built-in usage metrics show “how those dashboards and reports are being used” — number of views, popular reports, and who is viewing them. These metrics are important for assessing analytics adoption, but they say nothing about whether the visuals themselves are displaying correctly. Similarly, Performance Analyzer shows load times for each visual, helping identify s

Developer felt "beaten up," with "no choice" but to shrink data center.

Managing a multi-million product catalog on Magento presents unique challenges around performance, scalability, and operational efficiency. At Rave Digital, we recently undertook a Magento performance optimization project for a large-scale eCommerce merchant struggling with slow site speed, infrastructure bottlenecks, and backend instability. This use case breakdown details how we modernized their Magento architecture, optimized database performance, and scaled infrastructure to deliver a stable, high-speed shopping experience. This post is tailored for eCommerce managers, directors, and Magento merchants—especially those running Adobe Commerce or Magento Open Source platforms—who want to understand practical strategies for Magento architecture scaling and performance tuning for large catalogs. The Problem: Performance Bottlenecks in a Complex Magento Environment: Our client operated an enterprise Magento store with a multi-million product catalog. Despite Magento’s robust capabilities, the site suffered from: Slow page load times impacting user experience and SEO Scalability challenges as product volume and traffic grew Infrastructure bottlenecks causing backend instability and downtime Complex integrations and manual processes limiting operational efficiency Platform limitations in handling large catalog management and real-time inventory updates These issues collectively threatened the site’s ability to support growth and deliver a seamless customer experience. The client sought a comprehensive Magento platform modernization to address these challenges. Context: Why Magento Architecture and Infrastructure Matter Magento’s flexibility and extensibility make it ideal for enterprise eCommerce, but large catalogs require careful architecture and infrastructure planning. Key technical pain points include: Database performance under heavy read/write loads Indexing delays and cache invalidation impacting site speed Integration complexity with third-party systems and API

The Australian data center operator plans to set up 5GW of capacity in India.

Dropbox has unveiled Nova, an internal platform designed to orchestrate and operationalize AI coding agents across the company's engineering workflows. By Craig Risi

TL;DR Reconciliation is the part of a payments stack nobody architects for on day one and everyone pays for on day 200. The job: prove that every internal transaction matches the acquirer's settlement file, in the right currency, with the right fees, on the right value date — or surface the diff fast. The mechanics: normalize files → land into an events table → project to a read model → diff against the internal read model → buckets for ops to resolve. The boring details (file formats, fee parsing, FX rounding, value dates) are where 90% of the work lives. If you've ever opened a CSV from an acquirer at the end of the month, sorted by amount, and tried to "just match it in Excel" — yes, this post is for you. What "reconciled" actually means A transaction is reconciled when, for the same logical payment, three views agree: What you sent — your internal record of the charge/payout (your read model). What the acquirer says happened — their settlement file or API report. What the bank actually credited / debited — the bank statement. Disagreements are normal. Persistent disagreements are how you lose money slowly and never know. The shape of a settlement file Across the major acquirers, settlement files look broadly similar — and broadly different in the places that matter: Field Variants you'll see Transaction reference acquirer's transaction_id , sometimes plus a merchant_reference round-tripped from you Gross amount minor units / decimal; transaction currency vs settlement currency Fees inline per-row, or aggregated at the file footer, or in a separate fees file FX inline rate vs separate FX file; sometimes only the converted amount Value date when the bank actually moves money — often T+1/T+2 from event date Adjustments refunds, chargebacks, fee corrections, reserves — usually mixed in Encoding UTF-8 if you're lucky; CP1252 / fixed-width / SWIFT MT940 if you're not Granularity one row per transaction or daily aggregates per merchant or both There's no industry-clean

When we started working on Krenalis , we spent a lot of time reviewing how customer data typically flows through a modern data stack. One pattern kept showing up often enough that we started questioning it. In many modern stacks, customer data already lands in a warehouse. Yet we often copy that same data into a CDP before we can start building customer profiles. During one of those discussions, someone asked a question that sounded almost naive: Why are we moving all this data in the first place? Nobody had a particularly strong answer ready. The answer was mostly: Because that's how CDPs work. We expected the question to have an obvious answer. It didn't. The warehouse is no longer just for analytics Over the last few years, the role of the data warehouse has changed significantly. Warehouses are no longer just analytical systems. They're increasingly becoming the place where organizations centralize the context used by applications, AI agents, copilots, and business processes. Customer data from systems like Shopify, Stripe, CRMs, support platforms, and internal applications often ends up there long before anyone starts thinking about segmentation or activation. In many organizations, the warehouse is already the place where teams answer questions about customers, revenue, retention, and product usage. That made us wonder: If the warehouse is already becoming the operational center of the data stack, why does customer identity usually live somewhere else? Consider a customer who buys through Shopify, pays through Stripe, opens support tickets in Zendesk, and uses the product under a different email address. In many organizations, all of those records already end up in the warehouse. Yet building a unified profile often requires exporting that same data into another platform before identity can be resolved. The cost of another copy To be clear, data duplication is not inherently bad. Most software systems rely on some form of replication, caching, or denormalizati

TypeORM 1.0 is the first major release of the open-source TypeScript and JavaScript ORM since its inception in 2016. This version modernizes platform requirements, removes deprecated APIs, and introduces numerous bug fixes and new features. TypeORM now supports ECMAScript 2023, dropping older Node.js versions and dependencies while enhancing security and migration processes. By Daniel Curtis

Introduction Excel is one of the most used tools for data analysis. It allows beginners like myself to easily clean, organize, analyze and visualize data.Excel enables users to work with large datasets and extract meaningful insights without requiring advanced technical skills. What is Excel Excel is a spreadsheet that allows you to collect, organize, analyze, calculate, and visualize data efficiently.Despite the emergence of other data analysis tools like SQL and Power BI, Excel remains one of the most widely used tools for both personal and professional data management.This can be credited to its ease of access, learning, and use. Ways Excel is used in real-world data analysis This week, I had the opportunity to explore how Excel is used in real-world data analysis.I discovered that Excel is not just a basic spreadsheet tool, but a powerful application that helps make sense of data and support decision-making. Data organization and cleaning Excel is used to structure raw data, remove duplicates, and fix errors. This improves data quality, making it easier to analyze and more reliable for decision-making.This improves data quality, making it easier to analyze and more reliable for decision-making. Financial Excel is commonly used in finance to create budgets, calculate profits and losses, and monitor expenses.It helps organizations keep accurate financial records and understand their financial situation. Business decision-making Businesses use Excel to track sales, compare performance over time, and identify trends.This helps managers understand what is working well and what needs improvement. Excel features and formulas In just a week, I have learned several Excel formulas that simplify data management and make working with data more efficient. SUM function The SUM function is used to add a range of values together in Excel, making it one of the most essential tools for quick calculations.It's used to automatically add a range of numerical values together, elimina