标签：#datascience

Introduction Excel is a spreadsheet application developed by Microsoft that helps users organize, analyze and visualize data. It is used by businesses, organizations, researchers and students worldwide because it makes working with data easier and more efficient. Business Decision Making One of the ways Excel is used in real-world data analysis is in supporting business decision-making. Companies collect data such as customer information, financial transactions and sales records. Excel helps in organizing and analyzing this data using tools such as formulas and PivotTables. This makes it easier to identify trends and patterns in business performance, such as which products to stock and when to restock them. For example, a supermarket can analyze the monthly sales in Excel to identify the best-selling products and ensure that they remain in stock. Marketing Performance Excel is also used to analyze marketing performance. Businesses use it to track data from marketing campaigns such as website visits, social media engagement and sales conversions. This information is organized using charts and reports, which help evaluate which strategies are producing the best results. This allows companies to allocate their resources more effectively and improve future campaigns based on data rather than assumptions. As a result, Excel plays an important role in helping businesses understand their customers and improve the effectiveness of their marketing efforts. Financial Reporting Excel is widely used in financial reporting. It helps businesses to organize and analyze financial statements such as income statements, cash flow reports and balance sheets. It is also used to record transactions, calculate totals, and generate summaries that show the financial health of the business. By using built-in formulas and functions, accountants can quickly compute profits, expenses, taxes and forecasts with a high level of accuracy. Excel also allows the creation of financial charts and dashb

Introduction In today's fast-paced business environments, data is considered the cornerstone of decision-making, policy formulation, and other organizational needs. MS Excel is a robust spreadsheet developed by Microsoft for organizing, analyzing, and visualizing data in rows and columns. In the data science and analytics domain, MS Excel is critical for analyzing and managing data to generate insights that enhance decision-making. Excel's polarity is characterized by its ease of use, flexibility, automation, and visualization. Ways Excel Is Used in Real-World Data Analysis Across the data science and analytics domain, MS Excel is frequently employed in the following ways; a) Data Cleaning and Preprocessing At the beginning of every data science and analytics project, data cleaning is required, and MS Excel is the primary tool. Typical Excel features and functions applied during data cleaning include Text to Columns, Remove Duplicates, Find and Replace, and Power Query. b) Exploratory Data Analysis Before performing data science and analytics activities, it is crucial to understand the dataset at hand, its structure, and trends. MS Excel features Pivot Tables, Pivot Charts, and Slicers that provide instant aggregation, sorting, and visualizations. c) Data Analysis and Reporting Modern organizations and businesses operate based on insights generated from data. MS Excel features such as pivot tables, charts, and conditional formatting help data analysts analyze and visualize data for clear, actionable insights that enhance decision-making. MS Excel Features or Formulas The typical MS Excel features and formulas employed in the data science and analytics domain include the following. Data Cleaning Functions Function Purpose Example Result UPPER() Converts text to uppercase =UPPER("john") JOHN LOWER() Converts text to lowercase =LOWER("JOHN") john PROPER() Capitalizes the first letter of each word =PROPER("john doe") John Doe TRIM() Removes extra spaces from text =TRIM(

Introduction A traditional database. That is what many who have not really interacted with Excel to a great extent would define it as in its most basic form. Not that they are wrong, only that is the scope their utilization of Excel covers. Mostly record keeping, basic operations, and data representation. But for those whose utilization scope of Excel is broader, we definitely know better. This underestimation of Excel is a grave mistake for anyone considering themselves as tech-oriented, especially for anyone dealing with data operations, be it simple record keeping or complex concepts involving data. What is Excel A spreadsheet program or tool that facilitates data organization, analysis, and visualization through mathematical operations, chart creation, and building financial models. Real-world application of Excel in Data Analytics Reporting and visualisation Excel facilitates data representation in the form of charts(bar charts, pie charts, line graphs) and dashboards. Businesses and organisations utilize this to get an organised, more insightful, and simplified view and report of their raw data. Financial Accounting Excel's provision for mathematical operations, functions, and formulas in analysis facilitates financial accounting. Balance sheets and income statements preparation, budgeting, and expense tracking are just some of the ways Excel can be used in accounting. Decision-Making Businesses and organisations heavily rely on analysis to support their decision-making. Excel helps in the analysis through different data metrics comparisons, e.g., sales across seasons and locations, forecasting, and tracking key performance indicators. This helps businesses make the best decisions based on the insights gathered from the analysis. Beginner Excel Features and Formulas for Data Analysis Learnt so far Sort and Filter By applying the Filter feature for each column, data in specific columns can not only be sorted from newest to oldest, but also be filtered based on

Key Use Cases Power BI Visual Monitoring can be used for: power bi visual monitoring power bi report visual monitoring visual regression testing for Power BI power bi screenshot monitoring monitoring Power BI visuals visual monitoring for Power BI Report Server automated Power BI dashboard validation visual correctness control for BI reports Power BI Visual Monitoring: Automatically Detecting Broken Visuals in Power BI Reports In large Power BI environments, analytics teams often face the problem of silent regressions : even minor changes in data or models can break individual visuals without any obvious errors. Report owners frequently don’t notice that a visual has stopped rendering or is showing incorrect data — this can happen due to changes in data source structure, access rights, deleted fields, broken measures, or refresh failures. Manually checking hundreds of report pages across multiple dashboards in such conditions is extremely inefficient and nearly impossible. We, a team of BI developers and analysts, encountered this pain point during a large analytics implementation project and decided to create a solution for automated Power BI visual monitoring . Project Source Code: GitHub: https://github.com/svergio/Power-bi-report-visual-monitoring Documentation: https://svergio.github.io/Power-bi-report-visual-monitoring/ Wiki: https://github.com/svergio/Power-bi-report-visual-monitoring/wiki Why Standard Power BI Tools Don’t Solve the Problem Standard Power BI tools such as Usage Metrics and Performance Analyzer help analyze report usage and performance but do not detect visual issues. For example, built-in usage metrics show “how those dashboards and reports are being used” — number of views, popular reports, and who is viewing them. These metrics are important for assessing analytics adoption, but they say nothing about whether the visuals themselves are displaying correctly. Similarly, Performance Analyzer shows load times for each visual, helping identify s

Introduction Excel is one of the most used tools for data analysis. It allows beginners like myself to easily clean, organize, analyze and visualize data.Excel enables users to work with large datasets and extract meaningful insights without requiring advanced technical skills. What is Excel Excel is a spreadsheet that allows you to collect, organize, analyze, calculate, and visualize data efficiently.Despite the emergence of other data analysis tools like SQL and Power BI, Excel remains one of the most widely used tools for both personal and professional data management.This can be credited to its ease of access, learning, and use. Ways Excel is used in real-world data analysis This week, I had the opportunity to explore how Excel is used in real-world data analysis.I discovered that Excel is not just a basic spreadsheet tool, but a powerful application that helps make sense of data and support decision-making. Data organization and cleaning Excel is used to structure raw data, remove duplicates, and fix errors. This improves data quality, making it easier to analyze and more reliable for decision-making.This improves data quality, making it easier to analyze and more reliable for decision-making. Financial Excel is commonly used in finance to create budgets, calculate profits and losses, and monitor expenses.It helps organizations keep accurate financial records and understand their financial situation. Business decision-making Businesses use Excel to track sales, compare performance over time, and identify trends.This helps managers understand what is working well and what needs improvement. Excel features and formulas In just a week, I have learned several Excel formulas that simplify data management and make working with data more efficient. SUM function The SUM function is used to add a range of values together in Excel, making it one of the most essential tools for quick calculations.It's used to automatically add a range of numerical values together, elimina

Have you ever trained a model that performed beautifully on your training data but fell apart the moment it saw new data? Or perhaps you built something so simple it couldn't even learn the training data properly? These are the classic traps of overfitting and underfitting — and every machine learning practitioner runs into them. In this article, we'll cover what they are, how to detect them, how to fix them, and where the bias-variance tradeoff ties it all together — with real-world examples and code throughout. What is Model Fitting? Model fitting is the process of training a predictive model on a dataset to find the optimal parameters that best capture the underlying patterns in the data. The goal is simple: the model should generalize well to unseen data — not just memorize the training examples. There are three possible outcomes when fitting a model: Outcome Description Good fit Captures underlying patterns, generalizes well Underfitting Too simple, misses patterns even in training data Overfitting Too complex, memorizes noise, fails on new data What is Underfitting? Underfitting occurs when a model is too simple to capture the underlying patterns in the data. It performs poorly on both the training set and on new, unseen data. Think of it like this: imagine asking a child to predict house prices and they only use the rule "all houses cost $100,000." That model ignores all relevant features (size, location, age) and will be wrong almost every time. Why Does Underfitting Occur? Model is too simple : A linear model trying to fit a curved, nonlinear relationship Too few features : Important variables are left out Too much regularization : Penalizing complexity so heavily that the model can't learn anything meaningful Insufficient training : The model hasn't been trained long enough Real-World Example Suppose you're predicting whether an email is spam. If you only use the feature "email length" and ignore word content, sender, and links, your model will underfit —

Hand the same paired before/after dataset (n = 25) to ChatGPT five times. Same prompt: "These are the same subjects measured before and after an intervention. Did their scores change significantly?" Four of the five runs return p = 0.009 from a paired t-test. The fifth run does a Shapiro–Wilk normality check on the differences first, decides they're non-normal, switches to a Wilcoxon signed-rank test, and reports p = 0.000018 . All five reach the same conclusion (significant). But notice what happened: only one run out of five thought to check an assumption you'd want it to check. The other four skipped it. The choice of method — and the test statistic, and the p-value — depended on whether the LLM happened to run an assumption check that time. On borderline data, this is the difference between reject and don't reject. If you're using LLMs for exploratory data analysis on a weekend project, you might shrug. If you're using them for anything that gets cited, gets submitted to a regulator, or gets handed to a clinician, this is a problem. It's a known problem — Cui & Alexander (2026) documented exactly this kind of method-divergence empirically; AIRepr (Zeng et al., 2025) shows the same thing across reproducibility metrics. The current answer in the literature is to constrain the agent so its execution is replayable. But replayability fixes "did we run the same code." It doesn't fix "did we run the right analysis." I've spent the last two months building a different fix. The more interesting half is the architecture. Let me walk through it. The real problem isn't temperature The first reflex is "set temperature=0 ." It's not enough. temperature=0 doesn't make a tool-using agent deterministic across runs. Three reasons: Inference isn't bitwise deterministic, even at temperature=0. Production LLM serving batches requests dynamically, and the attention kernels aren't batch-invariant — so the same input produces different output tokens depending on what other requests it

With shiftbloom studio. I build tools and projects about a variety of experimental approaches to real-world problems. The issue for such use-case often was how most small media systems start out: one big always-on recorder that keeps costing money even when nothing is happening. For live capture you obviously need to stay ready at all times — sometimes you can’t risk losing the first minutes. But for everything else it’s complete overkill. The Core Problem Backfills, VOD downloads, clip imports, repairs and re-encodes are queue work. They can wait a few seconds, run on burst capacity, or even on a regular VPS or laptop. They don’t need the same always-hot infrastructure as the live recorder. That’s why I split the system. Instead of one large monolith, I deployed: Observer cells — only for live streams (time-critical) Harvest cells — for all queue processing (can be delayed) The Three Roles 1. Mothership A small control-plane cron job. It checks queue sizes, currently live channels and running observer tasks, then decides: how many harvest cells should exist right now which channels need an observer cell It’s intentionally simple. The database remains the single source of truth. 2. Observer Cells Each observer cell records exactly one live channel. It receives its assignment through environment variables: +++env OBSERVER_VOD_ID OBSERVER_CHANNEL_ID OBSERVER_CHANNEL_LOGIN OBSERVER_CHANNEL_NAME +++ It starts recording immediately, writes HLS segments to object storage, sends heartbeats, and waits a short standby window after the stream goes offline. This window is important because streams sometimes drop and reconnect quickly. Without it you end up with many small broken VOD fragments. 3. Harvest Cells These handle all background work: downloading VODs, re-encoding, recovering broken files, etc. They can run anywhere Docker is available — AWS tasks, a small VPS, or even a spare laptop. They only need outbound access to Postgres and object storage. What Changed Previous

This article was originally published on davidohnstad.com . I cross-post here to reach the Dev.to community. { " @context ": " https://schema.org ", " @graph ": [ { "@type": "Person", " @id ": " https://davidohnstad.com/#author ", "name": "David Ohnstad", "url": " https://davidohnstad.com ", "sameAs": [ " https://www.linkedin.com/in/davidohnstad/ ", " https://orcid.org/0009-0007-9023-7456 ", " https://davidohnstad5.mystrikingly.com/ ", " https://github.com/davidohnstad40-netizen ", " https://hashnode.com/@davidohnstad ", " https://davidohnstad.com ", " https://davidohnstad.net ", " https://davidohnstad.info ", " https://david-ohnstad.com ", " https://davidohnstadminnesota.com " ], "jobTitle": "Senior Data Product Manager", "worksFor": { "@type": "Organization", "name": "Veeam Software", "url": " https://www.veeam.com " }, "alumniOf": { "@type": "CollegeOrUniversity", "name": "College of St. Scholastica" }, "address": { "@type": "PostalAddress", "addressLocality": "Duluth", "addressRegion": "MN", "addressCountry": "US" }, "description": "Senior Data Product Manager at Veeam Software, MS and MBA from the College of St. Scholastica, based in Duluth, Minnesota. Specializes in data architecture, AI/ML integrations, and SaaS platform development." }, { "@type": "Article", " @id ": " https://davidohnstad.com/data-product-manager-org-structure-reporting#article ", "headline": "Data Product Manager Org Structure: Reporting Lines That Matter", "description": "David Ohnstad reveals where data product managers actually fit in org charts and why reporting lines determine success. Real insights from a data PM restructure.", "url": " https://davidohnstad.com/data-product-manager-org-structure-reporting ", "datePublished": "2026-05-29T14:06:18Z", "dateModified": "2026-05-29T14:06:18Z", "author": { "@type": "Person", " @id ": " https://davidohnstad.com/#author " }, "publisher": { "@type": "Organization", "name": "David Ohnstad", "url": " https://davidohnstad.com ", "logo": { "@type"

What is nbwipers? nbwipers is a CLI tool that strips outputs and metadata from Jupyter notebooks before git commit. Written in Rust - faster than nbstripout Supports git clean filter Works with .ipynb files Why use it? Jupyter notebooks store cell outputs inside the .ipynb file (JSON). This causes problems: Noisy diffs - output changes pollute every commit Repo size - images and large outputs bloat the repo Security - sensitive data can leak in outputs (API keys, query results) The solution: strip outputs automatically on git add via a clean filter. Why not nbstripout? nbstripout is written in Python. It is slow - git status , git diff , and git add all became noticeably slow on this repo because nbstripout was invoked for every .ipynb file. The main cause is Python startup time. With 100+ notebooks, nbstripout can take 40+ seconds where a Rust-based tool takes ~1 second. Faster alternatives: Tool Language Notes nbstripout-fast Rust Up to 200x faster; no git filter install support nbwipers Rust Inspired by nbstripout-fast; adds git filter + pyproject.toml config nbwipers is essentially nbstripout-fast with better git integration. Switching to nbwipers fixed the slowness. Setup 1. Install felixgwilliams/nbwipers is now in the aqua registry as of v4.517.0 . Using aqua , add to aqua.yaml : packages : - name : felixgwilliams/nbwipers@v0.6.2 Then run: aqua install 2. Configure git filter Run once per repo (writes to .git/config ): git config filter.nbwipers.clean "nbwipers clean -" git config filter.nbwipers.smudge cat git config filter.nbwipers.required true Or edit .git/config directly: [filter "nbwipers"] clean = nbwipers clean - smudge = cat required = true required = true makes the commit fail if nbwipers is not installed. This prevents accidentally committing outputs. 3. Add .gitattributes In the repo root, add .gitattributes : *.ipynb filter=nbwipers **/.ipynb_checkpoints/*.ipynb !filter **/.virtual_documents/*.ipynb !filter The !filter lines exclude checkpoint an

Everyone can build it. Almost no one can afford to run it at scale. And the companies selling the picks and shovels are about to get undercut by the same forces they unleashed. by VEKTOR Memory — 20 min read How This Article Started: 20 Forums, 40 Headlines, and a Growing Sense That Everyone Was Confused I woke up to clear skies and the sun finally shining, and I set out to understand this idea, the truth behind it, and the nagging suspicion that the narrative around AI and software costs had become so loud, so uniform, and so confidently confusing that someone needed to sit down and actually go through it. No tweets, or are they now X's? No LinkedIn thought leader infomercials, no Substack hype, just actual research and deep thoughts. So I spent time reading, collating data. Forums, whitepapers, LinkedIn posts, Hacker News threads, VC essays, Reddit arguments. I went looking for the real signal underneath the noise. What I found instead was the full spectrum of human overconfidence, lots of moat real estate. On one end: the hype machine at full throttle. “Software is going to zero.” “A solo dev can now build what a 50-person team built in 2021.” “The era of the $500/month SaaS subscription is over.” “Vibe coding will replace your entire engineering org.” These headlines were everywhere. Breathless. Confident. Shared tens of thousands of times, this angle gets views, of course, the algorithm loves being fed claps, shares, comments, and reposts. Most were written by people who had a very good Tuesday with Codex, Windsurf, Claude and Cursor and decided that instant dev, open source to Github and getting oodles of stars, maybe even roping in a celebrity, was now the permanent condition of software development. “We are now famous on GitHub!" Very hipster, very vibes, see you on the playa.. On the other end: the backlash. Experienced engineer, people with 15 to 25 years in production systems are pushing back hard. “Show me the vibe-coded app that survived its first real

Most scraping projects start by finding a website to scrape. This one started from the opposite direction: I knew the data existed as official government downloads, and my job was to make it accessible via a clean API. The data source Ofsted (the UK school inspections body) publishes monthly management information as CSV files on GOV.UK. The file covers all 22,000+ state-funded schools in England with their latest inspection grades, local authority, postcode, phase, and size data. It's 16 MB, published under the Open Government Licence v3.0 — explicitly permitting commercial use. No scraping needed. No authentication. Just a CSV download and some parsing logic. The architecture The actor is deliberately simple: Fetch the GOV.UK stats page to find the current month's CSV URL (the URL hash changes with each release) Download the CSV (~16 MB from assets.publishing.service.gov.uk ) Parse it with csv-parse Apply the user's filters (name, local authority, region, postcode prefix) Push matching records to the Apify dataset No Crawlee. No browser. No proxy. Just fetch() and a CSV parser. const match = html . match ( /href=" ( https: \/\/ assets \. publishing \. service \. gov \. uk \/[^ " ] +latest_inspections_as_at [^ " ] + \. csv ) "/ ); That one regex does the URL discovery. The GOV.UK page lists files in reverse chronological order, so the first match is always the latest release. The interesting part: Ofsted changed their grading system mid-build I built this in May 2026. In November 2025, Ofsted scrapped their 20-year-old four-word judgement system (Outstanding / Good / Requires Improvement / Inadequate) and replaced it with a report card format — six separate grade areas, each on a five-point scale: Exceptional Strong Expected standard Needs attention Urgent improvement Plus a standalone Safeguarding verdict (Met / Not met). The April 2026 CSV reflects this change entirely. There's no "Overall effectiveness" column. Schools inspected before November 2025 have null gr