标签：#pytest

Most data engineers write pipelines the way most people write shell scripts: run it, eyeball the output, ship it. That works right up until a schema changes upstream, a null slips through a join, or someone "fixes" a transformation and silently breaks three downstream tables. By then the bug isn't your problem anymore — it's a bad number in someone's dashboard. Software engineers solved this problem decades ago with automated testing. Data engineering has been slower to adopt the habit, partly because our code touches messy external reality (files, databases, clusters) in a way a typical web app doesn't. But that's exactly why testing matters more here, not less. This article is a practical, DE-flavored crash course in pytest — the dominant Python testing framework — plus the patterns you actually need for pandas, Polars, and PySpark pipelines. Why bother testing a data pipeline? A few concrete failure modes that tests catch before production does: A column gets renamed upstream and your join silently produces all-null matches instead of erroring. A "cleaning" function that's supposed to drop duplicates accidentally drops valid rows too. A date-parsing function works on your local machine's locale and breaks in the CI environment. A refactor changes an aggregation from sum to mean and nobody notices until finance asks why revenue looks 90% smaller. None of these require exotic testing techniques. They require the habit of writing small, deterministic checks against small, deterministic inputs — which is exactly what pytest is built for. Where pytest fits — and where it doesn't Before diving in, it's worth being precise about scope, because "testing a data pipeline" actually covers two different questions, and conflating them is a common source of confusion: Is my code correct? Given a known input, does the transformation logic produce the right output? This is a property of your code , and it doesn't change based on what day it is or what a source system decided to

Introduction Hello from Japan! 🇯🇵 I am tosane932 , a professional truck driver working in logistics while teaching myself Python. In my previous article, I tested a Docker multi-stage build and measured the actual change in image size. At the end of that article, I said that I would write next about pytest and CI/CD. This article was supposed to be the practical follow-up. However, while preparing for that work, I encountered an unexpected side issue. I only intended to introduce Flask-Migrate. Instead, the pip installation logs revealed that the version of a library in my local development environment had been changed without me noticing. The library was: google-genai From there, I went through the following process: Identify the version mismatch Restore the version that had already been tested locally Update requirements.txt Manually verify the Gemini API functionality Run pytest to check for regressions This article records that process without hiding the inconvenient parts. https://github.com/tosane932/sales_data_app Overview While installing Flask-Migrate, I noticed a mismatch between: The version of google-genai installed in my local development environment The version declared in requirements.txt The local environment had been using: google-genai 2.10.0 However, requirements.txt still specified: google-genai==2.4.0 When I ran: pip install -r requirements.txt pip followed the configuration file and replaced the newer local version with the older declared version. This article explains how I discovered the issue, synchronized the environments, and verified the application behavior with automated tests. 1. The Problem and Its Background I was preparing to introduce Flask-Migrate. During that work, I ran: pip install -r requirements.txt The installation log contained the following lines: Attempting uninstall: google-genai Found existing installation: google-genai 2.10.0 Uninstalling google-genai-2.10.0: That message caught my attention. After checking the environ