Stop Fighting Your Fitness Data: Build a Serverless Warehouse with DuckDB and dbt

If you’ve ever tried to reconcile a night of sleep from an Oura Ring , a morning run from a Garmin watch, and active minutes from an Apple Watch , you know the "Dirty Data" struggle is real. Each platform has its own schema, its own definition of "active calories," and its own idiosyncratic export format. In the world of Data Engineering , this is a classic multi-source integration problem. But you don't need a massive Snowflake cluster to solve it. Today, we’re building a high-performance, serverless data pipeline to clean and normalize wearable data using DuckDB , dbt , and GitHub Actions . By leveraging a modern Serverless Data Pipeline and DuckDB's lightning-fast processing, we can turn a mess of CSVs into a structured Parquet -based personal data warehouse. The Architecture: From Chaos to Clarity Before we dive into the code, let’s look at how the data flows from your wearables to a clean, queryable state. graph TD A[Oura JSON] -->|Python Ingestion| D[(DuckDB Raw)] B[Garmin CSV] -->|Python Ingestion| D C[Apple Health XML] -->|Python Ingestion| D D --> E{dbt Models} E -->|Cleaning| F[stg_models] E -->|Normalization| G[int_health_metrics] G -->|Final Output| H[Gold Layer: Parquet Files] H --> I[Visualization / BI] subgraph GitHub Actions D E F G H end Prerequisites To follow along, you'll need: DuckDB : The "SQLite for OLAP" that makes local analytical processing insanely fast. dbt-duckdb : The adapter that lets dbt talk to DuckDB. GitHub Actions : Our free "orchestrator." Tech Stack : DuckDB, dbt, Python, Parquet. Step 1: The Ingestion Layer (Python + DuckDB) The first hurdle is getting disparate files (JSON, CSV, XML) into a unified storage format. DuckDB is magical here because it can query these files directly. We'll use a simple Python script to load these into a local .duckdb file. import duckdb def ingest_raw_data (): # Initialize the database con = duckdb . connect ( ' health_data.duckdb ' ) # Ingest Garmin CSV con . execute ( """ CREATE TABLE raw_garmin