AWS Serverless Weather Data Pipeline

Building a Serverless Weather Pipeline on AWS: A Step-by-Step Walkthrough This is a build log for someone who's used AWS a bit — deployed a Lambda from the console, poked around S3 — but hasn't touched CDK, Step Functions, EventBridge Scheduler, or GitHub's OIDC setup before. I'll explain each concept the first time it comes up, and show the actual code behind every piece, roughly in the order I built it. Here's what it ends up doing: every 10 minutes, EventBridge Scheduler kicks off a Step Functions workflow that pulls current weather for five cities in parallel from a free public API, reshapes the results into JSON Lines, drops them into S3 in a partitioned layout, and makes them queryable in Athena with plain SQL. No crawler, and no AWS credentials sitting anywhere in the GitHub repo that deploys it. kasukur / serverless-weather-pipeline AWS Serverless Weather Pipeline Serverless Weather Data Pipeline A small but complete serverless data pipeline on AWS walkthrough: EventBridge Scheduler → Step Functions → Lambda → S3 → Glue/Athena , deployed by GitHub Actions with no AWS access keys stored anywhere (authentication is via GitHub's OIDC provider). flowchart TD A["EventBridge Scheduler (every 10 min)"] --> B["Step Functions state machine"] B --> C["PrepareCities (Pass)"] C --> D["ForEachCity (Map, concurrency 4)"] D --> E["FetchWeather (Lambda -> Open-Meteo public API)"] E -.-> F["retries transient errors (up to 2 attempts)"] E -.-> G["FetchFailed (Pass): per-city failure absorbed here, other cities continue"] E --> H["TransformWeatherData (Lambda, pure function, no AWS calls)"] H -.-> I["splits successes vs failures"] H -.-> J["builds JSON-Lines body + partitioned S3 key"] H --> K["LoadToS3 (Lambda, writes to S3 via boto3)"] K --> L["S3 (processed/dt=YYYY-MM-DD/hour=HH/*.jsonl)"] L --> M["Glue Data Catalog table (partition projection -- no crawler)"] M --> N["Athena (query with plain SQL)"] D -.-> … View on GitHub Table of Contents What we're building, and why eac