Using Zstd Frames to Egress Partial Parquet Files
Jump Tables, TLV Footers, and the Real Cost of Reading What You Don't Need You're paying for bytes you never read. A data engineer on a busy pipeline touches dozens of Parquet files a day: schema discovery, predicate pushdown, column pruning, metadata scrapes for a data catalog sync. In each case, the application needs maybe 200 KB of context from a file that is 4 GB on disk. Without a seekable archive format and a jump table to find the right frame, your HTTP client fetches the whole thing, and your cloud egress invoice reflects every unnecessary gigabyte. This post quantifies the problem, then walks through how HuskHoard uses seekable Zstd frames, a per-volume jump table, and TLV-encoded footer metadata to make partial egress a first-class citizen across multi-volume archives — disk, cloud, and LTO tape alike. The Problem, In Dollars S3 standard egress runs $0.09/GB. GCS is $0.08/GB. Even Cloudflare R2, which is free for egress from R2 to the internet , still costs you in latency and API call count when you cannot bound the range of bytes you need. Here is a representative read pattern for a cold analytics archive: Operation Bytes Needed Bytes Fetched (naïve) Ratio Schema discovery ~50 KB (Parquet footer) 1–8 GB (full file) ~1:16,000 Single column scan ~200 MB (one column chunk) 4 GB (full row group) 1:20 Data catalog sync (1M files) ~50 GB (footers only) ~4 PB (full files) 1:80,000 Selective restore (1 row group) ~128 MB 4 GB 1:32 On 100 TB of cold Parquet data with $0.09/GB egress: Full read for schema sync : 100 TB × $0.09 = $9,216 Partial read (footers only, avg 100 KB/file, 1M files) : ~100 GB × $0.09 = $9.00 Savings per catalog sync: $9,207 — 99.9% reduction Even a conservative column-scan scenario (pulling 15% of each file's bytes) cuts a $9,216 monthly read bill to $1,382 . The ceiling on savings is determined entirely by how precisely you can address the bytes you actually need. That precision is what frames and jump tables buy you. Zstd Frames: What They