Three TODOs, three weeks, one weekend: finishing pq v0.14
This is a submission for the GitHub Finish-Up-A-Thon Challenge What I Built pq — jq for Parquet. A 50 MB Rust single binary that wraps DuckDB's query engine in a jq-style expression DSL, optimized for terminal one-liners and unix pipes. $ pq sales.parquet 'group_by .country | sum .revenue | top 3 by sum_revenue' ┌─────────┬─────────────┐ │ country ┆ sum_revenue │ ╞═════════╪═════════════╡ │ US ┆ 19065.00 │ │ FR ┆ 999.99 │ │ DE ┆ 312.00 │ └─────────┴─────────────┘ Where it started. I work in adtech. I look at parquet files dozens of times a day — campaign deliveries, partner exports, audience snapshots. Every existing option was painful: Tool Pain pyarrow / pandas 5-second cold start, 200 MB virtualenv parquet-tools JVM, slow, no query support pqrs Inspector only — can't filter or project duckdb CLI Great engine, but SELECT email FROM 'file.parquet' WHERE country='US' is too verbose to type 50 times a day Spark Are you serious pq is the tool I actually want — single binary, no JVM, no Python, jq-style syntax for piping into the rest of the unix toolbox. It's been my default cat for parquet since v0.5. Demo Repo : github.com/thehwang/parq Latest release : v0.14.0 (this submission) Install : brew install thehwang/parq/pq Tutorial : doc/tutorial.md — 30-minute hands-on walkthrough A taste of what shipped in v0.14: # Streaming JSON output (was the only buffered format until v0.14) $ pq big.parquet '.id, .country' -o json | head -c 200 # returns instantly even on a 40 GB file # Schema-drift gate for CI $ pq diff baseline.parquet candidate.parquet # Schema diff - a: ` baseline.parquet ` - b: ` candidate.parquet ` ## Added (1) | column | type | nullable | |-----------|---------|----------| | ` country ` | VARCHAR | yes | $ echo $? 1 # exits non-zero on drift, slots into CI without scripting And the new TUI Explain panel — press capital E for EXPLAIN ANALYZE , get row-group pruning per scan (this is exactly the panel you see on the cover image at the top of this post): Expla