标签：#zip

I run ezyZip , a browser-based archive tool, so "which format should I use?" is a question I field constantly. The honest answer is usually "it depends," which satisfies nobody. So I stopped hand-waving and measured it. We benchmarked 42 archive and compression formats, spanning four decades, from 1984's Unix compress through today's Zstandard, Brotli, and context-mixing paq8px. Everything ran against the same realistic 55 MB corpus, every archive was round-trip verified byte for byte, and the whole thing reproduces from a single command. Here's what came out of it, and what I'd actually reach for. The setup Most compression benchmarks measure raw codecs on standardized corpora like Silesia. That's the right call for algorithm research and the wrong call for answering "what should I zip my folder with?" I wanted end-user formats, real CLI tools, container overhead and all, on data that looks like an actual folder. So the corpus is deliberately mixed: about 11 MB of text, 15 MB of office documents, 16 MB of images, and 13 MB of video, all public domain so it can be committed and redistributed. That mix matters. Office documents ( .docx , .xlsx , .pptx ) are themselves ZIP containers, so they stress how a tool handles already-compressed data. The JPEG and H.264 media is near-incompressible and sets an honest lower bound. The plain text and uncompressed images are where formats actually separate. Two rules kept it fair and practical: Only two levels per tool: its default, and its one "maximum compression" dial. No method tuning, no dictionary sizes, no thread-count games. That's what a normal person can reach. Everything is round-trip verified. Each archive gets extracted, and every file is hashed with SHA-256 against the original manifest. Exit codes are not trusted. That last rule earned its keep immediately. The verification gotcha On the image category, a 1985-era ARC build produced an archive that its own extractor happily unpacked, while printing a CRC warning an

I recently read Jérémie Rodon's excellent article On-Demand Archives on S3 , where he describes an elegant Rust solution for zipping 3,000 × 5MB files from S3 within a single Lambda function. His approach is impressive: streaming a ZIP archive through a custom Rotating Slab Buffer, saturating bandwidth with concurrent downloads, all within 512MB of RAM. The result: 3 minutes 35 seconds . I thought it was a good challenge to reach better performance. His article ends with an open invitation: "do you think you can do better with your favorite language?" Well, my favorite language is not Rust nor Go nor.. however, I'm fluent in serverless ;) so I took a different angle entirely. A Different Approach: Why Not Parallelize the Problem? Jérémie's constraint was a single Lambda. That's elegant, but it means you're bound by one machine's network bandwidth (~600 Mbps). No matter how perfect your streaming is, physics wins: 15GB at 600 Mbps ≈ 200 seconds minimum. My question was: what if we break that single-machine bottleneck? The key insight is that ZIP files in STORE mode (no compression) have deterministic byte offsets . Each entry is exactly 50 + len(filename) + filesize bytes (local header + ZIP64 extra field + data). If you know all filenames and sizes upfront, you can pre-calculate exactly where every file will land in the final archive, before downloading a single byte. This means independent workers can each build their portion of the zip in parallel, and S3's multipart upload lets them write their chunks independently (parts can be uploaded in any order by different processes sharing the same upload ID). Architecture Planner Lambda → Step Functions Distributed Map → N Worker Lambdas → Finalizer Lambda │ │ │ │ │ │ CreateMultipartUpload │ │ │ UploadPart (parallel) │ CompleteMultipartUpload ▼ ▼ ▼ ▼ ▼ S3 Output Bucket Planner : Lists all source files, computes zip byte offsets, initiates multipart upload, divides work into balanced batches (equal data volume per worker)