今日已更新 80 条资讯 | 累计 20052 条内容
关于我们

标签:#DNA

找到 7 篇相关文章

开发者

Using Zstd Frames to Egress Partial Parquet Files

Jump Tables, TLV Footers, and the Real Cost of Reading What You Don't Need You're paying for bytes you never read. A data engineer on a busy pipeline touches dozens of Parquet files a day: schema discovery, predicate pushdown, column pruning, metadata scrapes for a data catalog sync. In each case, the application needs maybe 200 KB of context from a file that is 4 GB on disk. Without a seekable archive format and a jump table to find the right frame, your HTTP client fetches the whole thing, and your cloud egress invoice reflects every unnecessary gigabyte. This post quantifies the problem, then walks through how HuskHoard uses seekable Zstd frames, a per-volume jump table, and TLV-encoded footer metadata to make partial egress a first-class citizen across multi-volume archives — disk, cloud, and LTO tape alike. The Problem, In Dollars S3 standard egress runs $0.09/GB. GCS is $0.08/GB. Even Cloudflare R2, which is free for egress from R2 to the internet , still costs you in latency and API call count when you cannot bound the range of bytes you need. Here is a representative read pattern for a cold analytics archive: Operation Bytes Needed Bytes Fetched (naïve) Ratio Schema discovery ~50 KB (Parquet footer) 1–8 GB (full file) ~1:16,000 Single column scan ~200 MB (one column chunk) 4 GB (full row group) 1:20 Data catalog sync (1M files) ~50 GB (footers only) ~4 PB (full files) 1:80,000 Selective restore (1 row group) ~128 MB 4 GB 1:32 On 100 TB of cold Parquet data with $0.09/GB egress: Full read for schema sync : 100 TB × $0.09 = $9,216 Partial read (footers only, avg 100 KB/file, 1M files) : ~100 GB × $0.09 = $9.00 Savings per catalog sync: $9,207 — 99.9% reduction Even a conservative column-scan scenario (pulling 15% of each file's bytes) cuts a $9,216 monthly read bill to $1,382 . The ceiling on savings is determined entirely by how precisely you can address the bytes you actually need. That precision is what frames and jump tables buy you. Zstd Frames: What They

2026-06-24 原文 →
AI 资讯

Why We Chose AGPL Instead of MIT for Neural Inverse Cloud

When we open sourced Neural Inverse Cloud, the easiest choice would have been MIT. Most developers like MIT. It's short, permissive, and widely adopted. If you've released an open-source project before, MIT is probably the first license you considered. We didn't choose it. We chose AGPL. Not because we dislike permissive open source. Not because we want to restrict users. We chose it because infrastructure software plays by different rules. The Infrastructure Problem MIT works incredibly well for libraries. You publish code, developers use it, and occasionally improvements flow back into the project. Nobody is forced to contribute, but community norms often make it happen anyway. Infrastructure software is different. Cloud IDEs, databases, developer platforms, deployment systems, and backend services can be monetized without ever distributing the source code. A company can: Fork your project Add proprietary features Launch a hosted version Build a competitive advantage on top of community work Never contribute anything back The original project does all the R&D. The fork captures the value. We've seen this pattern repeatedly across open-source infrastructure over the last decade. Why AGPL Exists AGPL closes a loophole that traditional open-source licenses leave open. With GPL, if you distribute modified software, you must publish your changes. But what if you never distribute the software? What if you simply run it as a hosted service? That's where AGPL comes in. If you modify AGPL software and provide it to users over a network, you must also provide the source code for those modifications. That applies to everyone. Including us. If we improve Neural Inverse Cloud, those improvements stay open. If someone else builds a SaaS business on top of it, their modifications stay open too. Why This Matters for Users We wanted users to have guarantees. With AGPL: You can self-host the latest version Community improvements remain accessible No company can create a permanently

2026-06-22 原文 →
AI 资讯

CloudNativePG: Running PostgreSQL in Kubernetes Without the Pain

A CloudNativePG cluster that sits in Setting up primary forever, with zero error events on the Cluster resource and a perfectly healthy operator, is one of the more frustrating ways to spend an afternoon. The operator says it's working. The pods never appear. And the actual cause has nothing to do with the database at all. Running stateful databases on Kubernetes used to be the thing everyone told you not to do. CloudNativePG (CNPG) changed that calculus for a lot of people, including me. It's a proper operator: it handles failover, backups, connection routing, and rolling upgrades through native Kubernetes primitives instead of bolting Postgres onto a StatefulSet and praying. If you run a hardened cluster with admission controllers, network policies, and least-privilege RBAC, this post is about the friction you'll hit that the quickstart never mentions. Who should care If your cluster is vanilla, kubectl apply the operator and a Cluster manifest, and you're done in ten minutes. The CNPG docs are genuinely good for that path. This is for the rest of us: people running Kyverno or OPA Gatekeeper, self-signed cert chains, and the kind of policy-as-code setup where every workload has to justify its existence. That's where CNPG stops being a ten-minute install and starts being an integration project. What I tried first The first instinct, when a CNPG cluster hangs, is to assume you got the database config wrong. So you go read your Cluster manifest line by line. You check the storage class. You check that the PVC bound. You bump the operator log level and watch it cheerfully report that it's reconciling, over and over, with no complaints. Here's the trap: the CNPG operator doesn't run initdb itself. It creates a Kubernetes Job to bootstrap the primary. That Job spawns a Pod. And in a hardened cluster, the Pod is where everything dies, because your admission controller is judging it against policies the operator's own Pods were exempted from but the bootstrap Job was not.

2026-06-16 原文 →
AI 资讯

Building CompanioxVPS — I'd Love to Hear What Developers Actually Want From a VPS Platform

I've been working on CompanioxVPS, a VPS and cloud infrastructure platform that aims to make deploying and scaling applications much simpler for developers. Right now, I'm building core features such as: VPS provisioning Load balancing Autoscaling Custom domains Developer APIs Simple billing and infrastructure management Before going too far down the road, I'd love to hear from developers who actively ship products: What do you dislike most about current VPS or cloud providers? What features do you wish existed? What would make you switch to a new platform? What's one thing that would make your deployment workflow significantly easier? I'm also open to connecting with developers, DevOps engineers, and infrastructure enthusiasts who find this space interesting and would like to contribute ideas, provide feedback, or potentially collaborate as the project grows. Still early, still building, and still learning. Every piece of feedback helps shape the direction of CompanioxVPS. Looking forward to hearing your thoughts. 🚀

2026-06-13 原文 →