标签：#duckdb

DuckDB Labs has previewed DuckDB v2.0, codenamed "Cyanoptera." This release includes over 10000 commits and introduces a client/server mode, enabling network connections. Improvements also encompass extension portability, advanced data types, and a new parser. Performance enhancements include asynchronous I/O and storage optimisations. General availability is expected in fall 2026. By Olimpiu Pop

This is the architecture of Shibui Finance , an MCP server that gives Claude direct SQL access to 64 years of US stock market data. About 10,000 symbols, 31 million daily price records, quarterly financials back to 1990, 56 pre-computed technical indicators, and 6.4 million SEC filing records. Free to use. Stack: Python, PostgreSQL, dbt, DuckDB, FastMCP, Caddy. Runs on a single VPS. Data pipeline Three stages: ingest into PostgreSQL, transform with dbt, export to DuckDB. Data APIs / SEC EDGAR / FRED | Python ETL (Polars, ADBC) | PostgreSQL clean_* schemas (~50 raw tables) | dbt (27 models) staging -> integration schema (17 analytical tables) | DuckDB export (daily, ~14 GB file) | FastMCP server (read-only, streamable-http) | Caddy (TLS) -> mcp.shibui.finance Multiple sources feed the pipeline: commercial data APIs for prices, fundamentals, valuations, and estimates. SEC EDGAR for filing metadata and insider transactions (bulk historical + a 5-minute Atom feed for near-real-time). FRED for FX rates to normalize non-USD fundamentals. Public registries for ticker classification. The ETL is a Python CLI organized by data source. Each module has its own fetcher, loader, and CLI. A single all command runs everything in fixed sequence. You can't refresh 10,000 tickers daily without hitting rate limits, so the ETL rotates: each run refreshes the stalest 5% of tickers. Full universe cycles in about 20 runs. Recent prices always refresh on every run. Every table write is a single transaction. DROP + CREATE inside a transaction, rollback on failure. The database never serves partial data, and dbt always sees complete tables even when ingest jobs overlap. The dbt layer 27 models in two tiers. The process layer handles standardization: enriching symbols with security types and exchange mappings, linking SEC amendment filings to their originals, repairing filer date typos. The integration layer produces the 17 tables that Claude actually queries. This is where raw normalized tabl

TL;DR Traditional decoupled ETL pipelines (like the "Modern Data Stack") are too brittle and complex to handle the unpredictable, heavily nested data generated by AI and LLM features. Agentic data serving solves this by focusing on dynamic query routing and semantic discovery, letting AI agents discover and query data autonomously using schema-resilient tools and codified business logic. You can build an agentic data stack by pairing S3 storage with DuckDB's native JSON handling and schema-agnostic Parquet reading ( union_by_name=true ), eliminating failure-prone parsing steps. The open Model Context Protocol (MCP) replaces custom, hacky LangChain tools by providing a standard interface for agents to discover schemas and execute queries securely. The open Model Context Protocol (MCP) and DuckDB's embeddable architecture make it practical to connect agents directly to your data with minimal infrastructure overhead and elastic, consumption-based compute. For years, broken ETL jobs powered my pager and my morning coffee. I am a staff engineer, and like many of you, I have spent a ridiculous amount of my career babysitting data pipelines. It is a thankless job that often feels like patching holes in a sinking ship. You are not alone in this. A Forbes survey shows data teams notoriously spend up to 80% of their time just moving and cleaning data instead of doing the interesting work of analysis. And the financial magnitude of this bottleneck is staggering: the ETL market is projected to reach $20.1 billion by 2032 at a 13% CAGR. This proves that massive industry capital is flowing into solving these pipeline bottlenecks, but throwing more money at the same old architecture was not going to save my mornings. This constant firefighting was frustrating, but manageable. Then came the new mandate: build the data backbone for our next-gen AI and LLM-based product features. The unpredictability of the queries and the sheer complexity of the data, nested JSON everywhere, were th

DuckDB has recently announced Quack, a new remote protocol over HTTP that lets multiple DuckDB instances connect to and work with the same database over a network. The protocol introduces client-server capabilities to a database that was previously mostly local and embedded. By Renato Losio