Building an MCP Server on 31 Million Rows of Financial Data

This is the architecture of Shibui Finance , an MCP server that gives Claude direct SQL access to 64 years of US stock market data. About 10,000 symbols, 31 million daily price records, quarterly financials back to 1990, 56 pre-computed technical indicators, and 6.4 million SEC filing records. Free to use. Stack: Python, PostgreSQL, dbt, DuckDB, FastMCP, Caddy. Runs on a single VPS. Data pipeline Three stages: ingest into PostgreSQL, transform with dbt, export to DuckDB. Data APIs / SEC EDGAR / FRED | Python ETL (Polars, ADBC) | PostgreSQL clean_* schemas (~50 raw tables) | dbt (27 models) staging -> integration schema (17 analytical tables) | DuckDB export (daily, ~14 GB file) | FastMCP server (read-only, streamable-http) | Caddy (TLS) -> mcp.shibui.finance Multiple sources feed the pipeline: commercial data APIs for prices, fundamentals, valuations, and estimates. SEC EDGAR for filing metadata and insider transactions (bulk historical + a 5-minute Atom feed for near-real-time). FRED for FX rates to normalize non-USD fundamentals. Public registries for ticker classification. The ETL is a Python CLI organized by data source. Each module has its own fetcher, loader, and CLI. A single all command runs everything in fixed sequence. You can't refresh 10,000 tickers daily without hitting rate limits, so the ETL rotates: each run refreshes the stalest 5% of tickers. Full universe cycles in about 20 runs. Recent prices always refresh on every run. Every table write is a single transaction. DROP + CREATE inside a transaction, rollback on failure. The database never serves partial data, and dbt always sees complete tables even when ingest jobs overlap. The dbt layer 27 models in two tiers. The process layer handles standardization: enriching symbols with security types and exchange mappings, linking SEC amendment filings to their originals, repairing filer date typos. The integration layer produces the 17 tables that Claude actually queries. This is where raw normalized tabl