标签：#database

Twelve pricing tabs open. Neon, Hetzner, Supabase, Prisma, Scaleway, OVH. My database is half a gigabyte. I was comparing ten-terabyte price curves. At some point this week I typed the words "I am super lost here" about my own infrastructure. I advise companies on this exact class of decision. That sentence still came out of my hands. If you have ever spent an evening deep in provider pricing pages for a workload that fits on a USB stick from 2009, this one is for you. All numbers below come from the live pricing pages as of July 2026. Rates move, so verify before you commit. Three fears, all pointed at the wrong layers I went in worried about getting attacked, running out of space, and being locked in. All three dissolved under ten minutes of honest reading. DDoS lands on the website edge, not the database. My site already sits behind Cloudflare and Vercel, and a database is never publicly exposed. Only the app talks to it. Whichever provider I picked, that attack surface stayed identical. Here is the shape of the stack, and where each fear lives. MANAGED (what I run today) visitors ──> Cloudflare edge ──> Vercel app ──> managed Postgres [DDoS absorbed] [stateless] [never public, app-only access, provider patches, provider backups, provider on-call] SELF-HOSTED (the alternative I priced) visitors ──> Cloudflare edge ──> Vercel app ──> Hetzner CAX11 [DDoS absorbed] [stateless] [Postgres :5432 firewalled to app, SSH hardened, fail2ban + auto- patching = MINE] │ pg_dump every 6h ▼ encrypted ────> Cloudflare R2 [off-site copies] Same edge, same app, same attack surface. Everything in the right-hand box is what changes owners. Storage was a rounding error. My data is 0.5 GB. Even the cheapest self-hosted box includes 40 GB, eighty times headroom before the first extra cent. Lock-in was a phantom too. Managed Postgres is still stock Postgres. Exiting means a dump, a restore, and one connection string change in the deployment environment. Minutes of cutover, no rewrite an

SQLite Internals: lcd-ex vs hctree; PostgreSQL 19 SQL/PGQ Rewrites & pg_timetable Migration Today's Highlights This week's highlights feature a deep dive into SQLite's internal data structures, offering insights for advanced optimization. Also, PostgreSQL users gain practical guidance on migrating to pg_timetable for robust job scheduling and understanding how SQL/PGQ translates to efficient joins in PostgreSQL 19. Replacing pgAgent with pg_timetable: Installing as a Linux Service (Planet PostgreSQL) Source: https://postgr.es/p/9pE Regina Obe presents a crucial guide for PostgreSQL administrators looking to modernize their task automation by replacing pgAgent with pg_timetable . This second part of the series focuses specifically on the practical steps of installing and configuring pg_timetable as a systemd service on Linux, ensuring it runs reliably in a production environment. The article details the process from downloading binaries and creating dedicated user accounts to setting up service files and enabling autostart, providing a comprehensive walkthrough for seamless integration. pg_timetable offers significant advantages over pgAgent , including advanced scheduling capabilities, event-driven task execution, parallel job processing, and improved logging. This migration strategy is vital for enhancing the robustness and efficiency of database maintenance, data synchronization, and complex ETL pipelines within the PostgreSQL ecosystem. By following this guide, developers and DBAs can transition to a more powerful and flexible job scheduler, leading to greater control and reliability over their automated PostgreSQL operations. Comment: Migrating to pg_timetable from pgAgent is a significant step forward for job scheduling in PostgreSQL. This guide provides the hands-on steps needed to get it running as a service, which is essential for any production deployment. SQLite Forum Discusses lcd-ex vs hctree (SQLite Forum) Source: https://sqlite.org/forum/info/3494bff42

RocheDB v0.5.0 has been released. Release: github.com/puffball1567/rochedb/releases/tag/v0.5.0 RocheDB is an open-source NoSQL document and vector store written in Nim. The project is built around one idea: Data locality should be part of the database model, not only an accidental result of indexes, caches, or application code. A lot of database performance discussions start with indexes, query syntax, or caches. Those are important. But layout often decides whether a system is working with the hardware or asking it to fight back. RocheDB v0.5.0 is a step toward making locality explicit at three levels: logical placement with rings; related-data retrieval with stellar locality; physical layout visibility with WAL locality reporting. Ring locality RocheDB uses a ring as a semantic and structural placement unit. An application, import rule, or operator chooses a ring when writing data: users/123/profile users/123/orders shops/1123/orders docs/japan/support tenant/acme/orders/2026 That ring is not only a directory-like label. It is a coordinate in the retrieval space. When a request already knows its natural locality, RocheDB can open that local region first instead of scanning unrelated records and filtering later. roche put --ring = docs/japan --payload = '{"title":"Refund guide","status":"draft"}' --codec = json roche get --ring = docs/japan --filter = '{"status":"draft"}' --selection = '{ title }' The important part is not the string syntax. The important part is that placement and retrieval scope are connected. Short version: a ring is both where data is placed and where retrieval can start. Not only point reads Point reads are important, but many real applications need related data. For example, a user page may need profile data, orders, billing information, and support metadata. In a relational database, that often becomes joins. In a document database, it often becomes manual denormalization or multiple application-side reads. RocheDB v0.5.0 adds a locality mec

One of the most common questions I hear is: "My API is slow. Where do I start?" The first instinct is usually: Upgrade the server Increase CPU Add more RAM But in many cases, the database query is the real bottleneck . Whenever I investigate a slow Laravel application, I follow the same checklist. It helps me identify performance issues before making unnecessary infrastructure changes. Let's go through it. 1️⃣ Find the Slow Queries First Don't start optimizing random queries. Start with the queries that are executed the most or take the most time. Useful tools: Laravel Telescope Laravel Debugbar (development) MySQL Slow Query Log Application Performance Monitoring (APM) You can't optimize what you haven't measured. 2️⃣ Stop Using SELECT * One of the easiest improvements. ❌ Instead of: SELECT * FROM users WHERE id = 10 ; Use: SELECT id , name , email FROM users WHERE id = 10 ; Why? Less data transferred Lower memory usage Faster response Easier for MySQL to use covering indexes Only fetch the columns your application actually needs. 3️⃣ Always Check the Execution Plan Before changing anything, run: EXPLAIN SELECT id , name FROM users WHERE email = 'john@example.com' ; Things I usually look for: Is MySQL scanning the whole table? Is an index being used? How many rows are examined? Is there a temporary table? Is filesort being used? EXPLAIN often tells you exactly why a query is slow. 4️⃣ Verify Your Indexes Indexes are one of the biggest performance improvements you can make—but only when they match your queries. Example: SELECT * FROM orders WHERE customer_id = 100 ; Create an index: CREATE INDEX idx_customer_id ON orders ( customer_id ); Now MySQL can jump directly to the matching rows instead of scanning the entire table. 5️⃣ Look for Composite Index Opportunities Suppose your query is: SELECT id , total FROM orders WHERE customer_id = 10 AND status = 'paid' ; Instead of two separate indexes: customer_id status A composite index is often better: CREATE INDEX idx_cu

7 MongoDB Query Mistakes That Return the Wrong Results VisuaLeaf VisuaLeaf VisuaLeaf Follow Jul 14 7 MongoDB Query Mistakes That Return the Wrong Results # mongodb # coding # software # database 3 reactions Add Comment 5 min read

MongoDB queries look simple. You type a field, give it a value, hit run, and you get your data back. But just because a query runs without throwing an error doesn't mean it worked right. Sometimes you get a blank screen. Sometimes you get way too many records. Other times, the data looks fine at first glance, but it doesn't actually match what you asked for. Most of these slip-ups happen for one basic reason: the query structure doesn't match the way the data actually sits in the database. To show you what we mean, we’ll use a clinic database with a collection called visits . Here is what a typical document looks like: JSON { "_id": "6871b6f9c3f1d1a4c2a10001", "status": "completed", "visitDate": "2026-07-01T09:30:00.000Z", "patient": { "name": "Anna Keller", "age": 34 }, "doctor": { "name": "Dr. James Carter", "specialty": "Cardiology" }, "symptoms": ["cough", "fever"], "prescriptions": [ { "name": "Ibuprofen", "active": false }, { "name": "Paracetamol", "active": true } ], "invoice": { "paid": true, "method": "card", "total": 250 } } You can run these examples right in the VisuaLeaf MongoDB Shell . Using visual tools makes a big difference because you can see exactly what MongoDB is returning in real time. 1. Forgetting the Curly Braces This is just a quick typo, but it breaks things right away. The Mistake: db . visits . find ( status : " completed " ) The Correct Query The find() tool always expects an object. Even if you are only looking for one specific thing, you still need to wrap that condition in curly braces {} . 2. Treating $or Like a Regular Object This one trips a lot of people up because the broken version looks like it should work. The Mistake: db.visits.find({ $or: { status: "completed", "invoice.paid": false } }) What is wrong: $or expects an array of conditions, but this query gives it one object. The error will usually be something like: MongoServerError: $or must be an array The Correct Query The first query is wrong because $or needs an array, n

DuckDB Iceberg MERGE, PostgreSQL GUCs, SQLite Optimization Checklist Today's Highlights This week's highlights include powerful new Iceberg data manipulation features in DuckDB v1.5.3 and a deep dive into an obscure PostgreSQL GUC. Plus, the SQLite community discusses a practical optimization checklist for embedded databases. New DuckDB-Iceberg Features in v1.5.3 (DuckDB Blog) Source: https://duckdb.org/2026/05/29/new-iceberg-features.html The latest DuckDB v1.5.3 release significantly enhances its integration with Apache Iceberg, introducing a suite of powerful new features for data engineers and analysts. Key among these are the support for MERGE INTO and ALTER TABLE statements, allowing for more robust data manipulation directly within DuckDB for Iceberg tables. This update enables complex operations like upserting data based on conditions, schema evolution (e.g., adding/dropping columns), and modifying table properties, all achievable through a familiar SQL environment. This capability is crucial for maintaining data integrity and adapting schemas without complex external tooling. Furthermore, DuckDB-Iceberg now supports partition transforms, making it easier to manage and query partitioned Iceberg datasets efficiently by defining how data is distributed across files. The release also brings support for Iceberg V3, ensuring compatibility with the latest features of the Iceberg format, including new manifest list and manifest file layouts which offer performance improvements. These additions position DuckDB as an even stronger tool for building performant data pipelines and performing complex analytics directly on large-scale Iceberg data lakes, fully leveraging DuckDB's in-process analytical capabilities and the flexibility of the Iceberg table format. Comment: This update is a game-changer for working with Iceberg tables directly in DuckDB. MERGE INTO support means simplified ETL for incremental loads, and V3 compatibility ensures we're ready for future Iceberg

The ScyllaDB PHP driver is not a C++ extension anymore. As of 1.4.0 it's pure C23, the ZendCPP template layer we leaned on for the object embed and allocate pattern is deleted, and the build no longer needs a C++ compiler at all. Every hand-written .cpp file is a .c file now (71 of them), the descriptor generator emits .c , and CMake builds with c_std_23 and nothing else. That's the biggest change to how this extension is built since we forked it for PHP 8.0. This is also the release where a plan I opened back in 2023 finally landed. PR #50 laid it out: rewrite the src/Cluster directory to be more maintainable, use Zend Fast Argument Parsing, remove some memory allocations, and add .stub.php files that generate the C headers so nobody has to hand-maintain Zend arginfo by hand. 1.4.0 is that plan finished, and a lot more that grew out of it. The things you'll actually feel: persistent session connect() doesn't allocate a 200-character key string on every call anymore, and the minimum PHP is now 8.3 (8.2 is gone). Nothing in your application code changes, this is almost all under the surface. The .stub.php Build The idea from PR #50 was small. Instead of writing ZEND_BEGIN_ARG_INFO_EX blocks by hand and keeping them in sync with the actual method bodies, write the signature once in a .stub.php file and generate the C arginfo from it. In v1.4.0 that's the whole build. There are 75 .stub.php files now, and each one is just the PHP signature of the class: // src/Keyspace.stub.php interface Keyspace { public function name (): string ; public function replicationClassName (): string ; /** @return array<string, mixed> */ public function replicationOptions (): array ; public function hasDurableWrites (): bool ; /** @return Table|false */ public function table ( string $name ): Table | false ; public function aggregate ( string $name , mixed ... $types ): Aggregate | false ; } At build time CMake runs gen_stub.php (vendored from PHP 8.5's build/gen_stub.php , with two small p

How I Built CodeGraph: A Living Knowledge Graph That Tells You What Breaks Before You Break It Built for HACKHAZARDS '26 — powered by Neo4j AuraDB, tree-sitter, Groq LLaMA, and Next.js The Problem That Frustrated Me Every developer knows this feeling. You join a new codebase. There are 50,000 lines of code. Your manager says "just fix this small bug in the authentication module." You make the change. You push. And suddenly three completely unrelated features are broken — a payment flow, a notification system, and a dashboard widget you've never even looked at. You spend the next four hours tracing function calls manually, reading code you've never seen, trying to understand why changing one function in auth.py broke something in notifications.py on the other side of the codebase. This is not a rare experience. According to JetBrains' developer survey, engineers spend 58% of their time reading and understanding code — not writing it. One wrong change in a large codebase can cost hours of debugging, failed deployments, and frustrated users. I built CodeGraph to solve this. Not with another AI chatbot that guesses at your code. With a real, queryable knowledge graph that actually understands how your codebase is connected. What CodeGraph Does CodeGraph takes any public GitHub repository URL and within seconds: Parses every function in the codebase using tree-sitter Maps every call relationship between functions as a directed graph Stores everything in Neo4j AuraDB as a live knowledge graph Lets you ask questions in plain English — answered by AI grounded in real graph data The result: paste a GitHub URL, see your entire codebase as an interactive graph, click any function, and instantly know what breaks if you change it. The Tech Stack Here's what I used and why each choice mattered: Backend: Python + FastAPI (REST API server) Neo4j AuraDB (graph database — the core of everything) tree-sitter (AST parser for Python, JS, TS, TSX) Groq API with LLaMA 3.3 70B (free-tier L

If you ask me what part of the system I protect the most, the answer is the database. I've been writing software alone for twenty-four years, and across every platform I've built, the rule has stayed the same: the web servers can take whatever you throw at them, the batches can be rebuilt, but the database has to stay idle on purpose. Not because I love idle databases, but because the day a database actually starts to struggle is a day with very few good options. This article is about what "keep the database idle on purpose" actually means in practice, and about one particular kind of graph that, in my experience, almost nobody is watching. The three layers and what each of them gets I think of a production system as having three tiers, and each tier gets a different rule. The web server tier can be horizontally scaled. If load grows, you add machines. If something is wrong, you take a machine out of the pool, and the others handle it. Failures here are visible immediately, and they're cheap to recover from. The batch server tier can be scaled up or out depending on the work. A batch that's too slow can be split. A batch that crashes can be retried. End users don't see batch servers, so a stuck batch is a problem for me and not for them. Some headroom up here is fine. The database tier is the one I treat completely differently. The database is not where you absorb load. The database is what you protect from load. The reason is simple: the other tiers can be rebuilt or re-scaled. The database is the irreplaceable record. If it slows down, everything slows down. If it falls over, you don't have many minutes before the rest of the stack notices. So my rule for the database is: keep it idle. Not idle in the sense of "doing nothing." Idle in the sense of "running well below its capacity, at all times, so that any extra load it picks up has somewhere to go." For more than a decade I ran a large appliance-grade database where I kept the load average below 1 at all times. N

Hackathons are intense. You've got a brilliant idea, a tight deadline, and often, limited sleep. The last thing you want is to spend half your precious time wrestling with database boilerplate, ORM setup, or SQL query syntax. This guide will walk you through building a functional Node.js backend for your hackathon project, focusing on speed and minimal friction, so you can spend more time on your core idea. The Hackathon Backend Challenge Typically, setting up a database and its interaction layer involves several steps: Schema Definition: Deciding on tables/collections, fields, types, and relationships. ORM/Driver Setup: Installing and configuring your database driver or ORM (e.g., Mongoose, Sequelize). Model Creation: Translating your schema into code, often with verbose syntax. Query Writing: Crafting SELECT , INSERT , UPDATE , DELETE statements or ORM methods for every data operation. Debugging: Fixing typos, schema mismatches, and complex join logic. This process, while fundamental, eats up valuable time that could be spent on features, UI, or even sleep. For a hackathon, you need to iterate rapidly, and database interactions should be the least of your worries. Strategy 1: Embrace Simplicity For many hackathon projects, you don't need highly optimized, production-grade queries from day one. You need functional queries that work quickly. Focus on getting data in and out reliably. Strategy 2: Natural Language for Data Modeling Instead of writing verbose schema definitions, think about how you'd describe your data to a non-technical person. For example, if you're building a task management app, you might say: "We need a collection of tasks. Each task has a title, a description, a due date, and a status (like 'pending' or 'completed'). Each task belongs to one user." This natural language description contains all the essential information for a data model, including relationships and field types. Strategy 3: Expressive Querying Similarly, when you need to fetch dat

Introdução Validar dados é uma responsabilidade que pode ficar na aplicação, no banco de dados, ou em ambos. Deixar tudo na aplicação é arriscado: diferentes sistemas podem acessar o mesmo banco, migrações podem rodar diretamente, um bug pode deixar passar um valor inválido. Constraints são regras definidas no próprio banco de dados — uma camada de proteção que age independente de quem está escrevendo os dados. PRIMARY KEY A chave primária identifica cada linha de forma única. Ela combina duas restrições implicitamente: NOT NULL e UNIQUE . Nenhuma linha pode ter o mesmo valor de chave primária, e nenhuma pode tê-la nula. CREATE TABLE clientes ( id INT PRIMARY KEY , nome VARCHAR ( 100 ) NOT NULL ); Quando a chave primária envolve mais de uma coluna, ela é declarada separadamente: CREATE TABLE matriculas ( aluno_id INT , curso_id INT , PRIMARY KEY ( aluno_id , curso_id ) ); Na maioria dos bancos, é comum usar uma chave primária auto-incremental para não precisar gerenciar os IDs manualmente: -- PostgreSQL id SERIAL PRIMARY KEY -- MySQL id INT AUTO_INCREMENTPRIMARY KEY -- SQL padrão (suportado por ambos) id INT GENERATED ALWAYS AS IDENTITY PRIMARY KEY FOREIGN KEY A chave estrangeira garante integridade referencial : um valor só pode existir numa coluna se ele existir como chave primária na tabela referenciada. É o que torna os relacionamentos entre tabelas confiáveis. CREATE TABLE pedidos ( id INT PRIMARY KEY , cliente_idINT REFERENCES clientes ( id ) ); Tentar inserir um pedido com cliente_id = 99 quando não existe cliente com esse id resulta em erro imediato. O banco rejeita a operação antes mesmo de ela chegar ao disco. O comportamento quando o registro referenciado é deletado pode ser configurado: CREATE TABLE pedidos ( id INT PRIMARY KEY , cliente_id INT REFERENCES clientes ( id ) ON DELETE CASCADE -- deleta os pedidos junto com o cliente ON UPDATE CASCADE -- atualiza o cliente_id se o id do cliente mudar ); As opções disponíveis são: Opção Comportamento RESTRICT

Introdução Consultas individuais respondem perguntas como "qual o email do cliente 42?". Mas as perguntas mais valiosas em qualquer sistema são de outro tipo: "qual o produto mais vendido?", "qual a receita média por pedido?", "quantos clientes se cadastraram esse mês?". Para responder isso, o SQL oferece as funções de agregação — operações que recebem um conjunto de linhas e devolvem um único valor resumido. Para os exemplos a seguir, considere esta tabela: pedidos: | id | cliente | produto | categoria | quantidade | valor | |----|------------|-------------|--------------|------------|--------| | 1 | Ana Lima | Notebook | Eletrônicos | 1 | 3500.00| | 2 | Ana Lima | Mouse | Periféricos | 2 | 80.00| | 3 | Bruno Melo | Teclado | Periféricos | 1 | 150.00| | 4 | Bruno Melo | Notebook | Eletrônicos | 1 | 3500.00| | 5 | Carla Nunes| Monitor | Eletrônicos | 2 | 1200.00| | 6 | Carla Nunes| Mouse | Periféricos | 1 | 80.00| As Funções de Agregação COUNT Conta o número de linhas — ou de valores não nulos em uma coluna específica. -- Total de pedidos SELECT COUNT ( * ) AS total_pedidosFROM pedidos ; -- Resultado: 6 -- Clientes distintos que fizeram pedidos SELECT COUNT ( DISTINCT cliente ) AS clientes_unicosFROM pedidos ; -- Resultado: 3 COUNT(*) conta todas as linhas, incluindo as que têm nulos. COUNT(coluna) conta apenas as linhas onde aquela coluna não é nula. COUNT(DISTINCT coluna) conta valores únicos — útil para saber quantos clientes, produtos ou categorias distintos aparecem no resultado. SUM Soma os valores de uma coluna numérica. -- Receita total SELECT SUM ( valor ) AS receita_total FROM pedidos ; -- Resultado: 8510.00 -- Total de itens vendidos SELECT SUM ( quantidade ) AS itens_vendidos FROM pedidos ; -- Resultado: 8 AVG Calcula a média aritmética dos valores. -- Valor médio por pedido SELECT AVG ( valor ) AS ticket_medio FROM pedidos ; -- Resultado: 1418.33 AVG ignora valores nulos automaticamente — calcula a média apenas sobre os registros que têm valor preenchid

No hacking — a passive scan only looks at what your browser already downloads when it opens a page. Here's what I found: 6 of 15 load their Supabase database directly client-side. The public API key sits in the page source. That's fine if Row-Level Security is configured right — but it's one wrong setting away from "anyone can read the whole table." 14 of 15 ship no Content-Security-Policy — a simple, high-value hardening against script injection, almost always missing. Is this theoretical? No. Two apps I audited with the owner's permission: A social app: the profiles table — user names, cities, and a password hash — readable by a logged-out stranger. Closed in an afternoon. A paid learning app: 155 paid study sheets and 4,872 answers were readable by anyone, with no account and no subscription — its entire paid catalogue, a single API call away. The paywall lived only in the front-end; the database served everything to everyone. Loading Supabase in the browser isn't the mistake. Not enforcing access in the database (RLS) is. And the tools you build with won't tell you — they'll happily ship it. If you built something on Lovable / Bolt / Replit with real users (or paying ones), it's worth 60 seconds to check what a stranger can already see. I made a free tool that runs the surface check (passive, no signup): sealdy.dev Happy to answer questions on how RLS leaks happen and how to lock them down.

We Taught a Snowflake Warehouse to Judge World Cup Conviction and Write the Verdict Back to Solana Soumyadeep Dey Soumyadeep Dey Soumyadeep Dey Follow Jul 12 We Taught a Snowflake Warehouse to Judge World Cup Conviction and Write the Verdict Back to Solana # devchallenge # weekendchallenge # snowflake 5 reactions 1 comment 16 min read

Losing PostgreSQL Gains? Blame Inline JSONB!! PostgreSQL's jsonb is a favorite among developers for its flexibility - but it hides a dark side. When used carelessly, especially in-line within rows under 2KB, it can silently destroy performance, even if you're using indexes. Here's why. 🔍 The Hidden Cost of JSONB (Inline Storage) PostgreSQL stores table rows in 8KB pages, packing as many tuples as possible. For a typical row with 10–12 columns, and small text/integers, 40–100 rows can easily fit per page. Typically row count = Page Size(8kb) / row size + row metadata (30-50 bytes approx.) But the game changes when you add jsonb. Example CREATE TABLE events ( id serial PRIMARY KEY, user_id int, action text, metadata jsonb ); Suppose metadata which is a jsonb column contains: { "ip": "127.0.0.1", "device": "Android", "country": "IN" } This JSON might be just 100–500 bytes, so PostgreSQL stores it in-line inside the same page (no TOASTing). Result Each row size jumps from ~80 bytes → ~200–400 bytes Row count per page drops from 100 → 20–40 Index scan still needs to read each page for matching rows More pages = more I/O, slower performance 🔢 Real Benchmark Insight Performance comparisonEven with a GIN or B-tree index on the JSONB column, PostgreSQL still needs to scan all matching pages to retrieve the full tuple. 🧠 Why Index Doesn't Save You Say you index a JSONB key like: CREATE INDEX ON events ((metadata->>'ip')); And query: SELECT * FROM events WHERE metadata->>'ip' = '127.0.0.1'; PostgreSQL will: Use the index to find matching tuples Still need to fetch the row from disk Because JSONB is in-line, many pages are touched More page fetches = more IO = slower queries 🩹 What You Can Do ✅ Force TOAST: Add padding to make JSONB exceed 2KB: UPDATE events SET metadata = metadata || jsonb_build_object('padding', repeat('x', 2000)); ✅ Split into separate table: If JSONB is rarely queried ✅ Stick to well defined schema and avoid using jsonb unless absolutely necessary. 🧾 TL;DR

In Part 2 , we learned why AI sometimes hallucinates. One of the biggest reasons is that an LLM can only answer based on what it learned during training and the information available in its context window. We also introduced grounding -providing the model with reliable information at runtime instead of expecting it to know everything. But that raises an important question: Where does that information come from? Modern AI applications don't simply dump an entire database or a thousand-page PDF into the prompt. Instead, they first identify the most relevant pieces of information and only send those to the model. In this article, we'll learn how that works. Running Example Let's continue building our AI-powered Travel Planner . So far, it can answer general travel questions using the knowledge it learned during training. Now we want to make it much smarter by uploading several documents into our application: Lonely Planet's Japan travel guide A PDF containing train schedules A document listing recommended local restaurants Hotel information Internal travel policies for our company Together, these documents contain hundreds of pages. Now the user asks: I'm staying near Tokyo Station. Which ramen restaurant from our travel guide is within walking distance and is known for vegetarian options? Somewhere in those hundreds of pages is the answer. The challenge is no longer generating text-it's finding the right information first. The Problem: An LLM Can't Read Your Entire Knowledge Base Every Time A common misconception is that AI applications simply send all their documents to the model. Imagine our travel guide contains 450 pages, thousands of restaurant listings, hotel descriptions, transportation details, and sightseeing recommendations. Sending all of that to the LLM every time someone asks "Where should I eat tonight?" creates several problems. First, many documents are simply too large to fit inside the model's context window. Second, even if they did fit, making the

Every data strategy document written this decade contains some version of the same sentence: we need a single place to access all our data. The sentence is right. The trouble starts on the next page, because there are two fundamentally different ways to build that single place, and the industry has spent years arguing about them as if they were rivals. Road one is consolidation: bring the data together. Land everything in one governed store, in this era an open lakehouse, Apache Iceberg tables on object storage, and point every consumer at it. Road two is federation: leave the data where it lives and bring the access together instead. A query engine that speaks to your databases, warehouses, lakes, and applications in place, presenting one surface over many sources, with no copies made. I work at Dremio, a company whose platform is built on the conviction that this is a false choice, that the right architecture uses both roads with judgment, and I will declare that bias now and then earn it with an honest treatment. Because the truth practitioners live is messier than either camp's marketing: federation without a lakehouse hits performance and scale ceilings, a lakehouse without federation spends years and fortunes migrating the long tail, and the teams that win treat the two as phases and partners rather than competitors. So this article is the full playbook. What federation and the lakehouse each actually are, mechanically. The honest strengths and limits of each, including the failure modes their advocates gloss over. A concrete decision framework for when each one carries a workload. The lifecycle pattern that connects them, federate first, promote deliberately. And the unified architectures, mine included, that put both behind one governed door, which matters more than ever now that the consumers walking through that door increasingly are AI agents. Why Unify at All: The Cost of the Status Quo Before the two roads, the destination deserves a paragraph, because

Let's be real: our personal medical history is a mess. It’s a chaotic mix of PDF lab results, grainy scans of prescriptions, and cryptic Electronic Medical Records (EMR) scattered across different hospital portals. If you’ve ever tried to remember exactly when a specific symptom started or how your cholesterol has trended over the last decade, you know the "search" struggle is real. In this guide, we are moving beyond simple folders. We are architecting a Personal Health Knowledge Base using a modern Vector Database and RAG (Retrieval-Augmented Generation) pipeline. We’ll leverage Qdrant for high-performance similarity search, Unstructured.io for complex document parsing, and Sentence-Transformers to turn 10 years of medical jargon into searchable embeddings. By the end of this post, you'll have a system capable of cross-year symptom correlation and instant medical history retrieval. The Architecture: From Pixels to Insights 🏗️ The biggest challenge with medical records isn't storage; it's ingestion . Medical PDFs are notoriously difficult to parse because they often contain nested tables and checkboxes. Our pipeline handles this by isolating the layout before embedding. graph TD A[Raw Medical Data: PDFs, Scans, EMRs] --> B[Unstructured.io: Partitioning & OCR] B --> C[Text Chunking & Cleaning] C --> D[Sentence-Transformers: Vector Embedding] D --> E[(Qdrant Vector DB)] F[User Query: 'Show me my blood sugar trends since 2015'] --> G[FastAPI Interface] G --> H[Query Embedding] H --> I[Vector Search in Qdrant] I --> J[Contextual Results + LLM Synthesis] J --> K[Actionable Health Insight] Prerequisites 🛠️ To follow along, you'll need: Python 3.9+ Unstructured.io : For the heavy lifting of PDF/Image parsing. Qdrant : Our vector engine (run it via Docker: docker run -p 6333:6333 qdrant/qdrant ). Sentence-Transformers : To generate local embeddings without sending sensitive data to the cloud. FastAPI : To wrap it all in a slick API. Step 1: Parsing the Chaos with Unstructu

It's Friday evening. An endpoint that normally answers in 200 milliseconds is suddenly taking eight seconds. You open Grafana. Every graph is green. CPU is calm, memory is fine, the disk isn't full. By every dashboard you have, the database is healthy. It is not healthy. This is the failure mode monitoring is worst at: the server is unmistakably alive , so nothing alerts, while inside the database something is slowly rotting. A table has bloated. An index nobody uses is dragging down every INSERT . A forgotten transaction is sitting open, holding a lock and quietly making everything worse. None of it crashes. It just degrades, a little at a time, until one Friday evening it tips over. The good news is that Postgres will tell you all of this — you just have to ask. The queries below run on bare PostgreSQL (13 or newer; one version note along the way), need no agent and no paid monitoring, and use an extension in exactly one place where it genuinely earns it. Open psql and check your own database as you read. 1. The cheapest signal: dead rows Start here, because it costs nothing and catches the most. Postgres never deletes a row in place. An UPDATE or DELETE leaves behind a dead tuple — an old version of the row — and autovacuum cleans those up later. Until it does (or if it can't keep up), the dead rows sit in the table, taking space and forcing every scan to page past them. The fastest look is pg_stat_user_tables , always available, no extension: SELECT schemaname , relname AS table , n_live_tup , n_dead_tup , round ( n_dead_tup * 100 . 0 / nullif ( n_live_tup + n_dead_tup , 0 ), 1 ) AS dead_ratio , last_autovacuum FROM pg_stat_user_tables WHERE n_dead_tup > 0 ORDER BY n_dead_tup DESC LIMIT 20 ; A dead_ratio above ~20% on a large table is worth investigating. And watch for a table where the ratio is high and last_autovacuum is empty — that means autovacuum has never successfully run on it, which is its own red flag (we'll see why in section 5; the whole story conver