标签：#base

Every data strategy document written this decade contains some version of the same sentence: we need a single place to access all our data. The sentence is right. The trouble starts on the next page, because there are two fundamentally different ways to build that single place, and the industry has spent years arguing about them as if they were rivals. Road one is consolidation: bring the data together. Land everything in one governed store, in this era an open lakehouse, Apache Iceberg tables on object storage, and point every consumer at it. Road two is federation: leave the data where it lives and bring the access together instead. A query engine that speaks to your databases, warehouses, lakes, and applications in place, presenting one surface over many sources, with no copies made. I work at Dremio, a company whose platform is built on the conviction that this is a false choice, that the right architecture uses both roads with judgment, and I will declare that bias now and then earn it with an honest treatment. Because the truth practitioners live is messier than either camp's marketing: federation without a lakehouse hits performance and scale ceilings, a lakehouse without federation spends years and fortunes migrating the long tail, and the teams that win treat the two as phases and partners rather than competitors. So this article is the full playbook. What federation and the lakehouse each actually are, mechanically. The honest strengths and limits of each, including the failure modes their advocates gloss over. A concrete decision framework for when each one carries a workload. The lifecycle pattern that connects them, federate first, promote deliberately. And the unified architectures, mine included, that put both behind one governed door, which matters more than ever now that the consumers walking through that door increasingly are AI agents. Why Unify at All: The Cost of the Status Quo Before the two roads, the destination deserves a paragraph, because

Let's be real: our personal medical history is a mess. It’s a chaotic mix of PDF lab results, grainy scans of prescriptions, and cryptic Electronic Medical Records (EMR) scattered across different hospital portals. If you’ve ever tried to remember exactly when a specific symptom started or how your cholesterol has trended over the last decade, you know the "search" struggle is real. In this guide, we are moving beyond simple folders. We are architecting a Personal Health Knowledge Base using a modern Vector Database and RAG (Retrieval-Augmented Generation) pipeline. We’ll leverage Qdrant for high-performance similarity search, Unstructured.io for complex document parsing, and Sentence-Transformers to turn 10 years of medical jargon into searchable embeddings. By the end of this post, you'll have a system capable of cross-year symptom correlation and instant medical history retrieval. The Architecture: From Pixels to Insights 🏗️ The biggest challenge with medical records isn't storage; it's ingestion . Medical PDFs are notoriously difficult to parse because they often contain nested tables and checkboxes. Our pipeline handles this by isolating the layout before embedding. graph TD A[Raw Medical Data: PDFs, Scans, EMRs] --> B[Unstructured.io: Partitioning & OCR] B --> C[Text Chunking & Cleaning] C --> D[Sentence-Transformers: Vector Embedding] D --> E[(Qdrant Vector DB)] F[User Query: 'Show me my blood sugar trends since 2015'] --> G[FastAPI Interface] G --> H[Query Embedding] H --> I[Vector Search in Qdrant] I --> J[Contextual Results + LLM Synthesis] J --> K[Actionable Health Insight] Prerequisites 🛠️ To follow along, you'll need: Python 3.9+ Unstructured.io : For the heavy lifting of PDF/Image parsing. Qdrant : Our vector engine (run it via Docker: docker run -p 6333:6333 qdrant/qdrant ). Sentence-Transformers : To generate local embeddings without sending sensitive data to the cloud. FastAPI : To wrap it all in a slick API. Step 1: Parsing the Chaos with Unstructu

It's Friday evening. An endpoint that normally answers in 200 milliseconds is suddenly taking eight seconds. You open Grafana. Every graph is green. CPU is calm, memory is fine, the disk isn't full. By every dashboard you have, the database is healthy. It is not healthy. This is the failure mode monitoring is worst at: the server is unmistakably alive , so nothing alerts, while inside the database something is slowly rotting. A table has bloated. An index nobody uses is dragging down every INSERT . A forgotten transaction is sitting open, holding a lock and quietly making everything worse. None of it crashes. It just degrades, a little at a time, until one Friday evening it tips over. The good news is that Postgres will tell you all of this — you just have to ask. The queries below run on bare PostgreSQL (13 or newer; one version note along the way), need no agent and no paid monitoring, and use an extension in exactly one place where it genuinely earns it. Open psql and check your own database as you read. 1. The cheapest signal: dead rows Start here, because it costs nothing and catches the most. Postgres never deletes a row in place. An UPDATE or DELETE leaves behind a dead tuple — an old version of the row — and autovacuum cleans those up later. Until it does (or if it can't keep up), the dead rows sit in the table, taking space and forcing every scan to page past them. The fastest look is pg_stat_user_tables , always available, no extension: SELECT schemaname , relname AS table , n_live_tup , n_dead_tup , round ( n_dead_tup * 100 . 0 / nullif ( n_live_tup + n_dead_tup , 0 ), 1 ) AS dead_ratio , last_autovacuum FROM pg_stat_user_tables WHERE n_dead_tup > 0 ORDER BY n_dead_tup DESC LIMIT 20 ; A dead_ratio above ~20% on a large table is worth investigating. And watch for a table where the ratio is high and last_autovacuum is empty — that means autovacuum has never successfully run on it, which is its own red flag (we'll see why in section 5; the whole story conver

Reddit looks simple and hides two genuinely hard problems. Comments nest arbitrarily deep, and a naive tree structure makes loading a busy thread slow. The front page reorders itself constantly, so ranking cannot just count votes or old posts would never leave. Both problems have well-known answers, and both are good lessons in choosing the right model. The core problem A comment thread is a tree. Each comment can reply to any other, so depth is unbounded. If you store only "this comment's parent id" and then try to load a whole thread, you walk the tree one level at a time, one query per level, which gets slow for deep or wide threads. Loading a popular post with thousands of nested comments should not take thousands of queries. Ranking is the second problem. If the front page sorted by raw vote count, the highest-voted post of all time would sit at the top forever. If it sorted by newest, quality would drown in noise. You need a score that blends how good a post is with how fresh it is, so good new posts can climb and old ones fade even if they were once popular. Key design decisions Store the parent pointer, but do not traverse at read time. The simple model is a parent_id per comment, which is easy to write but expensive to read as a tree. To load a thread cheaply, fetch all comments for the post in one query, then assemble the tree in application memory. One read, in-memory tree building. This works because a single post's comments, while numerous, fit in memory to assemble. Consider a path or closure model for deep trees. For very deep threads, some systems store a materialized path on each comment, an encoded ancestor chain, so you can fetch an entire subtree with a single prefix query and sort by the path to get correct display order. Another option is a closure table that records every ancestor-descendant pair, which makes subtree queries direct at the cost of extra write work. The right choice depends on how deep threads get and how often you read subtrees

While doing a routine hardening pass on an internal Firebase panel — codename PanelControl , a management tool used daily by multiple operators with different roles — what was supposed to be "let's add a few Telegram alerts for suspicious activity" turned into discovering that the app's entire login system was just a UI filter. Anyone who opened the site already had, automatically, a Firebase identity with full read/write access to the database. Here's what happened, and how it got fixed in 5 phases without ever locking the team out mid-shift. The setup PanelControl is a vanilla-JS internal panel backed by Firebase Realtime Database + Firestore. Operators log in with email/password, checked client-side against a database node, with a lockout after failed attempts. Nothing unusual so far. The original ask was narrow: add Telegram notifications for a handful of suspicious events — brute-force attempts, a never-before-seen device for an operator, an unauthorized attempt to reach the Admin section, DevTools opened during use. Pure alerting work. Bug #1: the login button that always unlocks Before writing any alerting logic, a review of the existing Admin-area password check turned up this: // ❌ The "|| true" makes the whole condition always truthy function checkAdminPwd () { if ( el . value || true ) { unlockAdmin (); // runs regardless of what's typed, or nothing at all } } A debug leftover that made it to production. Anyone who landed on the Admin password overlay got in by clicking "Log in" — password or not. Fixed by actually wiring the real permission check, plus a server-side-verified fallback in case the function were ever called directly from the console. The real discovery: a shared, hardcoded Firebase credential Looking at the Realtime Database Rules ahead of the alerting work surfaced something much bigger. The Rules restricted read/write to a single fixed auth.uid — reasonable, until you check who actually gets that uid . This ran unconditionally, for every

When Neon became the official database partner of DEV Community, I was already a user. But the partnership made me look closer at why I chose Neon — and whether those reasons apply to other AI developers. They do. Here's why Neon is the ideal database for AI applications in 2026. The Problem: AI Apps Have Unique Database Needs AI applications have database requirements that traditional web apps don't: High write volume — every AI request generates logs, metrics, and cost data Variable load — traffic spikes when a model goes viral, then drops to zero Schema evolution — you're constantly adding models, routing rules, and analytics tables Dev/prod parity — you need to test routing changes against real production data Edge compatibility — AI APIs need sub-100ms response times globally Traditional PostgreSQL (RDS, Aurora) struggles with all five. Neon was built for them. Feature 1: Database Branching (The Game-Changer) This is Neon's killer feature. It works like git branch but for your entire database: # Create a branch from production neon branches create --parent main --name test-deepseek-v31 # Get a connection string for the branch neon connection-string test-deepseek-v31 # → postgresql://...@ep-test-deepseek...neon.tech/neondb # Run migrations on the branch npx prisma db push --url $BRANCH_URL # Test your new routing algorithm against REAL data # (the branch is a copy-on-write clone of production) # When tests pass, merge neon branches merge test-deepseek-v31 Why This Matters for AI Apps When I added DeepSeek V3.1 to my model pool, I needed to test: Would the new model break existing routing rules? Would the cost calculations be correct? Would the latency meet my SLA? With traditional PostgreSQL, testing against real data meant either: Copying production to a staging DB (hours, $$) Testing with synthetic data (unreliable) With Neon branching, I branched, tested in 30 seconds, and merged. Zero downtime, zero risk. Feature 2: Scale-to-Zero (Cost Optimization) Neon's c

I recently commented on Jonathan Lewis’s blog, Savepoint Funny , where I compared how PostgreSQL handles uniqueness differently: “PostgreSQL resolves uniqueness through heap tuple visibility". This deserves a more detailed explanation. In Oracle, unique indexes store unique entries because the B-tree key is the index key, preventing duplicates. Non-unique indexes add the ROWID to ensure that all entries are physically unique, even when indexed column values are duplicated. In PostgreSQL, all indexes, even unique ones, created explicitly by CREATE UNIQUE INDEX or implicitly to enforce a unique constraint, behave like non-unique indexes by appending the TID (tuple ID, similar to Oracle's ROWID) to the index key. This indicates that the index itself doesn't guarantee physical uniqueness, allowing multiple entries to have identical logical keys but point to different heap tuples. The actual uniqueness verification occurs at the heap level, not within the index entries. Initially, this might seem unusual—a unique index that permits duplicates. However, PostgreSQL requires this because of its MVCC system. MVCC allows duplicate entries to coexist in an index, since they can represent different versions of the same logical row. Still, PostgreSQL must guarantee that no MVCC snapshot views two rows with the same index key. Oracle doesn't face this issue because its MVCC implementation also versions index blocks, allowing a single index version to maintain unique keys. Let’s show that. Page inspect In PostgreSQL, the heap contains the table data, and index entries point to heap tuples. Visibility depends on the heap header, especially the transaction information. Index scans often visit the heap pages to check visibility, except for index-only scans, which use the heap's visibility maps as an optimization. B-tree indexes can store entries for multiple versions of the same logical row, including versions that are no longer visible to current snapshots. To ensure uniqueness, the

Episode 1/4 — 3 incidents, one root: default GRANTs open more than you think — [CANONICAL URL EPISODE 1: fill in after push] Episode 2/4 — await mutation() lies when nobody opens the { error } envelope — [CANONICAL URL EPISODE 2: fill in after push] The morning Françoise sees zero rows, again It's a Tuesday in April 2026. I've just added the agent_readonly role to the authenticated membership — a one-liner, meant to share a GRANT for a reporting job. First SELECT on cours , Sentry receives infinite recursion detected in policy for relation "user_roles" , code 42P17 . From the office next door, Françoise is already on the phone with the Maisons-Laffitte branch: "So they can't see anything over there — is that normal?" Foreman tone, not really a question. I read the error on my screen. The difference from episode 1: this time Postgres is talking. What came out of Sentry was no longer a silent empty set — it was an explicit error. That difference saved me two days. When Postgres shouts, you listen. The trap is that what it says isn't where you're looking. I won't pretend this is obscure. A policy on user_roles that queries user_roles to decide who can read user_roles is a loop. You avoid it, you work around it with SECURITY DEFINER , you move on. The problem: my user_roles policy didn't reference user_roles . I had already cleaned it up three weeks earlier. The recursion was coming from somewhere else. The diagnostic that targets the wrong object First reflex: re-read the user_roles policy. It's clean, reads auth.email() , never calls itself. Second reflex: disable policies one by one to find the culprit. Wrong angle. -- supabase/migrations/20260420_admin_write_cours_v1.sql -- "Admin write cours" policy — original version that loops CREATE POLICY "Admin write cours" ON public . cours FOR ALL TO authenticated USING ( EXISTS ( SELECT 1 FROM public . user_roles WHERE email = auth . email () AND role IN ( 'admin' , 'super_admin' ) ) ); The recursion doesn't come from a fau

The fourth call of the week Catherine calls from the Maisons-Laffitte site on a Tuesday afternoon in early May. "It's broken, but it's a quick fix." That's her line — I know it, and she's usually right. She describes it in three sentences: the newsletter export for the enrolled-students segment comes back with ninety-two names, the planning view shows ninety-two active courses, but the counter page shows eighty-nine. Three enrolled students missing. She'd checked the database directly — they're all there. "Why three steps for that?" She's not asking for my benefit. She's asking for herself. Except this time, hanging up, I realize it's the fourth time this week I've hung up thinking the same thing. Four Supabase incidents, four fixes, four closed tickets. And not a single exception raised by the database. I reopen the three previous ones and lay all four side by side on screen. This isn't four bugs. It's one failure mode, declinated four times. The first three Episode 1 was about the default GRANT s Supabase places on functions and policies. A SQL function created without an explicit REVOKE inherits anon access that nobody wrote in the migration, and that nobody caught in review because the diff doesn't show it. The function works. It's just callable from outside. [CANONICAL URL EPISODE 1: to fill in after publication of #48 — "3 Supabase security incidents, one shared root cause: SECURITY DEFINER inherits EXECUTE TO PUBLIC"] Episode 2, an ON DELETE SET NULL cascade coupled with a CHECK NOT NULL on the target column. The parent DELETE attempts the SET NULL , the CHECK rejects it, and the transaction surfaces an error we read as a deletion failure — while it actually masks a consistency assumption we'd held for three months. The query fails loudly, which is more charitable than the other three cases, but the diagnosis heads in the wrong direction because nobody had declared that the two constraints lived in tension. [CANONICAL URL EPISODE 2: to fill in after publicati

Google shipped AlloyDB AI functions GA with a proxy model architecture that trains a lightweight local model from LLM outputs, then runs queries at database speed without external calls. Smart batching delivers 2,400x throughput improvement. The proxy model reaches 100,000 rows per second in preview, but benchmark numbers apply only to ai.if in internal testing. By Steef-Jan Wiggers

Most "Debezium vs managed tool" articles get the question wrong. They frame it as a product bake-off, feature grid included, and declare a winner. But if you've actually run change data capture in production, you know the real decision isn't which tool captures a transaction log better. They mostly read the same logs the same way. The real decision is who operates everything that sits around the capture, and whether that work is a good use of your team's time. That's a build-vs-buy question, not a product question. This post is a framework for answering it for your own situation. First, let's kill an outdated assumption A lot of Debezium criticism floating around is two or three years stale, and if you repeat it in 2026 you'll get corrected fast. So let's set the record straight before we compare anything. Debezium is no longer just “the thing you run with Kafka Connect.” In recent Debezium 3.x releases, the project has become much more flexible than the old tutorials suggest. Today, you have several deployment options: Kafka Connect , the classic setup, which gives you the Kafka ecosystem, distributed fault tolerance, durable schema history, and access to Kafka Connect sink connectors. Debezium Server , a standalone application that streams changes to systems like Amazon Kinesis, Google Cloud Pub/Sub, Apache Pulsar, Redis Streams, or NATS JetStream without requiring Kafka. Debezium Management Platform , which builds on Debezium Server and the Debezium Operator to provide a higher-level way to configure and manage CDC pipelines in Kubernetes-style environments. Embedded usage , where you run Debezium Engine inside your own application. Recent Debezium releases also added framework support such as the Quarkus extension. A few more things are worth knowing so the comparison is fair: Kafka 4.x runs in KRaft mode, and ZooKeeper mode has been removed. “You need to babysit ZooKeeper” is no longer true for a modern Kafka deployment. Debezium's default remains at-least-once

SaaS software vendor HubSpot has described how its semantic search platform grew from a proof of concept into an internal service that now manages more than 20 billion vectors across 38-plus teams. The company says the system now supports agents, RAG, and contact deduplication, and that the increase in agent usage has made retrieval quality and latency more important than before. By Matt Saunders

I built a small Redis clone in C: a RESP parser, a command table, an append-only file for persistence. Recently I started building the same thing again in Rust, and rebuilding a project I had already finished has taught me more Rust than any from-scratch tutorial. The reason is simple. The second time, the design is already solved. I know what the AOF has to guarantee, what the command table dispatches, what the parser must reject. So none of my attention goes to what to build. All of it goes to how Rust wants it built. That turns the domain into a constant and the language into the only variable. Every difference I hit is pure signal about Rust, not noise about key-value stores. The first difference shows up before any logic runs. In C, I built the substrate first: my own dynamic strings, my own hashmap, my own linked list. Hundreds of lines before a single command worked. In Rust, Vec , String , and HashMap are just there, so that whole layer disappears and I start at the actual command logic. A standard library quietly decides where your project even begins. The sharper difference is in dispatch. In C it is a switch with argument counts I check by hand: if ( argc != 3 ) return err ( "wrong arg count" ); switch ( cmd ) { case CMD_SET : return do_set ( argv [ 1 ], argv [ 2 ]); case CMD_GET : return do_get ( argv [ 1 ]); /* forget a case and it is a runtime bug */ } In Rust the same dispatch is an enum and a match, and the compiler will not build until every case is handled: match cmd { Command :: Set { key , val } => self .set ( key , val ), Command :: Get { key } => self .get ( key ), } Same dispatch. One version cannot ship the missing-case bug I actually shipped in C. If you already know a project cold, rebuild it in the language you are learning. You stop thinking about the problem and start feeling the language.

PostgreSQL provides several planner configuration parameters, such as enable_seqscan and enable_indexscan , that influence how execution plans are generated. These settings affect planning, not the execution of an already-generated plan. With prepared statements, this raises an interesting question. Should planner settings be applied before PREPARE, before EXECUTE, or both? Let's look at a simple example: a "tasks" table with a due date and a "done" status: \ c drop table if exists tasks ; -- a table of tasks with status (done or not) and due date create table tasks ( id bigint generated always as identity primary key , due timestamptz , done boolean ); -- insert 500 tasks, with 1% not done insert into tasks ( due , done ) select now () + interval '1 day' * n , 42 != n % 100 from generate_series ( 1 , 500 ) n ; -- index the todo (partial index) create index on tasks ( due , id ) where done = false ; vacuum analyze tasks ; With a partial index, I indexed only the tasks that are not yet done ( done = false ) because that's my most frequent query pattern: postgres =# explain select id , due , done from tasks where done = false and id > 0 order by due limit 1 ; QUERY PLAN --------------------------------------------------------------------------------------- Limit ( cost = 0 . 13 .. 3 . 60 rows = 1 width = 17 ) -> Index Scan using tasks_due_id_idx1 on tasks ( cost = 0 . 13 .. 17 . 47 rows = 5 width = 17 ) Index Cond : ( id > 0 ) ( 3 rows ) With partial indexes, the condition covered by the index is not even visible in the execution plan because the index itself enforces the condition. Prepared statement I decided to use a prepared statement with all values as parameters. It is probably not a good idea in this case. When a parameter can have only a few different values and you expect different cardinalities for each, you should probably define one query per value, using literals. I'm doing this to illustrate what can happen, with a simple, extreme example: postgres =# pr

Episode 1/4 of the mini-series The week Supabase lied to me 4 times . The three following episodes cover a mutation silently swallowed by the SDK [CANONICAL URL EPISODE 2: to fill in after push], an RLS recursion resolved by a JWT hook [CANONICAL URL EPISODE 3: to fill in after push], and a query that stops at exactly 1000 rows without saying so [CANONICAL URL EPISODE 4: to fill in after push]. The Tuesday the security probe spoke It's 9:12am on a Tuesday in May. The daily drift probe has been running automatically for three weeks — an aclexplode query across all public objects, filtered on anon . I don't open it every morning. That morning, it's waiting for me with a row that has no business being there. Niran sets a coffee on the corner of my desk without a word. He reads the output over my shoulder. A PII backup table — personal data in plaintext, created two days earlier for a bulk reclassification — shows up in the list with SELECT , INSERT , UPDATE , DELETE granted to anon . Accessible to any unauthenticated curl request. He lets three seconds pass and says: "It's not RLS." Then he goes back to his hoodie. He's right. It's not an RLS bug. The table itself is open, at the GRANT layer, before RLS even applies. Three objects, three doors, one mechanism That week, I realize I'm not dealing with an isolated incident. Three distinct objects, in three different migrations, each open a door nobody thought they'd opened. The backup table first. Then a policy set TO public because the public landing page needs it, which lets a POST {} from anon through with an HTTP 400 NOT NULL response instead of 401 Unauthorized . And finally four SECURITY DEFINER functions written to execute transactional operations with their owner's privileges, all invocable by anon because EXECUTE defaults to TO PUBLIC at CREATE time. Three objects, three superficially distinct mechanisms, yet one shared root. At every CREATE , Postgres completes the migration with an implicit GRANT the author nev

The problem with a data breach is not finding evidence. It is connecting it. But let me start where I actually was: 4 AM, last day of the hackathon, staring at this in my terminal. RateLimitError: GroqException - Rate limit reached for model `llama-3.3-70b-versatile` on tokens per day (TPD): Limit 100000, Used 99787, Requested 1616. Please try again in 20m12s. Used 99,787 out of 100,000. My deployment was half done, my demo graph was empty on the server, and the free tier had 213 tokens left. The submission deadline was hours away. I had not slept. I had not eaten. My friends were asleep and I was swapping API keys like a gambler swapping chips. This post is the story of how we got there, and how it ended at 7 in the morning with the best sigh of relief I have ever taken. First, some honesty about how I got here When I joined my first WeMakeDevs hackathon, I did not believe in it. I thought it was one of those ordinary online events. Fake prizes, no follow-through, what would I even get out of it. I joined anyway, mostly out of boredom, got into the Discord, talked to people, made a few connections. I landed in the top 50. A few days later an email showed up: a free Claude Max subscription as a gift. I read it twice. I genuinely could not believe a hackathon had actually delivered something. So when this hackathon opened, I did not hesitate. I messaged my friends and said we are joining as a team this time. Three of us: me (Mehraan), Aqib, and Ubaid. The spark We spent the first evening in our group chat throwing ideas around and shooting most of them down. Then one of my friends dropped a thought that stuck: what happens after a company gets hacked? I started digging into it. The answer is honestly depressing. After a breach, the evidence is everywhere. VPN records. File access logs. The email gateway. Badge readers at the office doors. CCTV. HR notes. Anonymous tips. Each system tells one small piece of the story, and a human analyst has to stitch all of it togeth

SQLite Internals, Postgres 19 Checksums, & PL/CBMBASIC Extension Today's Highlights This week, we delve into SQLite's secure deletion and blob updates, explore upcoming data integrity features in PostgreSQL 19, and discover a unique PostgreSQL extension bringing Commodore 64 BASIC to your database. These updates offer insights into database internals, future resilience, and creative extensibility for the SQLite ecosystem. Secure Delete and BLOB Updates in SQLite (SQLite Forum) Source: https://sqlite.org/forum/info/6f3e886a1149c97e0ede9a243281efb05a043705393ea94437ed7c0556315972 This SQLite forum discussion delves into the nuances of secure data deletion and efficient BLOB updates within SQLite databases. Secure deletion is a critical concern for applications handling sensitive data, where simply deleting a row might not zero-out the underlying storage, leaving recoverable remnants. The thread explores methods and implications for ensuring data is truly eradicated when removed, potentially touching on PRAGMA settings or specific file system interactions. Understanding these mechanisms is crucial for developers building secure, embedded applications with SQLite. The conversation also extends to optimizing updates for BLOB (Binary Large Object) data. Efficiently handling large binary data, such as images or documents, in an embedded database like SQLite requires careful consideration to avoid performance bottlenecks and excessive disk I/O. The discussion likely covers strategies for in-place updates, managing free space, and the internal workings of SQLite's storage engine when dealing with variable-length BLOBs. This insight helps developers make informed decisions on schema design and update patterns for improved application performance and data integrity. Comment: This thread offers valuable insights into SQLite's low-level data management, essential for anyone needing to implement robust security or optimize BLOB storage. PostgreSQL 19 to Feature Checksums For All

Introduction I used to think "connecting to a database" was one skill. Turns out it's two: connecting to a database chilling quietly on your own laptop, and connecting to one living in the cloud, behind a login, in this case, an SSL certificate that will not let you in until you treat it with respect. This week I did both. Same tool (Power BI), same dataset, two very different vibes. Grab a coffee, here's the full walkthrough local PostgreSQL first, then Aiven's cloud version, side by side, screenshots and all. Part 1: Local PostgreSQL → Power BI Step 1 : Create a schema Nothing fancy, just giving my table a home: CREATE SCHEMA powerbi ; Step 2 : Import the dataset Right-click the new schema → Import Data in DBeaver, point it at your CSV, and let the wizard do its thing. Step 3 : Check the table landed properly A quick peek at the columns to make sure nothing got mangled on the way in. Step 4 : Connect Power BI In Power BI Desktop: Get Data → Database → PostgreSQL database. In the Server field, type localhost (or 127.0.0.1 ) and your database name. localhost Choose Import , hit OK, and log in with your local username and password. Click Load . That's it. That's the whole local experience. Part 2: Aiven PostgreSQL (Cloud) → Power BI Now for the part that actually taught me something. Step 1 : Grab your connection details Everything you need lives on Aiven's Overview page: Host, Port, Database name, User, SSL mode. Your service URI will look something like this (don't worry, this isn't a real password, Aiven masks it in the console): postgres : // avnadmin : •••••••• @ pg - xxxxxxxx - yourproject . c . aivencloud . com : 22016 / defaultdb ? sslmode = require Step 2 : Import the dataset into Aiven Same DBeaver wizard as before, just pointed at the Aiven connection instead of local. CREATE SCHEMA powerbi ; Step 3 : Aiven's certificate. Download the CA cert from the Overview page: Now here's the part that actually tripped me up: Power BI's PostgreSQL connector doesn't ha

Introduction Fixing N+1 queries with select_related / prefetch_related or selectinload (see the previous post ) gets you down to a small, sane number of queries per request. The next bottleneck is what each query costs once the table has millions of rows — and that is almost always about indexing. An index turns "scan every row" into "look it up directly." Skip it, and a query that's instant in development takes seconds once real data volume shows up in production. How Indexes Work: The B-Tree Intuition Without an index, a WHERE clause forces a sequential scan : the database reads every row and checks the condition — O(n) , cost grows linearly with table size. An index is a separate, sorted structure (almost always a B-tree ) mapping column values to row locations. Because it's sorted and balanced, finding a value is a tree walk: O(log n) . On a 10-million-row table, that's the difference between reading 10 million rows and roughly 23 tree nodes. This isn't free: Writes get slower — every INSERT / UPDATE / DELETE on an indexed column also updates the index. Storage grows — each index is a sorted copy of (part of) the data. An index trades write cost and storage for read speed. Indexing a column you rarely filter or sort on is pure cost, no benefit. Reading Query Plans: EXPLAIN ANALYZE Postgres' EXPLAIN ANALYZE shows what the planner actually did, not an estimate. Before an index , filtering orders by customer_id : EXPLAIN ANALYZE SELECT * FROM orders WHERE customer_id = 48291 ; Seq Scan on orders (cost=0.00..21453.00 rows=42 width=96) (actual time=0.021..118.442 rows=41 loops=1) Filter: (customer_id = 48291) Rows Removed by Filter: 1199959 Planning Time: 0.112 ms Execution Time: 118.471 ms Seq Scan means Postgres read all ~1.2 million rows and discarded all but 41. actual time is real elapsed time — 118ms for one lookup. After CREATE INDEX idx_orders_customer_id ON orders (customer_id); : Index Scan using idx_orders_customer_id on orders (cost=0.42..8.53 rows=42 wid

Introduction Fixing N+1 queries (see the previous post ) gets your Hibernate app down to a handful of queries per request. The next bottleneck is what each of those queries costs once your tables have millions of rows — and that is almost always a question of indexing. An index turns "scan every row" into "look it up directly." Get the index wrong — or skip it — and a query that took 2ms in development takes 4 seconds in production once real data volume shows up. How Indexes Work: The B-Tree Intuition Without an index, a WHERE clause forces a sequential scan : the database reads every row and checks the condition. That's O(n) — cost grows linearly with table size. An index is a separate, sorted data structure (almost always a B-tree ) that maps column values to row locations. Because it's sorted and balanced, finding a value is a tree walk: O(log n) . On a 10-million-row table, that's the difference between reading 10 million rows and reading roughly 23 tree nodes. The cost is not free: Writes get slower. Every INSERT / UPDATE / DELETE on an indexed column must also update the index structure. Storage grows. Each index is a copy of (part of) the data, sorted differently. An index is a trade: you pay on every write so that specific reads become fast. Indexing a column you rarely filter or sort on is pure cost with no benefit. Reading Query Plans: EXPLAIN ANALYZE Postgres' EXPLAIN ANALYZE shows what the planner actually did — not what you hope it did. Before an index , filtering orders by customer_id : EXPLAIN ANALYZE SELECT * FROM orders WHERE customer_id = 48291 ; Seq Scan on orders (cost=0.00..21453.00 rows=42 width=96) (actual time=0.021..118.442 rows=41 loops=1) Filter: (customer_id = 48291) Rows Removed by Filter: 1199959 Planning Time: 0.112 ms Execution Time: 118.471 ms Seq Scan means Postgres read all ~1.2 million rows and threw away all but 41 of them. actual time is the real elapsed time, not an estimate — 118ms for one lookup. After CREATE INDEX idx_orders