标签：#Database

Most "data lineage" you've seen answers a schema question: table B comes from table A , or column B.total comes from columns A.price and A.qty . That's genuinely useful, and tools like OpenLineage do it well. But notice what it doesn't tell you: it says which columns can influence an output. It never says which values actually did . That gap is the whole subject of this post. I built a small, self-contained reference pipeline that captures provenance at the row and field level — "the value in this destination row, this field, was computed from these specific source (row, field) pairs" — and I want to walk through two things: why you'd ever want provenance at that granularity, and why it's genuinely hard once you commit to it. Repo (dbt-core + DuckDB, no server, no cloud, runs on a clean checkout): https://github.com/stevenblough/row-level-prov The one distinction everything follows from Here's the sentence the entire project turns on: Column-level lineage is a schema-sized, static fact you can derive from code. Value-level provenance is a data-sized, dynamic fact you must capture at execution. Put it in complexity terms and the consequences become obvious: Column lineage is O(schema) . It scales with how many columns you have. You can compute it by parsing SQL, offline, without ever looking at a single row. Value provenance is O(rows × fan-in) . It scales with your data volume times how many source values feed each output value. It does not exist anywhere until the query runs, and it can only be captured there , piggybacked on the query that actually produced the values. You cannot "reconstruct" value provenance later by re-querying the sources — the moment the source changes, you'd reconstruct a different answer than what really happened. That single exponent change ( schema → rows × fan-in ) is why value-level provenance has an entire class of problems that column lineage never faces. Why bother? The reasons for this level of granularity Granularity is expensive,

TimescaleDB 2.27, released May 12 2026, extends bloom-filter batch pruning from reads to writes. UPDATE, DELETE, and UPSERT against compressed columnstore data can now skip decompressing batches that provably cannot contain the target rows. The reported gains are real: up to 160x for selective UPDATE/DELETE, and over 2x for UPSERT. The feature is automatic. Whether it is actually firing on your workload is not something you can assume, and the only way to confirm it is to read new EXPLAIN counters that the release notes mention but do not explain. Worse, the counter names are inconsistent between the write paths, so even a careful reader ends up guessing. This post is about reading those counters correctly, and about the two things in this release that will silently break a query if you upgrade without noticing them. What is actually being skipped A quick model of the mechanism, because the counters only make sense against it. Hypercore stores compressed data in batches, roughly a thousand rows each. For columns that are not the segmentby key, TimescaleDB maintains a sparse bloom filter per batch: a small probabilistic summary that answers one question, "could this batch contain column = X ?", without touching the compressed payload. A bloom filter has a useful asymmetry. A negative is certain: if the filter says no, the value is definitely absent, and the batch can be skipped whole. A positive is not: the filter says "maybe", you decompress, and sometimes the value is not there after all. That last case is a false positive, and it is the number that tells you whether the whole scheme is paying off. Before 2.27, a DELETE ... WHERE sensor_id = 'x' against compressed data decompressed every candidate batch to check. Now the bloom filter is consulted first, and batches that cannot match are never decompressed. The work you save is the decompression of the batches that get pruned. The work you waste, when the filter is poorly matched to your data, is the bloom check on

Metabase is an open-source BI tool for building charts and dashboards over MySQL, PostgreSQL, MongoDB, Redshift, and more. This guide deploys Metabase on Kubernetes, loads the Sakila sample dataset into MySQL, builds a dashboard, and secures it behind Nginx Ingress with cert-manager TLS. Prerequisites: a Kubernetes cluster with kubectl / helm configured, a Linux workstation, a reachable MySQL server, and a domain name. Load the Sakila Sample Database Sakila models a DVD rental store — films, actors, inventory, rentals. $ sudo apt install zip -y $ wget https://downloads.mysql.com/docs/sakila-db.zip $ unzip sakila-db.zip Connect to your MySQL server (replace host/port/user): $ mysql -h <HOST_ENDPOINT> -P <DATABASE_PORT> -u <ADMIN_USER> -p mysql > CREATE DATABASE sakila ; mysql > SOURCE sakila - db / sakila - schema . sql ; mysql > SOURCE sakila - db / sakila - data . sql ; Deploy Metabase $ nano metabase.yaml apiVersion : apps/v1 kind : Deployment metadata : name : metabase spec : selector : matchLabels : app : metabase replicas : 1 template : metadata : labels : app : metabase spec : containers : - name : metabase image : metabase/metabase:latest ports : - containerPort : 3000 protocol : TCP --- apiVersion : v1 kind : Service metadata : name : metabase-svc spec : type : LoadBalancer selector : app : metabase ports : - name : http port : 8080 targetPort : 3000 Your cloud provider may need a provider-specific LoadBalancer annotation here (e.g. to set the listener protocol) — check its Kubernetes docs if the default doesn't work. $ kubectl apply -f metabase.yaml $ kubectl get deployments $ kubectl get services Wait for metabase-svc to get an EXTERNAL-IP (can take a few minutes), then visit http://<external-ip>:8080 to confirm the Metabase welcome page loads. Connect Metabase to the Database Let's get started → pick language. Enter your name, email, company, and a password. Select your use case. Database engine: MySQL . Set a display name, then host/port/database/user/pa

A Patroni cluster needs an odd number of nodes to maintain quorum — with 3 nodes, losing 1 still leaves a majority, so the cluster keeps running. This guide builds a 3-node PostgreSQL cluster on Ubuntu 24.04 with Patroni handling replication and automatic failover, etcd as the coordination store, and HAProxy load-balancing client connections — all secured with TLS. Prerequisites: three Ubuntu 24.04 servers (2 vCPU / 4GB RAM minimum) with PostgreSQL installed, non-root sudo access, and a domain with three A records: node1.example.com , node2.example.com , node3.example.com . Replace these placeholders with your actual subdomains throughout. Install Dependencies Run on all three nodes unless noted otherwise. 1. Install packages: $ sudo apt update $ sudo apt install haproxy certbot pipx -y $ sudo pip3 install --break-system-packages 'patroni[etcd3]' psycopg2-binary psycopg 2. Install etcd: $ wget https://github.com/etcd-io/etcd/releases/download/v3.6.4/etcd-v3.6.4-linux-amd64.tar.gz $ tar -xvf etcd-v3.6.4-linux-amd64.tar.gz $ sudo mv etcd-v3.6.4-linux-amd64/etcd etcd-v3.6.4-linux-amd64/etcdctl /usr/local/bin/ 3. Open firewall ports — 80 (Certbot), 2379/2380 (etcd), 5432/5433 (PostgreSQL + Patroni-managed PostgreSQL), 8008/8009 (Patroni REST API): $ sudo ufw allow 80,2379,2380,5432,5433,8008,8009/tcp $ sudo ufw reload $ sudo ufw status Configure SSL Certificates 1. Request a certificate per node (run on each node for its own subdomain): $ sudo certbot certonly --standalone -d node1.example.com -m admin@example.com --agree-tos --no-eff 2. Create a cert-prep script on each node (set HOSTNAME to that node's subdomain): $ sudo nano /usr/local/bin/prepare-ssl-certs.sh #!/bin/bash HOSTNAME = "node1.example.com" # Update for each node CERT_DIR = "/etc/letsencrypt/live/ $HOSTNAME " ARCHIVE_DIR = "/etc/letsencrypt/archive/ $HOSTNAME " getent group ssl-users > /dev/null || sudo groupadd ssl-users for user in etcd patroni haproxy postgres ; do if ! id " $user " > /dev/null 2>&1 &&

Manticore Search 28.6.6 has been released. The headline additions are UUID document IDs for real-time tables and ordered, limited GROUP_CONCAT() for grouped queries. The release also includes 16 fixes for backups, replication, query processing, SQL compatibility, and secondary indexes. This post covers everything shipped from 28.4.5 through 28.6.6 . Upgrade notes There are no new mandatory data migrations in this release. UUID IDs are an opt-in table definition: existing numeric-ID tables keep working as they are. If you want UUID identifiers, create a real-time table with id uuid ; ALTER TABLE cannot convert an existing table between numeric and UUID IDs. Two fixes are particularly useful for production installations. Successful backups now always unfreeze real-time tables when they finish (previously in rare cases they didn't), rather than leaving writes blocked. And authenticated replication can again add an existing populated RT table with ALTER CLUSTER ... ADD . UUID document IDs for real-time tables Applications often already have UUID identifiers from the system of record. Until now, using them with Manticore Search meant maintaining a separate numeric ID mapping. Real-time tables can now use UUID document IDs directly: CREATE TABLE products_uuid ( id uuid , title text , price int ); Manticore accepts an explicit UUID string, or generates one when id is omitted from an insert or replace. UUID equality and IN filters work in queries, and UUID IDs can be used with REPLACE , UPDATE , and DELETE . This is currently a real-time-table capability, including columnar and replicated RT tables. Plain, percolate, and sharded tables continue to use their existing ID models. Ordered and limited GROUP_CONCAT() Grouped results often need a compact preview of the most relevant values in each group. GROUP_CONCAT() can now sort values and retain only the requested number of them in explicit SQL GROUP BY queries: SELECT category , GROUP_CONCAT ( title ORDER BY price DESC SEPARA

Whether you are using off-the-shelf AI coding tools like Claude Code and Cursor or building custom autonomous AI agents with TypeScript and LLM APIs, you hit the exact same fundamental wall: AI agent amnesia . As an agent user , you spend forty-five minutes explaining your architecture, deployment quirks, and database rules. The agent writes brilliant code. You close the CLI or tab, open a new session the next morning, and the agent suggests the exact legacy library you rejected yesterday. As an agent builder , you struggle to keep your custom agentic loops focused. As multi-step agent trajectories expand, LLM token limits force context compaction, wiping out subtle rules and past decisions while escalating API costs. The intelligence is real. The amnesia is structural. Context Windows Are Working Memory, Not Long-Term Memory The AI industry’s standard reflex to agent amnesia has been pushing context windows to 1M+ tokens. But a context window is working memory (RAM), not long-term storage (disk). Relying on massive context windows introduces three critical engineering bottlenecks for both users and builders: Context Compaction Destroys Rationale : When a session reaches token limits, agents automatically compact their context history. Compaction summarizes conversations into short summaries, quietly wiping out subtle architectural constraints, edge cases, and past decisions. Context Drift & Attention Loss : LLMs struggle with needle-in-a-haystack attention degradation when context windows are stuffed with 100k+ lines of raw conversation history. Escalating API Costs & Latency : Re-sending full project transcripts on every prompt burns tokens rapidly and adds seconds of input processing delay for users while skyrocketing LLM bills for agent builders. Agents do not need larger transcripts. They need a durable, inspectable, versioned memory layer . Why Vector Databases Fall Short for Local & Workspace Agent Workflows When developers and AI engineers realize raw contex

Murat Demirbas discusses the shift toward disaggregated cloud database architectures driven by cloud economics. He explains how decoupling compute from storage enables elastic scaling, cost efficiency, and fault isolation. He shares how classical Paxos roles foreshadowed disaggregation, while analyzing network tradeoffs, shared-memory evolution, and self-assembling database designs. By Murat Demirbas

Introduction When building a Flutter application, you'll often need to store data on the user's device. For example: Saving user preferences Storing login information Caching API responses Creating offline applications Building note-taking or to-do apps While there are several local storage solutions available, Hive is one of the fastest and easiest local storage for Flutter developers. In this tutorial, you'll learn Hive from scratch by building a simple example. No prior database knowledge is required. What is Hive? Hive is a lightweight, NoSQL database written entirely in Dart. It stores data directly on the device, making it perfect for Flutter applications. Why use Hive? Extremely fast Works offline No native platform code required Simple API Easy to learn Great for small and medium-sized applications Think of Hive as a collection of boxes where each box stores your application's data. Hive ├── User Box ├── Settings Box ├── Notes Box └── Products Box Each Box is similar to a table in traditional databases. Step 1: Create a Flutter Project Create a new Flutter project. flutter create hive_demo Open the project. cd hive_demo Step 2: Install Hive Open pubspec.yaml and add the following packages. dependencies : flutter : sdk : flutter hive : ^2.2.3 hive_flutter : ^1.1.0 Then install them. flutter pub get Step 3: Initialize Hive Before using Hive, initialize it inside main() . import 'package:flutter/material.dart' ; import 'package:hive_flutter/hive_flutter.dart' ; void main () async { WidgetsFlutterBinding . ensureInitialized (); await Hive . initFlutter (); await Hive . openBox ( 'settings' ); runApp ( const MyApp ()); } Here we open a box called settings . Step 4: Understanding Boxes A Box is where Hive stores data. Imagine this box: Settings Box theme -> dark username -> Alex loggedIn -> true Keys are on the left. Values are on the right. Step 5: Save Data Saving data is incredibly simple. var box = Hive . box ( 'settings' ); box . put ( 'username' , 'John' );

On May 20, 2026, AWS ended support for DMS Fleet Advisor. Fleet Advisor answered a question every migration team asks first: what is actually in my database estate, and how hard will it be to move? It was free. It was fully managed. It was backed by the largest cloud provider on earth. It still lost. AWS's official notice says only: "After careful consideration, we decided to end support for AWS DMS Fleet Advisor." No reason given. But you don't need one — the documentation tells you. Here is what Fleet Advisor required before it would tell you a single thing about your databases: Install a standalone data collector in your local environment Create an Amazon S3 bucket Create IAM policies, roles, and users — via CloudFormation, which was the recommended path Create database users with the minimum required permissions on every source Establish network access from the collector to each database server Then you'd meet the ceilings: recommendations for up to 100 databases at a time, one-to-one target mapping only, no multitenant server support. Now picture running that gauntlet inside a bank. You are a Business Solution Architect. You have been asked to scope a migration. You do not yet have approval for the migration — that approval is what the assessment is for . And to produce the assessment, you must first request production database credentials, get an agent binary through software approval, provision an S3 bucket, and get an IAM stack past a security review. That is a six-week procurement conversation to answer a question you were hoping to answer this week. AWS's replacement recommendation is Migration Evaluator — a consulting-led engagement. Read that as the finding it is: AWS looked at self-serve migration assessment, and concluded that humans and services do it better than a product. I think they were half right. And the half they got wrong is the interesting part. The lesson: friction is a competitor, and it usually wins We talk about developer tools as if the

How a Missing Database Index Turned an 8ms API into an 8-Second Nightmare Every developer...

Before we start This is a series about the intermediate results of an ongoing effort, not a finished story. It isn't an academic paper — it's a record of real engineering work and the insights that emerged along the way. Also, it's not about AI generating code. The AI angle here is about investigation and research — a careful, governed use of AI as a tool, not an autopilot — something I'll come back to in the final part. A word on why now, with the project still unfinished: details fade — the small technical decisions, the intermediate observations, the context in which a given call was made. Writing this down while the work is still ongoing is partly how I keep that context from slipping away. And that context matters: it's a reminder that every past decision, mine or anyone else's, was made for reasons that made sense at the time. One more note: none of this happened instead of product work. All of it ran alongside building new features and fixing bugs — the roadmap never paused for it. On confidentiality: I don't name the Customer, and I avoid any personal data or details a competitor could use. For the same reason, I don't mention anyone by name and refer to colleagues only by role. I won't name them, but I want to acknowledge up front that much of what follows was only possible thanks to the people I work with. The numbers are approximate and rounded — the point is the order of magnitude and the reasoning, not the exact figure. And a framing to carry through the series: at this scale, optimization is less a sprint than a marathon — yes, probably the most overused metaphor around, but here it genuinely fits: steady pacing beats sprinting, and you get there one careful step at a time. How I ended up here I'm a software engineer, and I've spent most of my career close to backends and databases. I've also led teams as a technical team lead — though over time I've deliberately shifted back toward more hands-on technical roles, which is where I'm most effective and m

Published on Aug 18, 2025 A New Era of AI-Powered Coding Begins I have installed Cursor on my laptop this weekend, and I am amazed at how much it speeds up my coding. I have a new debugging buddy!! This week, I have made several updates to the Portfolio website. The Challenge: When OpenAI Falls Short In my previous post, I shared the excitement of implementing a chatbot based on ChatGPT for my portfolio website. The initial experience was promising - I successfully created content embeddings and integrated them with OpenAI's API. However, as many developers know, relying on a single service provider can lead to unexpected roadblocks. When my OpenAI account encountered issues, I faced a critical decision: abandon the chat functionality or find an alternative solution. I chose the latter, embarking on a journey that would transform my portfolio's AI capabilities and teach me valuable lessons about building robust, fallback-ready systems. The Migration: Embracing Open Source AI The transition from OpenAI to Hugging Face wasn't just a simple API swap - it was a complete architectural evolution. Here's what I learned: 1. Model Selection Complexity Finding the right model on Hugging Face proved more challenging than expected. After testing several options: microsoft/DialoGPT-medium - No inference provider available gpt2 and distilgpt2 - Limited conversational capabilities Qwen/Qwen3-4B - Perfect fit with the nebius provider 2. Database Architecture Evolution The migration also prompted a database upgrade from MongoDB to Neon PostgreSQL. This wasn't just about changing providers - it was about building a more scalable, production-ready foundation for my portfolio. Technical Implementation: Building Resilience Streaming Responses for Better UX One of the most significant improvements was implementing streaming text responses. Instead of waiting for complete AI responses, users now see text appear word-by-word, creating a ChatGPT-like experience: // Streaming implementation

INTODUCTION Creating Greenwood academy database is essential for managing the students, subject and exam results efficiently. PostgreSQL, a powerful open-source relational database system, offers the perfect foundation for such a project. The main areas areas in SQL covered in this projects are : 1. DDL (Data Definition Language) DDL commands define, modify, and change the physical structure of database objects like tables and schemas. The first step is to create a greenwood academy schema using the create command. create schema greenwood_academy ; set search_path to greenwood_academy ; Next is to crete tables in the schema; The schema has 3 tables students,subject and exam results. create table greenwood_academy . students ( student_id INT PRIMARY key , first_name VARCHAR ( 50 ) NOT null , last_name VARCHAR ( 50 ) NOT null , gender VARCHAR ( 1 ), date_of_birth DATE , class VARCHAR ( 10 ), city VARCHAR ( 50 ) ); create table greenwood_academy . subject ( subject_id INT PRIMARY key , subject_name VARCHAR ( 100 ) NOT null unique , department VARCHAR ( 50 ), teacher_name VARCHAR ( 100 ), credits INT ); create table greenwood_academy . exam_results ( result_id INT PRIMARY key , student_id INT NOT null , subject_id INT NOT null , marks INT NOT null , exam_date DATE , grade VARCHAR ( 2 ) ); ALTER - This command changes the structure of tables in a database. Core Actions You Can Perform Add columns : Insert a new column and its data type into a table. The school realised that the nthey forgot to add phone numbers in the students table. The following command is used to add the data alter table greenwood_academy . students add column phone_number VARCHAR ( 20 ); Rename colums : Change the name of a table or a column. The column credit has to be changed to credit hours alter table greenwood_academy . subject rename column credits to credit_hours ; Drop columns : Delete an unwanted column from a table. Later the school relised that the phone number column is nolonger needed. a

SQL is quite flexible, making it easy to write a single query that works for two situations: one without a parameter and a WHERE clause, and another with a parameter for filtering, all in the same SQL query. For example, I came across a benchmark comparing MongoDB and PostgreSQL that shows how to handle pagination effectively—by avoiding OFFSET and instead using the last value to fetch the next set of results. The first page includes a WHERE clause along with ORDER BY and LIMIT, while the following pages add an extra WHERE condition. In the MongoDB version of this benchmark, the filter is handled within the application, which leads to two separate queries for these scenarios. export async function getOrders ( cursor ) { const match = cursor ? { timestamp : { $gt : new Date ( cursor ) } } : {}; const rows = await orders . aggregate ([ { $match : match }, { $sort : { timestamp : 1 } }, { $limit : PAGE_SIZE }, ]) We can do the same in PostgreSQL using a single prepared statement. SQL is such a powerful language that it often feels tempting to write it this way: SELECT * FROM orders WHERE $ 1 :: timestamptz IS NULL OR "timestamp" > $ 1 ORDER BY "timestamp" ASC LIMIT $ { PAGE_SIZE } If $1 is NULL, it skips the second condition in the OR clause and retrieves all rows without filters, resulting in a broad fetch. When $1 has a value, it filters the results using that specific value, enabling a more targeted search. However, using a generic query can sometimes lead to a less-than-ideal execution plan that's not perfectly tailored for each specific situation. I gave it a try: drop table if exists orders ; create table orders ( order_id text primary key , "timestamp" timestamptz not null ); create index idx_orders_timestamp on orders ( "timestamp" ); insert into orders select 'ORD-' || g , '2025-01-01' :: timestamptz + g * interval '1 minute' from generate_series ( 1 , 5000000 ) as g ; analyze orders ; prepare getorders ( timestamptz , int ) as select * from orders where $ 1 :

The moment that started it A colleague of mine was cleaning up some old records in a staging environment. Same query he'd run a dozen times before, except this time he was connected to production. He hit F5. No WHERE clause. 47,000 rows gone in milliseconds . We recovered from a backup, lost about two hours, and nobody got fired. But it stuck with me:** SSMS will let you delete an entire production table with the same amount of friction as running a SELECT 1**. No pause, no confirmation, nothing. The tool that DBAs and backend developers spend all day in has zero built-in protection against the single most common way people destroy data. So I built one. What SQL Guard does SQL Guard is a free SSMS extension (18 through 22) that inspects your query the moment you press F5, and pauses execution if it matches a known destructive pattern: DELETE without WHERE UPDATE without WHERE TRUNCATE TABLE DROP TABLE / DROP DATABASE / DROP PROCEDURE ALTER DATABASE Dangerous EXEC calls MERGE without a filter When one of these fires, you get a dialog showing exactly what object would be affected, with three options: cancel, run anyway, or run and ignore for the rest of the session. The point isn't to block you — it's to make the action conscious. Most accidental damage happens because muscle memory took over, not because someone genuinely meant to wipe a table. How the detection works Nothing exotic here, and honestly that's by design. SQL Guard runs a lightweight pattern-matching pass over the query text before it hits the connection. No parsing tree, no round-trip to the server, no measurable latency — it runs in under a millisecond, so you never notice it on normal queries. The core idea is simple: certain statements are only safe when scoped by a WHERE clause, and the extension checks for that clause's absence rather than trying to understand the full semantics of the query. That keeps false positives low and means it works the same way whether you're on SQL Server 2016 or the la

Most backend engineers implement multi-tenant quota checks using a standard "read-then-write" pattern. In production, this pattern is highly unsafe: SELECT grading_scans_remaining FROM profiles; If greater than 0, execute the application logic. UPDATE profiles SET grading_scans_remaining = grading_scans_remaining - 1; Under high volume or rapid concurrent requests, two independent processes will read the exact same balance before either one deducts usage. This race condition allows multi-tenant users to bypass your billing gates entirely. To solve this, you have to bypass the frontend and application-level checks, enforcing an atomic database operation that serializes the row update first. I have open-sourced a reference framework that outlines explicit subscription enums, core multi-tenant schemas, and a native VS Code / Cursor snippets configuration to speed up your local database modeling. 📂 Check out the repository on GitHub: { https://github.com/dollykm49/PostgreSQL-SaaS-Multi-Tenant-Subscription-Architecture-reference-framework- } What's inside the repository: Strictly Typed Enums: Centralized business rules handled natively by the database engine. Granular Balance Tracking: Optimized data-layer mapping for profiles and reset states. postgres-saas.code-snippets Engine: A local IDE configuration file that lets you deploy this core schema straight from your code editor by typing pg- shortcuts. For teams building commercial applications looking to skip weeks of writing custom migrations, testing concurrency edge-cases, and debugging row-locking security rules, the repository also includes a link to the extended 28-page production system bundle. Feedback on the multi-tier validation parameters is highly welcome!

I run a small self-hosted observability tool on the cheapest VPS I could find on purpose: 2 cores, 2 GB RAM, 20 GB SATA SSD . It ingests errors, traces and metrics from two low-traffic sites of mine. The stack is three containers — a Go app, PostgreSQL, and ClickHouse. One evening docker stats showed ClickHouse sitting on 880 MB of its 1 GB limit and the box swapping, with basically zero events coming in. So I went looking for where the memory and disk had gone. The answer turned out to be a good lesson in how a database can spend almost all of its I/O talking to itself. 543 KB of my data, 579 MB of ClickHouse talking about ClickHouse First thing I checked: how much data had my app actually stored versus how much ClickHouse had stored about itself . My application database: 543 KB, 16k rows The system database: 579 MB, 46.3M rows Roughly a thousand to one. Disk was 12 GB used out of 20 — on a tool that had recorded half a megabyte of real telemetry. The culprit was ClickHouse's own system logs, several of which have no TTL by default and therefore grow forever: trace_log — 404 MB, 26M rows (the query profiler writes here; it's on by default, sampling once per second) asynchronous_metric_log — 16.6M rows text_log — 132 MB plus query_log , latency_log Only metric_log , processors_profile_log and part_log ship with a TTL. Everything else just accumulates. Then I looked at the insert rate over 30 seconds: trace_log — 227 rows/s asynchronous_metric_log — 157 rows/s text_log — 44 rows/s my application — about 5 rows/s 98.8% of all inserts were ClickHouse narrating its own internals. The part that's expensive beyond disk Here's the number that made me stop. Over the same 30 seconds: rows inserted : 16,222 rows merged : 11,007,643 That's a 1 : 678 ratio. For every row written, the engine rewrote 678 already-sitting rows. The mechanics: MergeTree drops every insert into its own data part, then merges parts into bigger ones so reads stay fast. When the table is small this is

A normal secondary index can answer: status = "active" It cannot efficiently answer: documents containing "distributed database" LioranDB therefore has a dedicated text-segment architecture. Tokenization Text is split on non-alphanumeric characters. Depending on index options, tokens can be normalized to lowercase and filtered through stopwords. "Building Distributed Databases" becomes ["building", "distributed", "databases"] Segment contents A LioranDB text segment can contain several files and structures: Term dictionary Posting lists Document map Document-length norms Optional term positions Bloom filter Segment metadata A posting connects a term to the local documents containing it. "database" → [doc 2, doc 8, doc 19] Positions can record where the term appears inside each document. That enables more advanced query behaviour and phrase-aware features. Global and local document IDs Each segment assigns compact local IDs to its documents. A separate document map translates them back to global document IDs. This keeps postings smaller while preserving the external identity of the record. Bloom filters Each segment also maintains a Bloom filter for terms. Before reading a segment's postings, the query path can test whether the term might exist there. A negative answer is definitive. A positive answer means the segment may contain the term and should be checked. Query modes The text query layer supports modes such as: AND OR It also emits scored documents and metrics including: Query time Postings read Candidate documents Segments searched Full-text search is essentially a specialized database living beside the document database. Its data structures, compaction behaviour, scoring, and caching needs are different enough that treating it as a plain secondary index would be a mistake. Built by Swaraj Puppalwar under Lioran Group . Learn more: LioranDB Lioran Developer Solutions Lioran Group

Disk structures are durable, but updating them for every write is expensive. LioranDB uses memtables to absorb writes before flushing them to the on-disk B+ tree. What is a memtable? A memtable is an ordered in-memory map. In LioranDB, each entry contains either: Value ( bytes ) or: Tombstone A tombstone represents a deletion. The memtable also tracks: Approximate memory usage Minimum LSN Maximum LSN Entry count Put count Delete count The write path A simplified write path looks like this: Application write ↓ WAL durability ↓ Mutable memtable ↓ Immutable memtable queue ↓ Background flush ↓ Disk B+ tree The active mutable memtable accepts new writes. When it crosses a size limit, the engine rotates it into an immutable memtable. That immutable table is no longer modified and can safely be flushed in the background. Why ordered maps? LioranDB uses an ordered map for memtable entries. This helps because the flush process can emit keys in sorted order, which is friendly to the B+ tree and bulk-write paths. It also simplifies range merging between: Mutable data Immutable data On-disk pages Backpressure Background flushing cannot be allowed to fall behind forever. LioranDB therefore tracks limits such as: Maximum immutable memtables Maximum immutable bytes Partition-wide queue limits Maximum writer stall duration If the disk cannot drain the backlog quickly enough, the foreground write path slows down. That may sound undesirable, but controlled backpressure is much safer than consuming memory until the process dies. A memtable is not merely a cache. It is a pressure valve between CPU-speed writes and disk-speed persistence. Without that valve, the engine would either become slow on every commit or dangerously accumulate unbounded work. Built by Swaraj Puppalwar under Lioran Group . Links: LioranDB Lioran Developer Solutions Lioran Group

Most developers think of LioranDB as a document database. Internally, however, its storage engine does not understand documents, objects, fields, or JSON. It understands only: table + key bytes + value bytes That separation is intentional. The architecture LioranDB is split into two major layers: Application ↓ Document DBMS ↓ Transactional key-value engine ↓ WAL, memtables, B+ tree, pager and disk The engine exposes operations such as: get ( table , key ) put ( table , key , value ) delete ( table , key ) scan ( table , range ) The DBMS layer then adds document-oriented features: Collections JSON encoding Queries Updates Secondary indexes Text indexes Transactions For example, a secondary index can be represented as: idx:status:active → document_id A text index can be represented as: inv:database → posting_list The storage engine does not need to know what status , active , or database means. It only stores ordered bytes. Why this matters This architecture keeps the core engine small and reusable. The engine focuses on difficult low-level concerns: Durability Page management Transactions Recovery Ordering Concurrency Range scans The DBMS focuses on application-level semantics. This also makes it possible to build different data models over the same engine in the future. A document database is therefore not one giant component. It is a collection of carefully separated layers. That separation is one of the most important architectural decisions inside LioranDB. LioranDB is being developed by Swaraj Puppalwar under Lioran Group . Learn more: LioranDB Lioran Developer Solutions Lioran Group