标签：#base

PostgreSQL HA Risks, Replication Internals, & Rapid Branching Today's Highlights Today's highlights include critical insights into Patroni's replication slot management, an architectural deep dive into PostgreSQL's synchronous commit behavior, and a look at achieving sub-second database branching for enhanced developer workflows. When Patroni Silently Deletes Your Replication Slots (Planet PostgreSQL) Source: https://postgr.es/p/9lM This article uncovers a critical operational pitfall when using Patroni, a popular high-availability solution for PostgreSQL, with logical replication. It details how Patroni, under specific failure scenarios or configuration changes, can silently remove replication slots without warning. Replication slots are vital for ensuring that standbys or logical replication consumers do not miss any changes, making their deletion a potentially severe data integrity issue. The author explains the underlying reasons for this behavior, often related to how Patroni manages pg_basebackup or restores, and how it might not re-create logical replication slots automatically. The post provides concrete scenarios where this can occur, such as when a new primary is elected and old slots aren't re-established, or during certain recovery operations. It emphasizes the importance of diligent monitoring of replication slot status and proposes strategies to mitigate the risk of silent deletion, including careful Patroni configuration and robust alerting mechanisms. This insight is crucial for database administrators and developers relying on Patroni for resilient PostgreSQL deployments, highlighting a subtle but dangerous interaction between these two powerful components. Comment: This is a must-read for anyone running Patroni with PostgreSQL, especially if using logical replication. Understanding this specific behavior of Patroni deleting replication slots silently is essential to prevent unexpected data loss or integrity issues in production. Why Postgres Doesn'

I had a write that spanned two physically separate databases. A rename that had to propagate across several tables in one database and a couple of tables in another, and the two had to stay consistent. No distributed transaction coordinator was available to me. So I did the obvious thing: opened a transaction on each, did the work, and committed them one after the other inside a try/catch with rollbacks on both sides. It felt safe. It compiled. It passed tests. Then I drew the failure on a whiteboard, and the safety evaporated. The window that ruins everything Here's the structure, simplified: await using var txA = await dbA . Database . BeginTransactionAsync (); await using var txB = await dbB . Database . BeginTransactionAsync (); await DoWorkOnA ( dbA ); await DoWorkOnB ( dbB ); await txA . CommitAsync (); // <-- succeeds await txB . CommitAsync (); // <-- what if this throws? Two transactions do not make one atomic operation. CommitAsync is a point of no return, and there are two of them. Between the first commit returning and the second one starting, there is a window. If txB fails in that window — the connection drops, the process is killed, the database hiccups — then A is permanently committed and B never happens. Your rollback in the catch block is useless: you can't roll back txA , it's already durable. The two databases now disagree, and nothing in your code will heal that on its own. This is the dual-write problem , and it's not a bug you can fix by being more careful with try/catch. The atomicity you want simply isn't available from two independent commits. Ordering them, nesting them, wrapping them — none of it closes the window, because the window is inherent to having two commit points. Why "it's never failed" isn't reassurance The seductive thing about this pattern is that the window is small, so in practice it almost never triggers. You can run it for a year and never see an inconsistency. That's exactly what makes it dangerous: it trains you to tr

Sometimes you genuinely need the set of columns to update to be data, not code. An operator maps configuration fields to database columns, and you want to honor that mapping without redeploying every time it changes. The naive solution — build an UPDATE string from those column names — is also one of the easiest ways to hand-write a SQL injection vulnerability. This is how to get the flexibility without the hole. We'll build it up in three layers: make it work, make it safe, then count the cost. Layer 1: The dynamic update, the wrong way The tempting version concatenates column names into SQL: // DO NOT do this. var sql = $"UPDATE products SET { columnName } = { value } WHERE id = { id } " ; If columnName comes from configuration that an operator can edit, you've just made your schema writable by whoever controls that config. A value of name = 'x'; DROP TABLE products; -- is now your problem. Even "trusted" config is an injection surface the moment it flows into a SQL string. Layer 2: The same feature with EF.Property EF Core's ExecuteUpdateAsync lets you set a property by name without ever building SQL yourself. EF.Property<T> takes the property name as a string, and EF parameterizes the value and validates the property against the model: await db . Products . Where ( p => p . Id == id ) . ExecuteUpdateAsync ( setters => setters . SetProperty ( p => EF . Property < float ?>( p , columnName ), value )); This is already a different security posture: the value is a parameter, not interpolated text, and EF will throw rather than emit SQL if columnName isn't a real mapped property. But "EF will throw" is a runtime backstop, not a policy. We want to reject bad names before they reach the database, fail closed, and control exactly which columns are writable. Layer 3: Reflection as a whitelist The guard is to validate every incoming column name against the entity's actual properties, using reflection, and to keep an explicit blacklist of fields that must never be touched d

Paginated queries with automatic total count calculation. Supports specifying result fields. public < E > LimitResult < List < E >, E > query ( Class < E > cls , LimitSelect limitSelect ) public < E > LimitResult < List < E >, E > query ( Class < E > cls , LimitSelect limitSelect , LimitRange range ) public < E > LimitResult < List < E >, E > query ( Class < E > cls , LimitSelect limitSelect , List resultFields ) public < E > LimitResult < List < E >, E > query ( Class < E > cls , LimitSelect limitSelect , LimitRange range , List resultFields ) Returns: LimitResult — contains getResult() (data list) and getTotal() (total count). Example: // Define pagination query LimitSelect limitSelect = new LimitSelect () { public SelectOrderByStep from ( SelectSelectStep select ) { return select . from ( T ( "user_table" )) . where ( jt . conditions ( "name%" , name , "birthday>=" , beginDate )); } public List < OrderField > orderBy () { return Arrays . asList ( F ( "birthday" ). desc ()); } }; // Mode 1: return all data, no total count LimitResult res1 = jt . query ( User . class , limitSelect ); // Mode 2: return limit rows, no total count LimitResult res2 = jt . query ( User . class , limitSelect , LimitRange . of ( 20 )); // Mode 3: paginate (offset starts at 0), calculate total count LimitResult res3 = jt . query ( User . class , limitSelect , LimitRange . of ( 20 , 0 )); // res3.getResult() returns data, res3.getTotal() returns total count // Mode 4: specify result fields LimitResult res4 = jt . query ( User . class , limitSelect , LimitRange . of ( 20 , 0 ), Arrays . asList ( "id" , "name" )); // LimitRange.all(): return all data, no total count LimitResult res5 = jt . query ( User . class , limitSelect , LimitRange . all ()); // Access results List < User > data = res3 . getResult (); int total = res3 . getTotal (); About the LimitSelect interface: // LimitSelect is a interface: public interface LimitSelect { // Build the FROM clause; the select parameter allows specifyi

Cómo un backend de FastAPI + asyncpg comparte un solo Postgres chiquito con sus propios trabajadores en segundo plano y un segundo servicio, por qué el techo de conexiones, no el CPU, es lo que de verdad le pone tope a nuestro autoescalado, la caída en el cambio de hora que nos enseñó la cuenta, y una mirada honesta a RDS Proxy como la válvula de escape (incluyendo la trampa de asyncpg que lo puede dejar sin hacer nada). TL;DR Ajuste Valor Por qué Driver postgresql+asyncpg asíncrono hasta el fondo pool_size / max_overflow 8 / 12 (20 por proceso) subido desde 3/5 después de una caída pool_pre_ping True mata los sockets muertos tras un reinicio de RDS / inactividad pool_recycle 1800 s techo duro que el pre-ping no puede cubrir Tope de conexiones de RDS ~87 (t3.micro, menos las reservadas) la verdadera restricción de todo Pruebas NullPool sin pooling entre event loops en pytest La lección que replanteó todo el problema: en un RDS chico, tu cuenta máxima de tareas la fija max_connections , no el CPU ni la memoria. Un autoescalado que ignora el presupuesto de conexiones va a escalar directito hacia QueuePool limit reached , o peor, FATAL: too many connections del mismo Postgres. El pool, y la cuenta escondida adentro de él engine = create_async_engine ( settings . DATABASE_URL , connect_args = { " ssl " : " prefer " }, # negocia TLS en RDS, texto plano en local pool_pre_ping = True , pool_size = 8 , max_overflow = 12 , # 8 + 12 = 20 conexiones por proceso pool_recycle = 1800 , ) Veinte conexiones por proceso se ven modestas hasta que las multiplicas por cada capa entre ellas y la base de datos: pool_size + max_overflow = 20 por proceso de Python × 2 trabajadores de uvicorn = 40 por tarea de Fargate × 2 durante un despliegue rolling = 80 (tarea vieja drenando + tarea nueva arrancando) + servicio de inteligencia (3 + 7) = 10 sobre la misma base de datos + alembic / ad-hoc / psql ≈ unas pocas ---------------------------------------- ≈ 87 ← el techo de la t3.micro, con ~0 de

The thesis Quorum is built on is uncomfortable and true: the tools a team uses to coordinate an incident often live in the same region as the thing that is failing. When the region goes, the incident response goes with it. You are now coordinating a region outage over a status page that the region outage took down. Quorum is an incident command plane designed to survive a region loss. This post is about how the failover works, what the live demo does and does not prove, and where the survival story currently ends, because a database audience will ask all three and they deserve a straight answer. What DSQL gives you A multi-region DSQL cluster in the US set is three regions: two full regions, which for Quorum are us-east-1 and us-east-2, and a log-only witness in us-west-2 that has no cluster endpoint of its own. Both full-region endpoints present a single logical database with strong consistency, and the architecture is designed for 99.999% multi-region availability with no single point of failure and automated failure recovery . The behavior that matters for an incident tool is stated plainly in the GA announcement : applications can keep reading and writing with strong consistency even when they are unable to connect to a region's cluster endpoint, and the third region acts as a log-only witness with no cluster resource or endpoint. The survivor keeps serving; the witness holds the log so the surviving region keeps commit quorum. Quorum is, in effect, a live demonstration of that reference behavior with an incident-command product wrapped around it. Quorum's failover layer AWS's guidance for multi-region DSQL is to put routing in front of the endpoints: either DNS-based routing with Route 53, or application-level routing logic, so traffic redirects automatically when an endpoint becomes unreachable. This is laid out in Implement multi-Region endpoint routing for Amazon Aurora DSQL . Quorum, a Next.js app on Vercel, does the application-level version: it detects an

Quorum is an incident command plane built on Amazon Aurora DSQL. The failover story lives in another post. This one is about a narrower question that turned out to be the foundation: when several responders write to the same incident at the same moment, across regions, during the worst minutes of an outage, how do you guarantee the record never forks into two conflicting truths. The answer is two design choices that are really one choice seen from two angles: event sourcing, and DSQL's optimistic concurrency control. The data model is append-only Quorum is event-sourced across four tables. Every state change is an immutable event appended to a log, not an in-place update. The current state of an incident is a fold over its events. There is no UPDATE incidents SET status = ... ; there is an acknowledged event, a note event, a resolved event, and the status you render is computed from them. The event's UUID is its primary key and its idempotency key at the same time. A retried write carrying the same UUID cannot double-apply: the insert collides on the primary key and becomes a no-op. That property sounds minor until you remember what kind of system this is. A tool designed to survive network failure retries writes constantly, and "the responder tapped resolve twice because the first response was slow" must not produce two resolutions. Append-only also suits the domain directly. For an incident system the audit trail is the product, not a side effect. "Who acknowledged this, at what time, and what did the timeline look like at 02:14" is a first-class question for the post-incident review and a compliance requirement in regulated environments. Event sourcing gives you that for free. It also gives DSQL a write pattern it likes, which matters more than you would expect. The stack, briefly TypeScript end to end. Kysely as a typed query builder rather than an ORM, because I wanted type safety without surrendering control of the SQL: on a distributed database the exact shap

MVCC, WAL, vacuum, and replication slots under sustained delete load - and how to delete billions of rows without your database noticing Most "how to delete a lot of rows" articles stop at "batch it and delete children before parents." That advice is correct, it's table stakes, and everyone already knows it. This article is about everything after that - the parts that actually decide whether your cleanup runs quietly in the background for a week or pages you at 3 a.m. with a full disk and a replica that's six hours behind. The thesis: at scale, your DELETE statement is the easy part. The adversaries are the subsystems a delete feeds - MVCC tuple versioning, the write-ahead log, autovacuum, and the replication machinery. Bulk deletion is really an exercise in flow control across those subsystems . Get the SQL right and the systems wrong, and you'll still take production down. We'll assume PostgreSQL (the internals are PG-specific), a live OLTP primary with at least one physical replica and one or more logical/CDC consumers, and a target of hundreds of millions to billions of rows across many related tables. The one paragraph of "basics," so we can move on: delete in dependency order (referencing rows before referenced rows); collect parent keys once; never rely on ON DELETE CASCADE for huge deletes because you can't throttle a cascade. Done. Now the real material. 1. What a DELETE actually costs A delete is not "remove a row." Under MVCC it's "mark a row version dead and write that fact everywhere." For each deleted tuple, PostgreSQL: Sets xmax on the heap tuple to your transaction id. The row is still physically present; it becomes a dead tuple once your transaction commits and no snapshot can still see it. Writes a WAL record for the heap change. If this is the first modification of that page since the last checkpoint, it also writes a full-page image (FPI) - potentially 8 KB of WAL for a single-row change. Touches every index. Index entries aren't removed at delet

Six months ago I moved my RAG pipeline from Pinecone to self-hosted Qdrant. My vector search bill went from $210/month to $6.50/month. Same latency. Same recall. Here's exactly how. The Setup My app does document Q&A for legal contracts. The numbers: 5.2 million vectors (1536-dim, OpenAI embeddings) ~800K queries/month P99 latency requirement: < 50ms On Pinecone Serverless, this cost me roughly $210/month — storage plus read units plus write units for daily ingestion of new documents. What I Moved To A single Hetzner CX32 server: 4 vCPU, 8 GB RAM, 80 GB SSD €8.50/month (about $9.20) Qdrant running in Docker Automated daily backups to S3-compatible storage ($0.50/month) Total: ~$10/month. That's a 95% cost reduction. The Migration Was Easier Than Expected bash# Export from Pinecone (I used their scroll API) python export_pinecone.py --index legal-docs --output vectors.jsonl Start Qdrant docker run -d -p 6333:6333 -v ./storage:/qdrant/storage qdrant/qdrant Import python import_qdrant.py --input vectors.jsonl --collection legal-docs The whole migration took an afternoon. The Qdrant Python client is straightforward, and the API is surprisingly similar to Pinecone's. Performance Comparison I ran the same 10,000 test queries against both setups: MetricPinecone ServerlessQdrant Self-HostedP50 latency23ms4msP99 latency89ms12msRecall@100.970.97Monthly cost$210$10 The self-hosted Qdrant is actually faster because the data sits in memory on the same machine. Pinecone Serverless loads data from object storage on demand, which adds cold-start latency. When Self-Hosting Is a Bad Idea I want to be honest about the trade-offs: Don't self-host if: You have zero DevOps experience and no one on the team does You need 99.99% uptime SLA for enterprise customers Your vector count is growing unpredictably (10M one month, 100M the next) You're a team of 1-2 and every hour on infra is an hour not building product Do self-host if: Your scale is predictable (you know roughly how many vectors

Every backend engineer has seen this happen, you build an application on a relational database like MySQL, handling thousands of concurrent transactions effortlessly. Then, the business asks for a real time analytics dashboard. But when you run an aggregation query over historical data, suddenly the database that effortlessly managed live traffic starts thrashing, evicting your working set, and dragging application performance down. This isn't a tuning problem, a missing index, or a badly written query. It’s a fundamental architectural collision. OLTP (Online Transaction Processing) OLTP encompasses nearly every concurrent digital interaction triggered across a distributed system. A user downloading a PDF, a microservice firing an automatic maintenance log, a comment on a social feed these are all transactions. Data engineers rely on OLTP systems (like MySQL or PostgreSQL) to capture these concurrent streams of interactions for creating , updating and deleting records. The Tree Based In-Place Engine To reliably capture massive volumes of transactions without corrupting data or locking up the application, OLTP systems rely on a highly optimized, row oriented architecture built around the B+ Tree. Because they must provide immediate, atomic updates to existing records, transactional databases manage state through a strict sequence of physical tree traversal and in-memory page mutation: The B+ Tree Indexing: When a transaction reads or updates id: 1, the engine traverses a B+ Tree from the root, through the branch nodes, directly to the specific physical leaf node holding that row. This O(\log n) traversal guarantees a fast, isolated point-lookup. It ensures the application always hits the single version of the row without scanning irrelevant data. The Buffer Pool & In-Place Updates: OLTP systems perform in place updates. The database pulls the exact page containing id: 1 from the physical disk into memory (the Buffer Pool). The specific row is mutated directly in RAM

"Chat with your PDF / your notes / your docs" is everywhere. Today we build it from scratch and you'll see it's just three moves : retrieve, then generate — with one prompt trick that stops the hallucinations. This is Day 46 of TechFromZero. Yesterday (Day 45) we built the retrieval half with pgvector. Today we add the answer half and host it on Supabase. RAG in one line Find the relevant chunks of your documents, paste them into the prompt, and tell the model to answer using only those. That's Retrieval-Augmented Generation. The "augmented" part is just stuffing real context into the prompt so the model isn't guessing from memory. 1. Storage: Supabase is Postgres, so pgvector is one click Supabase is hosted Postgres with an auto-generated API. Because it's just Postgres , vector search needs no separate database: create extension if not exists vector ; create table documents ( id bigserial primary key , content text , embedding vector ( 384 ) ); -- one RPC the app calls to get the closest chunks create function match_documents ( query_embedding vector ( 384 ), match_count int ) returns table ( id bigint , content text , similarity float ) language sql stable as $$ select id , content , 1 - ( embedding <=> query_embedding ) as similarity from documents order by embedding <=> query_embedding limit match_count ; $$ ; 2. Ingest: chunk → embed → store Split your docs into paragraph-sized chunks, embed each with a free local model (all-MiniLM-L6-v2 via Transformers.js — no key, nothing leaves your machine), and insert the row + vector: const embedding = await embed ( chunk ); // 384 numbers await supabase . from ( " documents " ). insert ({ content : chunk , embedding }); Chunk size matters: too big buries the answer in noise, too small loses meaning. A few hundred characters is a good start. 3. Retrieve + Generate (the payoff) Embed the question with the same model, ask Supabase for the closest chunks, then hand them to the LLM: const query_embedding = await embed ( que

Traditional databases just can't keep up with high concurrency and low latency at the same time. The term "real-time" has become kind of meaningless. Everyone claims it, from batch-oriented cloud data warehouses to transactional database extensions. This makes picking the right architecture really hard without expensive trial and error. The best real-time analytics database in 2026 depends entirely on your workload shape. Key takeaways Real-time analytics (in this guide) = sub-second p95/p99 analytical queries on billions of rows, high concurrency , and milliseconds-to-seconds freshness . Best overall in 2026 for most workloads: ClickHouse (ingest throughput, query speed at scale, compression/TCO). Best for strictly predefined query paths via star-tree indexes: Apache Pinot . Best for time-series operational dashboards and observability: ClickHouse . ClickStack is its full observability offering for logs, metrics, and traces. Best for rigid ingestion-time roll-up aggregations: Apache Druid . Best for unified OLTP + real-time analytics: ClickHouse paired with its managed Postgres offering and native sync to ClickHouse , giving you a purpose-built OLTP engine and a purpose-built OLAP engine without rolling your own CDC pipeline. SingleStore is an alternative if you prefer a single HTAP engine for both. Traditional Data Warehouses: Snowflake and BigQuery are fine for batch BI if you already have one, but face latency, concurrency, and cost challenges under sub-second, high-concurrency workloads. Evaluate using 4 axes: ingest/freshness, latency under concurrency, TCO, operational complexity. What 'real-time analytics' means (and why warehouses and OLTP databases fail) Strict engineering thresholds define true real-time OLAP : sub-second query latency on complex aggregations, the ability to serve tens to thousands of concurrent queries per second (QPS), and data freshness measured in milliseconds to seconds. Traditional cloud data warehouses like Snowflake and BigQuery a

If you're building a modern data stack that requires either high-throughput transaction processing or large-scale analytical workloads, you've likely come across both Vertica and VoltDB (now rebranded as Volt Active Data). While both are distributed relational database management systems (RDBMS), they are architected for completely opposite use cases — choosing the wrong one can lead to 10x higher costs, missed latency SLAs, and poor application performance. In this guide, we break down every key difference between OpenText Vertica and Volt Active Data, with practical examples, real-world use cases, and best practices to help you make the right choice for your team. Table of Contents What is OpenText Vertica? What is Volt Active Data (Formerly VoltDB)? Core Differences Between Vertica and VoltDB Real-World Use Cases: When to Pick Which Best Practices & Common Mistakes Conclusion & Key Takeaways References What is OpenText Vertica? OpenText Vertica (formerly Micro Focus Vertica) is a columnar relational DBMS built exclusively for analytical (OLAP) workloads, first launched in 2005. As of 2026, the latest stable version is 26.1, with native lakehouse and Apache Iceberg export support for modern data ecosystems. Core Vertica Architecture Vertica's design is optimized for fast queries across massive datasets: Columnar storage : Data is stored by column instead of row, enabling significantly higher compression ratios and faster aggregation queries that only access a small subset of columns Massively Parallel Processing (MPP) : Query execution and data are distributed across hundreds of nodes for parallel processing Dual deployment modes : Enterprise Mode : Shared-nothing architecture with data stored locally on nodes for maximum performance Eon Mode : Compute and storage separated, using shared object storage (S3, GCS, ADLS) to scale compute independently of storage for cloud workloads Projections : Physical, sorted copies of data optimized for common query patterns (ins

PostgreSQL Error 22P01: Floating Point Exception PostgreSQL error code 22P01 is raised when a floating-point operation produces an exceptional result that cannot be represented as a valid number. This typically occurs during division by zero on float types, operations involving NaN (Not a Number), or arithmetic that yields Infinity . It is most commonly encountered in analytics, financial calculations, and data pipelines processing external or sensor data. Top 3 Causes 1. Division by Zero on Float Types Unlike integer division (which raises 22012 ), dividing a float by zero triggers a floating-point exception. This is especially common in ratio and rate calculations where the denominator can become zero at runtime. -- Problematic query SELECT total_sales :: float / total_orders :: float AS avg_order_value FROM daily_stats ; -- Safe fix using NULLIF SELECT date , total_sales :: float / NULLIF ( total_orders , 0 ):: float AS avg_order_value FROM daily_stats ; 2. NaN Values in Arithmetic Operations Data ingested from external systems, CSVs, or APIs may silently introduce NaN values into float columns. Once NaN participates in arithmetic, results become unpredictable and can trigger exceptions downstream. -- Detect NaN values (NaN is the only value not equal to itself) SELECT id , value FROM sensor_readings WHERE value != value ; -- Replace NaN with NULL safely UPDATE sensor_readings SET value = NULL WHERE value != value ; -- Filter NaN in aggregations SELECT device_id , AVG ( value ) FILTER ( WHERE value = value ) AS clean_avg FROM sensor_readings GROUP BY device_id ; 3. Infinity Arithmetic Conflicts Storing 'Infinity'::float or '-Infinity'::float is valid in PostgreSQL, but performing certain operations on them produces mathematically undefined results (e.g., Infinity - Infinity = NaN ), which can cascade into a floating-point exception. -- Check for Infinity values SELECT id , measurement FROM raw_data WHERE measurement IN ( 'Infinity' :: float , '-Infinity' :: float

Dulu, nyari query yang bikin database spike itu bisa makan berhari-hari. Yang nyari capek, yang nge-fix juga capek. Sekarang? Hitungan jam — dan bonusnya, sambil belajar hal baru juga. Ceritanya begini. Kalau kamu pernah kerja di aplikasi yang pakai ORM (Object-Relational Mapping — semacam "penerjemah otomatis" antara code dan database), pasti familiar sama situasi ini: database tiba-tiba lambat, kamu dapet raw query yang jadi biang kerok, tapi di codebase kamu nulis pakai syntax ORM yang bentuknya beda jauh dari SQL mentah itu. Buat yang belum pernah deal sama ORM, bayangin gini: kamu nulis pesan dalam bahasa Indonesia, lalu ada "penerjemah otomatis" yang convert jadi bahasa Jepang sebelum dikirim ke penerima. Suatu hari ada masalah di pesan yang terkirim — tapi kamu cuma bisa lihat versi bahasa Jepang-nya. Nyari bagian mana dari tulisan Indonesia kamu yang bikin terjemahan-nya bermasalah? Itu effort-nya yang bikin pengen balik tidur aja. Sekarang dengan bantuan Kiro, cukup kasih raw query + akses ke codebase, dia otomatis nyari bagian mana di code yang nge-generate query bermasalah itu. Yang dulu butuh berhari-hari, sekarang bisa selesai dalam hitungan jam — dan itu baru tahap investigasi, belum termasuk fixing-nya. Ceritanya Kenapa Bisa Pakai Kiro Akhir-akhir ini lagi aktif pakai Kiro di tempat kerja. Awal tahun lalu kantor dapat credits melalui program Kiro for Startup , jadi ya sekalian dimaksimalkan. Selain buat debug dan explore query di MS SQL Server, kadang pakai Kiro juga buat analisa log AWS CloudWatch — sambil kasih context aplikasi yang running biar analisa-nya lebih akurat dan gak generic. Di tulisan kali ini, saya mau sharing gimana pakai Kiro sebagai partner beberapa minggu terakhir buat improve query performance di aplikasi .NET Core. Kenapa "partner"? Karena Kiro-nya gak boleh langsung akses ke database — jadi wajib melalui perantara saya. Kita discuss, kolaborasi, dan nge-solve bareng. Bukan AI yang dikasih tombol terus disuruh jalan sendiri. Wakt

One thing that confused me when I first started learning ClickHouse was the word FINAL . Because eventually you'll come across both: SELECT * FROM events FINAL ; and: OPTIMIZE TABLE events FINAL ; At first glance, they sound like they should do roughly the same thing. After all, both contain the word FINAL . But they actually solve two completely different problems. One affects query results. The other affects how data is physically stored. Understanding this distinction can save a lot of confusion when working with MergeTree tables. Why This Confusion Happens Most people encounter FINAL while working with engines like: ReplacingMergeTree SummingMergeTree AggregatingMergeTree Sooner or later they notice something like: SELECT * FROM users ; returns duplicate versions of rows. Then they discover: SELECT * FROM users FINAL ; and suddenly the results look correct. Naturally, many people assume: FINAL merges the table. But that's not exactly what is happening. What SELECT FINAL Actually Does When you run: SELECT * FROM users FINAL ; ClickHouse applies merge logic during query execution. Think of it as: "Show me what the table would look like if all relevant merges had already happened." The important part: It only affects the query result. After the query finishes: parts remain unchanged storage remains unchanged nothing is rewritten on disk The merge logic happens temporarily while the query is running. Once the query completes, the table is exactly as it was before. What OPTIMIZE FINAL Actually Does Now let's look at: OPTIMIZE TABLE users FINAL ; This is a completely different operation. Instead of modifying query results, ClickHouse physically merges parts on disk. The operation: rewrites data merges eligible parts removes obsolete versions creates larger merged parts Unlike SELECT FINAL , the effects remain after the command completes. This is a storage operation, not a query operation. The Simplest Way to Remember It Whenever I think about these commands, I use a v

TL;DR: This is not a cryptographic construction. It is a pragmatic engineering compromise for applications where encrypted storage is required but approximate alphabetical ordering is still useful. I sort encrypted strings using an external index: the sum of weighted Unicode code points for the first N characters with exponential positional weights, followed by quantization. Monotonicity is preserved, but accuracy predictably degrades after the first few characters. Not a cryptographic scheme; some ordering information leaks by design. The problem Some time ago, while implementing a project, I ran into the problem of sorting encrypted data in a database. I’d like to share the solution. I won’t go into detail describing the entire application. I’ll just say that, according to the required architecture, almost all data in the database must be stored exclusively in encrypted form: usernames, file names, tags, comments, dates, and so on (with the exception of identifiers and some system fields). That is, the table structure should be open, but the contents should not be. The encryption is symmetric: the same key is used for both encryption and decryption. This means that without the encryption key, even with a full database dump an attacker should not obtain any original data. And this is where two problems immediately arose: searching by the data without fully decrypting it, and sorting encrypted data. The first problem, with some caveats, is solved fairly simply. To search encrypted data, it is enough to additionally store hashes of the original values you plan to search on. This allows exact-match lookups (for example, users by login or files by tag) without storing the original values in plaintext. Yes, this won’t allow pattern searches, but it’s quite acceptable for the project’s goals. But sorting encrypted data turned out to be significantly more difficult. The solution A quick search showed that the problem is far from new, but there are no standard approaches t

DuckDB Data Inlining, SQLite Fossildelta OOB, Postgres 19 Temporal Data Today's Highlights Today's highlights include DuckDB's innovative data inlining for stream processing in data lakes, offering significant performance gains by eliminating the small files problem. Additionally, a critical out-of-bounds read vulnerability in SQLite's fossildelta extension and a peek into PostgreSQL 19's focus on temporal data capabilities are discussed. Data Inlining in DuckLake: Unlocking Streaming for Data Lakes (DuckDB Blog) Source: https://duckdb.org/2026/04/02/data-inlining-in-ducklake.html The DuckDB team has unveiled DuckLake’s new data inlining feature, designed to revolutionize how streaming data is managed in data lakes by effectively tackling the notorious “small files problem.” This issue, common in scenarios with frequent small updates or continuous ingestion, often leads to performance bottlenecks due to the overhead of managing numerous tiny files. DuckLake's solution involves intelligently storing these small updates directly within the catalog, thereby eliminating the need for physical small files on disk. This architectural innovation significantly improves the practicality of continuous streaming into data lakes, enabling more efficient real-time analytics. By inlining data, DuckDB reduces I/O operations and metadata management complexity, leading to substantial performance gains. A benchmark highlighted in the announcement demonstrates an impressive 926x speed improvement for certain operations, showcasing the feature's potential to transform data lake architectures for workloads requiring high-throughput ingestion and immediate query access without the traditional performance penalties. Comment: This DuckDB feature is a game-changer for data lake architectures, offering a simple yet powerful way to handle streaming data without the performance overhead of countless small files. Post: Out-of-bounds read in deltaGetInt() when input contains no in-buffer terminat

GBase 8a fully supports OLAP window functions, making it a powerful gbase database for analytical workloads. This guide uses real sales scenarios to demonstrate ROW_NUMBER / RANK , moving aggregates, LAG / LEAD for period‑over‑period comparisons, ROLLUP subtotals, and how these functions execute in an MPP environment. Window Function Syntax function_name () OVER ( [ PARTITION BY column ] -- window group [ ORDER BY column ] -- ordering within window [ ROWS | RANGE BETWEEN ... AND ...] -- frame definition ) Unlike GROUP BY , window functions do not collapse rows. Every row remains in the result set while still being able to “see” other rows in the window. Ranking Functions ROW_NUMBER / RANK / DENSE_RANK SELECT dept_id , salesperson , sale_amount , ROW_NUMBER () OVER ( PARTITION BY dept_id ORDER BY sale_amount DESC ) AS row_num , RANK () OVER ( PARTITION BY dept_id ORDER BY sale_amount DESC ) AS rnk , DENSE_RANK () OVER ( PARTITION BY dept_id ORDER BY sale_amount DESC ) AS dense_rnk FROM sales WHERE sale_date >= '2024-01-01' ; Sample output: dept_id salesperson sale_amount row_num rnk dense_rnk 101 Zhang San 95000 1 1 1 101 Li Si 95000 2 1 1 101 Wang Wu 82000 3 3 2 101 Zhao Liu 71000 4 4 3 Top 3 Sales per Department WITH ranked AS ( SELECT dept_id , salesperson , sale_amount , ROW_NUMBER () OVER ( PARTITION BY dept_id ORDER BY sale_amount DESC ) AS rn FROM sales WHERE YEAR ( sale_date ) = 2024 ) SELECT dept_id , salesperson , sale_amount FROM ranked WHERE rn <= 3 ORDER BY dept_id , rn ; NTILE: Bucketing SELECT dept_id , salesperson , sale_amount , NTILE ( 4 ) OVER ( PARTITION BY dept_id ORDER BY sale_amount DESC ) AS quartile FROM sales WHERE YEAR ( sale_date ) = 2024 ; Cumulative and Moving Aggregates Year‑to‑Date SELECT dept_id , sale_month , monthly_amount , SUM ( monthly_amount ) OVER ( PARTITION BY dept_id ORDER BY sale_month ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW ) AS ytd_amount FROM ( SELECT dept_id , DATE_FORMAT ( sale_date , '%Y-%m' ) AS sale_month ,

I inherited a SQL Server database with 2.3 million rows. Queries took 45 seconds. Users were frustrated. Dashboards timed out. Here is exactly what I did. Step 1: Find the slowest queries I used SQL Server's query store to identify the top 10 worst performing queries. Step 2: Check the execution plan Missing index warnings everywhere. Also saw table scans on a 2 million row table. Step 3: Add targeted indexes Created two non-clustered indexes on the most filtered columns. No over-indexing. Just what the queries actually needed. Step 4: Rewrite the worst join One query was joining 6 tables with a cross apply that made no sense. Restructured to inner joins with proper filter ordering. The result 45 seconds down to 8 seconds. An 82% improvement. Real-time dashboards started working again. Key lesson Check what is actually slow before changing anything. Most people skip this and waste time optimizing the wrong thing. What is your fastest query optimization win?