标签：#base

Murat Demirbas discusses the shift toward disaggregated cloud database architectures driven by cloud economics. He explains how decoupling compute from storage enables elastic scaling, cost efficiency, and fault isolation. He shares how classical Paxos roles foreshadowed disaggregation, while analyzing network tradeoffs, shared-memory evolution, and self-assembling database designs. By Murat Demirbas

Introduction When building a Flutter application, you'll often need to store data on the user's device. For example: Saving user preferences Storing login information Caching API responses Creating offline applications Building note-taking or to-do apps While there are several local storage solutions available, Hive is one of the fastest and easiest local storage for Flutter developers. In this tutorial, you'll learn Hive from scratch by building a simple example. No prior database knowledge is required. What is Hive? Hive is a lightweight, NoSQL database written entirely in Dart. It stores data directly on the device, making it perfect for Flutter applications. Why use Hive? Extremely fast Works offline No native platform code required Simple API Easy to learn Great for small and medium-sized applications Think of Hive as a collection of boxes where each box stores your application's data. Hive ├── User Box ├── Settings Box ├── Notes Box └── Products Box Each Box is similar to a table in traditional databases. Step 1: Create a Flutter Project Create a new Flutter project. flutter create hive_demo Open the project. cd hive_demo Step 2: Install Hive Open pubspec.yaml and add the following packages. dependencies : flutter : sdk : flutter hive : ^2.2.3 hive_flutter : ^1.1.0 Then install them. flutter pub get Step 3: Initialize Hive Before using Hive, initialize it inside main() . import 'package:flutter/material.dart' ; import 'package:hive_flutter/hive_flutter.dart' ; void main () async { WidgetsFlutterBinding . ensureInitialized (); await Hive . initFlutter (); await Hive . openBox ( 'settings' ); runApp ( const MyApp ()); } Here we open a box called settings . Step 4: Understanding Boxes A Box is where Hive stores data. Imagine this box: Settings Box theme -> dark username -> Alex loggedIn -> true Keys are on the left. Values are on the right. Step 5: Save Data Saving data is incredibly simple. var box = Hive . box ( 'settings' ); box . put ( 'username' , 'John' );

On May 20, 2026, AWS ended support for DMS Fleet Advisor. Fleet Advisor answered a question every migration team asks first: what is actually in my database estate, and how hard will it be to move? It was free. It was fully managed. It was backed by the largest cloud provider on earth. It still lost. AWS's official notice says only: "After careful consideration, we decided to end support for AWS DMS Fleet Advisor." No reason given. But you don't need one — the documentation tells you. Here is what Fleet Advisor required before it would tell you a single thing about your databases: Install a standalone data collector in your local environment Create an Amazon S3 bucket Create IAM policies, roles, and users — via CloudFormation, which was the recommended path Create database users with the minimum required permissions on every source Establish network access from the collector to each database server Then you'd meet the ceilings: recommendations for up to 100 databases at a time, one-to-one target mapping only, no multitenant server support. Now picture running that gauntlet inside a bank. You are a Business Solution Architect. You have been asked to scope a migration. You do not yet have approval for the migration — that approval is what the assessment is for . And to produce the assessment, you must first request production database credentials, get an agent binary through software approval, provision an S3 bucket, and get an IAM stack past a security review. That is a six-week procurement conversation to answer a question you were hoping to answer this week. AWS's replacement recommendation is Migration Evaluator — a consulting-led engagement. Read that as the finding it is: AWS looked at self-serve migration assessment, and concluded that humans and services do it better than a product. I think they were half right. And the half they got wrong is the interesting part. The lesson: friction is a competitor, and it usually wins We talk about developer tools as if the

Sharing a holiday home with family or friends is great until the admin starts. Who’s in next weekend? Did someone already claim Easter? Who was meant to book the cleaner? Where’s the WiFi password / insurance cert / “how to winterize the outdoor taps” note? For most groups this lives in five group chats, a half-maintained Google Calendar, and a Drive folder nobody trusts. I kept running into that pattern — so I built Shared Holiday Homes : software for families, friends, and co-owners who already share a place and need less chaos, not another generic calendar. The problem isn’t “finding a free date” Generic calendars are fine at showing blocks of time. Shared holiday homes need more than that: Double-booking protection that isn’t “hope nobody overwrites the event” Rules for peak weeks, min/max stays, booking windows, and optional approval Fairness visibility — who actually used the place this year Named jobs with owners and due dates (cleaning, maintenance, “fix the pump”) A home for house knowledge — docs, arrival notes, appliance quirks, emergency info If your group is small and high-trust, Google Calendar can work. Once you’re coordinating multiple households, peak seasons, and maintenance, the “calendar + WhatsApp” stack starts creating the arguments it’s supposed to prevent. I wrote a longer comparison here if you want the practical breakdown: Shared Holiday Homes vs Google Calendar What I built (and what I didn’t) The product is intentionally narrow. Private co-owner groups don’t need a full property-management system or a fractional-ownership marketplace. They need an operating layer for one shared house. In scope: One shared booking calendar Booking rules / seasonal rotations Shared task list Document library House guides (the handbook people can actually find) Out of scope on purpose: Selling property shares Matching investors Full bookkeeping / STR channel management That boundary mattered. Every time I was tempted to add “just one more admin feature,” I a

How a Missing Database Index Turned an 8ms API into an 8-Second Nightmare Every developer...

Before we start This is a series about the intermediate results of an ongoing effort, not a finished story. It isn't an academic paper — it's a record of real engineering work and the insights that emerged along the way. Also, it's not about AI generating code. The AI angle here is about investigation and research — a careful, governed use of AI as a tool, not an autopilot — something I'll come back to in the final part. A word on why now, with the project still unfinished: details fade — the small technical decisions, the intermediate observations, the context in which a given call was made. Writing this down while the work is still ongoing is partly how I keep that context from slipping away. And that context matters: it's a reminder that every past decision, mine or anyone else's, was made for reasons that made sense at the time. One more note: none of this happened instead of product work. All of it ran alongside building new features and fixing bugs — the roadmap never paused for it. On confidentiality: I don't name the Customer, and I avoid any personal data or details a competitor could use. For the same reason, I don't mention anyone by name and refer to colleagues only by role. I won't name them, but I want to acknowledge up front that much of what follows was only possible thanks to the people I work with. The numbers are approximate and rounded — the point is the order of magnitude and the reasoning, not the exact figure. And a framing to carry through the series: at this scale, optimization is less a sprint than a marathon — yes, probably the most overused metaphor around, but here it genuinely fits: steady pacing beats sprinting, and you get there one careful step at a time. How I ended up here I'm a software engineer, and I've spent most of my career close to backends and databases. I've also led teams as a technical team lead — though over time I've deliberately shifted back toward more hands-on technical roles, which is where I'm most effective and m

Published on Aug 18, 2025 A New Era of AI-Powered Coding Begins I have installed Cursor on my laptop this weekend, and I am amazed at how much it speeds up my coding. I have a new debugging buddy!! This week, I have made several updates to the Portfolio website. The Challenge: When OpenAI Falls Short In my previous post, I shared the excitement of implementing a chatbot based on ChatGPT for my portfolio website. The initial experience was promising - I successfully created content embeddings and integrated them with OpenAI's API. However, as many developers know, relying on a single service provider can lead to unexpected roadblocks. When my OpenAI account encountered issues, I faced a critical decision: abandon the chat functionality or find an alternative solution. I chose the latter, embarking on a journey that would transform my portfolio's AI capabilities and teach me valuable lessons about building robust, fallback-ready systems. The Migration: Embracing Open Source AI The transition from OpenAI to Hugging Face wasn't just a simple API swap - it was a complete architectural evolution. Here's what I learned: 1. Model Selection Complexity Finding the right model on Hugging Face proved more challenging than expected. After testing several options: microsoft/DialoGPT-medium - No inference provider available gpt2 and distilgpt2 - Limited conversational capabilities Qwen/Qwen3-4B - Perfect fit with the nebius provider 2. Database Architecture Evolution The migration also prompted a database upgrade from MongoDB to Neon PostgreSQL. This wasn't just about changing providers - it was about building a more scalable, production-ready foundation for my portfolio. Technical Implementation: Building Resilience Streaming Responses for Better UX One of the most significant improvements was implementing streaming text responses. Instead of waiting for complete AI responses, users now see text appear word-by-word, creating a ChatGPT-like experience: // Streaming implementation

INTODUCTION Creating Greenwood academy database is essential for managing the students, subject and exam results efficiently. PostgreSQL, a powerful open-source relational database system, offers the perfect foundation for such a project. The main areas areas in SQL covered in this projects are : 1. DDL (Data Definition Language) DDL commands define, modify, and change the physical structure of database objects like tables and schemas. The first step is to create a greenwood academy schema using the create command. create schema greenwood_academy ; set search_path to greenwood_academy ; Next is to crete tables in the schema; The schema has 3 tables students,subject and exam results. create table greenwood_academy . students ( student_id INT PRIMARY key , first_name VARCHAR ( 50 ) NOT null , last_name VARCHAR ( 50 ) NOT null , gender VARCHAR ( 1 ), date_of_birth DATE , class VARCHAR ( 10 ), city VARCHAR ( 50 ) ); create table greenwood_academy . subject ( subject_id INT PRIMARY key , subject_name VARCHAR ( 100 ) NOT null unique , department VARCHAR ( 50 ), teacher_name VARCHAR ( 100 ), credits INT ); create table greenwood_academy . exam_results ( result_id INT PRIMARY key , student_id INT NOT null , subject_id INT NOT null , marks INT NOT null , exam_date DATE , grade VARCHAR ( 2 ) ); ALTER - This command changes the structure of tables in a database. Core Actions You Can Perform Add columns : Insert a new column and its data type into a table. The school realised that the nthey forgot to add phone numbers in the students table. The following command is used to add the data alter table greenwood_academy . students add column phone_number VARCHAR ( 20 ); Rename colums : Change the name of a table or a column. The column credit has to be changed to credit hours alter table greenwood_academy . subject rename column credits to credit_hours ; Drop columns : Delete an unwanted column from a table. Later the school relised that the phone number column is nolonger needed. a

SQL is quite flexible, making it easy to write a single query that works for two situations: one without a parameter and a WHERE clause, and another with a parameter for filtering, all in the same SQL query. For example, I came across a benchmark comparing MongoDB and PostgreSQL that shows how to handle pagination effectively—by avoiding OFFSET and instead using the last value to fetch the next set of results. The first page includes a WHERE clause along with ORDER BY and LIMIT, while the following pages add an extra WHERE condition. In the MongoDB version of this benchmark, the filter is handled within the application, which leads to two separate queries for these scenarios. export async function getOrders ( cursor ) { const match = cursor ? { timestamp : { $gt : new Date ( cursor ) } } : {}; const rows = await orders . aggregate ([ { $match : match }, { $sort : { timestamp : 1 } }, { $limit : PAGE_SIZE }, ]) We can do the same in PostgreSQL using a single prepared statement. SQL is such a powerful language that it often feels tempting to write it this way: SELECT * FROM orders WHERE $ 1 :: timestamptz IS NULL OR "timestamp" > $ 1 ORDER BY "timestamp" ASC LIMIT $ { PAGE_SIZE } If $1 is NULL, it skips the second condition in the OR clause and retrieves all rows without filters, resulting in a broad fetch. When $1 has a value, it filters the results using that specific value, enabling a more targeted search. However, using a generic query can sometimes lead to a less-than-ideal execution plan that's not perfectly tailored for each specific situation. I gave it a try: drop table if exists orders ; create table orders ( order_id text primary key , "timestamp" timestamptz not null ); create index idx_orders_timestamp on orders ( "timestamp" ); insert into orders select 'ORD-' || g , '2025-01-01' :: timestamptz + g * interval '1 minute' from generate_series ( 1 , 5000000 ) as g ; analyze orders ; prepare getorders ( timestamptz , int ) as select * from orders where $ 1 :

Over the past few weeks, I've been focused on deepening my understanding of mobile analytics—not just by completing a course, but by putting those concepts into practice through hands-on implementation in React Native. Throughout this journey, I explored a wide range of topics, including: Firebase Analytics integration Google Analytics 4 (GA4) Event planning and naming conventions Screen view tracking Custom events and custom definitions Key Events (Conversions) User properties and User ID Acquisition and campaign tracking Audience segmentation Ecommerce measurement Checkout funnel analysis Promotions and marketing attribution BigQuery integration Realtime reporting and DebugView Analytics validation and best practices One of the biggest lessons I learned is that analytics is much more than logging events . A well-designed analytics strategy helps answer important questions about user behavior, feature adoption, user engagement, and conversion optimization. The quality of the insights you gain depends on having a well-planned event architecture, consistent naming conventions, and meaningful data collection from the very beginning. Completing this Udemy course gave me a strong foundation in Firebase Analytics and Google Analytics 4. Reinforcing that knowledge through hands-on implementation in React Native helped me better understand event planning, debugging, reporting, and analytics best practices for modern mobile applications. Course Certificate I'm happy to have successfully completed the Firebase Analytics & Google Analytics 4 (GA4) course on Udemy. Certificate: https://www.udemy.com/certificate/UC-4a23b92f-0857-4a75-83dd-ac186bdfdfbc The course covered both the fundamentals and advanced capabilities of mobile analytics, including GA4 reports, custom events, screen tracking, audiences, BigQuery integration, and data-driven decision making. It has been a valuable learning experience that strengthened both my theoretical understanding and practical implementation

The moment that started it A colleague of mine was cleaning up some old records in a staging environment. Same query he'd run a dozen times before, except this time he was connected to production. He hit F5. No WHERE clause. 47,000 rows gone in milliseconds . We recovered from a backup, lost about two hours, and nobody got fired. But it stuck with me:** SSMS will let you delete an entire production table with the same amount of friction as running a SELECT 1**. No pause, no confirmation, nothing. The tool that DBAs and backend developers spend all day in has zero built-in protection against the single most common way people destroy data. So I built one. What SQL Guard does SQL Guard is a free SSMS extension (18 through 22) that inspects your query the moment you press F5, and pauses execution if it matches a known destructive pattern: DELETE without WHERE UPDATE without WHERE TRUNCATE TABLE DROP TABLE / DROP DATABASE / DROP PROCEDURE ALTER DATABASE Dangerous EXEC calls MERGE without a filter When one of these fires, you get a dialog showing exactly what object would be affected, with three options: cancel, run anyway, or run and ignore for the rest of the session. The point isn't to block you — it's to make the action conscious. Most accidental damage happens because muscle memory took over, not because someone genuinely meant to wipe a table. How the detection works Nothing exotic here, and honestly that's by design. SQL Guard runs a lightweight pattern-matching pass over the query text before it hits the connection. No parsing tree, no round-trip to the server, no measurable latency — it runs in under a millisecond, so you never notice it on normal queries. The core idea is simple: certain statements are only safe when scoped by a WHERE clause, and the extension checks for that clause's absence rather than trying to understand the full semantics of the query. That keeps false positives low and means it works the same way whether you're on SQL Server 2016 or the la

Over the past few months, I challenged myself to build a complete uptime monitoring platform from scratch. The goal wasn't to build another CRUD application—it was to understand what it actually takes to design, build, and deploy a production-ready SaaS. The result is CheckForge, an uptime monitoring platform that monitors websites, APIs, and SSL certificates while sending alerts through Email, Slack, Discord, and Webhooks. Why I built it Most portfolio projects stop after authentication and dashboards. I wanted to build something that solves real backend engineering problems, including: Background workers Scheduled health checks Incident tracking SSL certificate validation Alert delivery Public status pages SVG uptime badges Production deployment Building these features taught me far more than another tutorial project. Tech Stack Fastify Node.js React Supabase (PostgreSQL) Cloudflare Workers Redis Docker Features HTTP Monitoring SSL Certificate Monitoring Expected Status Code Validation Keyword Monitoring Email Alerts Slack Alerts Discord Alerts Webhook Notifications Incident Timeline Response Time History Public Status Pages SVG Uptime Badges How it works A background worker schedules health checks at regular intervals. Each check is executed through a Cloudflare Worker, which validates: HTTP status Response time SSL certificate Expected content The results are stored in Supabase. If a failure is detected, CheckForge automatically creates an incident and sends notifications through the configured channels. ** What I learned** Building CheckForge forced me to solve problems I hadn't faced before, including: Reliable background scheduling Timeout handling Preventing duplicate alerts Incident lifecycle management SVG badge generation SSL certificate validation Cloudflare Worker integration Production deployment Building an end-to-end SaaS was a completely different experience from building standalone APIs. next set of features : Multi-location monitoring Team workspa

Most backend engineers implement multi-tenant quota checks using a standard "read-then-write" pattern. In production, this pattern is highly unsafe: SELECT grading_scans_remaining FROM profiles; If greater than 0, execute the application logic. UPDATE profiles SET grading_scans_remaining = grading_scans_remaining - 1; Under high volume or rapid concurrent requests, two independent processes will read the exact same balance before either one deducts usage. This race condition allows multi-tenant users to bypass your billing gates entirely. To solve this, you have to bypass the frontend and application-level checks, enforcing an atomic database operation that serializes the row update first. I have open-sourced a reference framework that outlines explicit subscription enums, core multi-tenant schemas, and a native VS Code / Cursor snippets configuration to speed up your local database modeling. 📂 Check out the repository on GitHub: { https://github.com/dollykm49/PostgreSQL-SaaS-Multi-Tenant-Subscription-Architecture-reference-framework- } What's inside the repository: Strictly Typed Enums: Centralized business rules handled natively by the database engine. Granular Balance Tracking: Optimized data-layer mapping for profiles and reset states. postgres-saas.code-snippets Engine: A local IDE configuration file that lets you deploy this core schema straight from your code editor by typing pg- shortcuts. For teams building commercial applications looking to skip weeks of writing custom migrations, testing concurrency edge-cases, and debugging row-locking security rules, the repository also includes a link to the extended 28-page production system bundle. Feedback on the multi-tier validation parameters is highly welcome!

I run a small self-hosted observability tool on the cheapest VPS I could find on purpose: 2 cores, 2 GB RAM, 20 GB SATA SSD . It ingests errors, traces and metrics from two low-traffic sites of mine. The stack is three containers — a Go app, PostgreSQL, and ClickHouse. One evening docker stats showed ClickHouse sitting on 880 MB of its 1 GB limit and the box swapping, with basically zero events coming in. So I went looking for where the memory and disk had gone. The answer turned out to be a good lesson in how a database can spend almost all of its I/O talking to itself. 543 KB of my data, 579 MB of ClickHouse talking about ClickHouse First thing I checked: how much data had my app actually stored versus how much ClickHouse had stored about itself . My application database: 543 KB, 16k rows The system database: 579 MB, 46.3M rows Roughly a thousand to one. Disk was 12 GB used out of 20 — on a tool that had recorded half a megabyte of real telemetry. The culprit was ClickHouse's own system logs, several of which have no TTL by default and therefore grow forever: trace_log — 404 MB, 26M rows (the query profiler writes here; it's on by default, sampling once per second) asynchronous_metric_log — 16.6M rows text_log — 132 MB plus query_log , latency_log Only metric_log , processors_profile_log and part_log ship with a TTL. Everything else just accumulates. Then I looked at the insert rate over 30 seconds: trace_log — 227 rows/s asynchronous_metric_log — 157 rows/s text_log — 44 rows/s my application — about 5 rows/s 98.8% of all inserts were ClickHouse narrating its own internals. The part that's expensive beyond disk Here's the number that made me stop. Over the same 30 seconds: rows inserted : 16,222 rows merged : 11,007,643 That's a 1 : 678 ratio. For every row written, the engine rewrote 678 already-sitting rows. The mechanics: MergeTree drops every insert into its own data part, then merges parts into bigger ones so reads stay fast. When the table is small this is

A normal secondary index can answer: status = "active" It cannot efficiently answer: documents containing "distributed database" LioranDB therefore has a dedicated text-segment architecture. Tokenization Text is split on non-alphanumeric characters. Depending on index options, tokens can be normalized to lowercase and filtered through stopwords. "Building Distributed Databases" becomes ["building", "distributed", "databases"] Segment contents A LioranDB text segment can contain several files and structures: Term dictionary Posting lists Document map Document-length norms Optional term positions Bloom filter Segment metadata A posting connects a term to the local documents containing it. "database" → [doc 2, doc 8, doc 19] Positions can record where the term appears inside each document. That enables more advanced query behaviour and phrase-aware features. Global and local document IDs Each segment assigns compact local IDs to its documents. A separate document map translates them back to global document IDs. This keeps postings smaller while preserving the external identity of the record. Bloom filters Each segment also maintains a Bloom filter for terms. Before reading a segment's postings, the query path can test whether the term might exist there. A negative answer is definitive. A positive answer means the segment may contain the term and should be checked. Query modes The text query layer supports modes such as: AND OR It also emits scored documents and metrics including: Query time Postings read Candidate documents Segments searched Full-text search is essentially a specialized database living beside the document database. Its data structures, compaction behaviour, scoring, and caching needs are different enough that treating it as a plain secondary index would be a mistake. Built by Swaraj Puppalwar under Lioran Group . Learn more: LioranDB Lioran Developer Solutions Lioran Group

Disk structures are durable, but updating them for every write is expensive. LioranDB uses memtables to absorb writes before flushing them to the on-disk B+ tree. What is a memtable? A memtable is an ordered in-memory map. In LioranDB, each entry contains either: Value ( bytes ) or: Tombstone A tombstone represents a deletion. The memtable also tracks: Approximate memory usage Minimum LSN Maximum LSN Entry count Put count Delete count The write path A simplified write path looks like this: Application write ↓ WAL durability ↓ Mutable memtable ↓ Immutable memtable queue ↓ Background flush ↓ Disk B+ tree The active mutable memtable accepts new writes. When it crosses a size limit, the engine rotates it into an immutable memtable. That immutable table is no longer modified and can safely be flushed in the background. Why ordered maps? LioranDB uses an ordered map for memtable entries. This helps because the flush process can emit keys in sorted order, which is friendly to the B+ tree and bulk-write paths. It also simplifies range merging between: Mutable data Immutable data On-disk pages Backpressure Background flushing cannot be allowed to fall behind forever. LioranDB therefore tracks limits such as: Maximum immutable memtables Maximum immutable bytes Partition-wide queue limits Maximum writer stall duration If the disk cannot drain the backlog quickly enough, the foreground write path slows down. That may sound undesirable, but controlled backpressure is much safer than consuming memory until the process dies. A memtable is not merely a cache. It is a pressure valve between CPU-speed writes and disk-speed persistence. Without that valve, the engine would either become slow on every commit or dangerously accumulate unbounded work. Built by Swaraj Puppalwar under Lioran Group . Links: LioranDB Lioran Developer Solutions Lioran Group

Most developers think of LioranDB as a document database. Internally, however, its storage engine does not understand documents, objects, fields, or JSON. It understands only: table + key bytes + value bytes That separation is intentional. The architecture LioranDB is split into two major layers: Application ↓ Document DBMS ↓ Transactional key-value engine ↓ WAL, memtables, B+ tree, pager and disk The engine exposes operations such as: get ( table , key ) put ( table , key , value ) delete ( table , key ) scan ( table , range ) The DBMS layer then adds document-oriented features: Collections JSON encoding Queries Updates Secondary indexes Text indexes Transactions For example, a secondary index can be represented as: idx:status:active → document_id A text index can be represented as: inv:database → posting_list The storage engine does not need to know what status , active , or database means. It only stores ordered bytes. Why this matters This architecture keeps the core engine small and reusable. The engine focuses on difficult low-level concerns: Durability Page management Transactions Recovery Ordering Concurrency Range scans The DBMS focuses on application-level semantics. This also makes it possible to build different data models over the same engine in the future. A document database is therefore not one giant component. It is a collection of carefully separated layers. That separation is one of the most important architectural decisions inside LioranDB. LioranDB is being developed by Swaraj Puppalwar under Lioran Group . Learn more: LioranDB Lioran Developer Solutions Lioran Group

The headline going around is that Spark 4.2 can retire your vector database. That's half true, which is the most dangerous kind of true. Spark did add real vector search, and for some workloads it genuinely removes a whole system from your stack. For others, you'd regret dropping your vector DB. Here's the honest version. The short answer If your vectors already live in your data platform and your searches are batch or analytical, Spark 4.2 can absolutely replace a separate vector database. If you're serving live, low-latency retrieval for a chatbot or search box, you probably still want a dedicated one. It's a "depends on the workload" answer, and the details matter. What Spark 4.2 actually added Spark 4.2 brought vector search into plain SQL. No bolt-on library, no separate engine. The new primitives include vector distance and similarity functions, vector normalization, vector aggregation like sum and average, and the headline one, NEAREST BY, a top-K ranking join that finds the closest matches by distance. In practice that means you can store embeddings in a Spark table and run a similarity search with SQL you already know. Those operations cover the real use cases: retrieval, recommendations, entity resolution, and candidate generation. Databricks is openly framing this as Spark becoming an AI serving layer, not just a batch engine. Why this is a big deal The value isn't that Spark invented vector search. Plenty of tools do it. The value is that you can keep your retrieval pipeline on one platform. Think about the normal setup today. Your data sits in a lakehouse or warehouse. To do vector search, you spin up a separate vector database, then build a pipeline to copy and sync embeddings into it, and keep the two in step forever. That's a second system to run, secure, pay for, and debug at 2 a.m. Spark 4.2 lets you skip that for a lot of cases. The embeddings stay where your data already is, and the search runs right there. Fewer moving parts is a real win, and i

How I Built an SSR Valorant Tracker with React, Supabase and Live Esports Data I recently built VALTRAIN , a player-focused Valorant platform that combines a Valorant Tracker, VCT match database and weapon skins explorer. The project started as a simple match history lookup page. It eventually became a much larger SSR application with player statistics, esports schedules, match detail pages, replay discovery, multilingual routes and searchable cosmetic data. The Main Product Areas VALTRAIN is divided into three primary areas. 1. Valorant Tracker The Valorant Tracker accepts a Riot ID and region. It can display: Current rank and RR Recent competitive and unrated matches KDA and combat score Headshot, body shot and leg shot data Competitive RR movement Lifetime performance statistics Individual match details and team compositions One challenge was handling incomplete or delayed data from an external player API. The interface needed useful loading, empty and error states instead of leaving users with an endless spinner. 2. VCT Match Database The VCT esports database stores upcoming and completed matches. Each public match can have: Tournament and stage information Team names and series scores Map-level results Player statistics Recent team form Official replay availability Related matches and internal links The project also includes an original VCT performance report generated from completed match records. 3. Valorant Weapon Skins The weapon skins database allows players to browse skins by collection, weapon type, rarity, price and chroma. The main performance challenge was preventing high-resolution media from slowing down the initial page load. Why I Moved the Site to SSR The original version relied heavily on client-side rendering. That worked for user interaction, but it created several problems: Public pages had limited initial HTML Search engines had to execute JavaScript Metadata was harder to control Dynamic routes occasionally returned weak fallback pages Firs

Redis gets reached for reflexively, "just add Redis," as if it were a single fix for slowness. It's genuinely one of the most useful tools in a backend engineer's kit, but using it well starts with understanding what it actually is: an in-memory data structure store, not just a cache. Once you see it as a fast, versatile store of real data structures, the range of problems it solves cleanly (caching, rate limiting, queues, sessions, leaderboards, locks) stops looking like a grab bag and starts looking like one idea applied many ways. This is the opening article of the Redis Masterclass, and it builds on the PostgreSQL series : Redis usually sits alongside a primary database like Postgres, not instead of it. In-memory is the whole point Redis keeps its data in RAM. That single fact explains most of its character. Reading from memory is orders of magnitude faster than reading from disk, so Redis operations typically complete in well under a millisecond, and a single instance handles a very high request rate. That speed is why it's the default choice for anything on the hot path, where a database round trip would be too slow. The tradeoff is that RAM is smaller and more expensive than disk, and volatile. Redis addresses durability with persistence options we'll cover later, but the mental model to start with is: Redis is fast because it's in memory, and you use it for data that benefits from being fast to access, not as the permanent home for everything. It's a data structure store, not a key-value blob The common misconception is that Redis is a simple key-value store, strings in and strings out. It's much more. Redis stores real data structures as values, each with its own commands: Strings for simple values, counters, and cached blobs. Hashes for objects with fields, like a user record. Lists for ordered sequences and simple queues. Sets for unique collections and membership checks. Sorted sets for ranked data like leaderboards and priority queues. Plus streams, bit