标签：#operations

HA dashboard cho Postgres replication: vì sao "có replica" không đồng nghĩa "đang HA", và cách để replica chết không âm thầm Streaming replication trong Postgres là một tunnel WAL từ primary sang standby: primary chạy một walsender cho mỗi standby, standby chạy một walreceiver nhận WAL và một startup process replay. Đau nhất trong vận hành không phải setup — mà là replica đã ngắt nhiều giờ mà không ai biết, tới lúc primary chết mới phát hiện HA thực ra là single-node từ tuần trước. HA dashboard là tập metric + alert được rút ra từ pg_stat_replication , pg_stat_wal_receiver , và pg_replication_slots để ép mọi trạng thái xấu — replica disconnect, replay đứng, slot pin WAL, sync standby biến mất — thành tín hiệu nhìn thấy được trước khi biến thành sự cố. Cơ chế hoạt động Trên primary, mỗi standby đang kết nối tạo ra một backend loại walsender — Postgres đọc WAL từ pg_wal/ (hoặc từ WAL buffers khi còn nóng) và stream qua replication connection. Trên standby, walreceiver nhận từng WAL record, ghi vào pg_wal/ local, fsync (tùy synchronous_commit ), rồi startup process apply record lên shared buffers — đây chính là replay. Bốn LSN xuất hiện trong luồng này và tương ứng với bốn cột trong pg_stat_replication : sent_lsn — byte cuối cùng primary đã gửi qua socket. write_lsn — byte cuối cùng standby đã write() vào OS page cache. flush_lsn — byte cuối cùng standby đã fsync xuống đĩa. replay_lsn — byte cuối cùng standby đã replay vào shared buffers (dữ liệu đã "thấy được" trên standby). Postgres docs quy định replay_lsn <= flush_lsn <= write_lsn <= sent_lsn <= pg_current_wal_lsn() — bốn "vạch" này chính là bốn nhịp của lag. Ba cột write_lag , flush_lag , replay_lag (kiểu interval ) là thời gian mà standby chậm hơn primary tương ứng với ba mốc write/flush/replay — được đo qua feedback message định kỳ từ standby. -- Trên primary: bức tranh đầy đủ cho một HA dashboard SELECT application_name , client_addr , state , -- streaming | catchup | startup | backup | stopping sync_state , --

Idle in transaction: connection ngồi không nhưng vẫn giữ snapshot, VACUUM đứng hình và bảng phình idle in transaction là trạng thái mà pg_stat_activity.state đặt cho một connection đã BEGIN , đã chạy xong statement gần nhất, và đang ngồi đợi statement kế tiếp hoặc COMMIT / ROLLBACK . Trên giấy nó "rảnh"; trong thực tế nó vẫn cầm một snapshot, vẫn giữ backend_xmin , vẫn ôm mọi lock đã claim từ đầu transaction. Một connection trong state này chỉ vài chục giây không nguy hiểm; cùng connection đó ngồi vài giờ trên cluster có write traffic là kịch bản kinh điển khiến bảng update-nóng bloat, autovacuum đứng yên, replica conflict, và DBA on-call bị page lúc 3 giờ sáng. Đây không phải bug Postgres mà là application code path đã lỡ rời transaction mở rồi đi làm việc khác (đợi user input, gọi HTTP ra ngoài, ngủ trong queue worker) — Postgres chỉ trung thực phản chiếu lại. Cơ chế hoạt động Khi một connection gửi BEGIN , Postgres ghi nhận transaction nhưng chưa cấp xid (chỉ cấp khi câu lệnh ghi đầu tiên xuất hiện) và lập tức gán cho backend này một snapshot : tập hợp các xid mà transaction sẽ coi là visible. Snapshot này được phản chiếu ra cột backend_xmin trong pg_stat_activity . Ngay sau BEGIN , state của connection là idle in transaction . Mỗi câu lệnh sau đó đẩy connection sang active trong lúc chạy, rồi trả về idle in transaction ngay khi statement hoàn tất — và nó ở đó vô thời hạn , cho tới khi client gửi câu lệnh kế tiếp, gửi COMMIT / ROLLBACK , hoặc connection bị terminate. -- Session A BEGIN ; -- pg_stat_activity.state = 'idle in transaction' -- pg_stat_activity.backend_xmin = <xid horizon ngay tại thời điểm BEGIN> SELECT count ( * ) FROM orders ; -- trong lúc chạy: state = 'active' -- xong: state = 'idle in transaction', snapshot KHÔNG bị nhả -- (client đi gọi HTTP ngoài, hoặc đơn giản là quên gửi COMMIT) -- nhiều giờ trôi qua. state vẫn 'idle in transaction'. -- backend_xmin vẫn pin xmin horizon của cluster. Điểm cốt tử là backend_xmin : VACUUM (và autovacuum ) chỉ đ

Why AI programs at PE portfolio companies stall at the same organizational seam, and what to do about it. Blake Aber · Predicate Ventures · 2026 There's a failure mode I've watched play out at enough portfolio companies that I've given it a name: the ownership dyad. It goes like this. The AI program is running. The product manager owns the roadmap (what the AI should do). Engineering owns the deployment (how it does it). Both parties are competent. Both are aligned on the goal. And the AI initiative quietly stalls anyway, usually somewhere between the promising pilot and the production system that was supposed to follow. The mechanism is diffuse accountability at the decision layer. What the dyad looks like in practice In the average portco planning meeting, the PM and the engineering lead sit across from each other. The PM has a change request: "The model is producing summaries that miss the key clause in contracts above a certain length. We should fix this." Engineering hears this and wants to know: is this a prompt change or a model change? Either requires scoping, and scoping requires the PM's input on acceptable behavior. So engineering asks the PM. The PM says "whatever's best technically." Engineering ships a prompt change. The next month, the same issue appears in a different context. The PM brings it back. Neither person is wrong. Neither person is slacking. The problem is structural: there's no single person who can describe (precisely and completely) what the AI should produce, evaluate whether it's producing it correctly, and approve a change to the system without requiring the other party's sign-off. The dyad looks like shared ownership. It functions as diffuse accountability. No one is in charge of the model's behavior. The failure mode at month nine Most portco AI programs that make it through a successful pilot still die quietly around month nine of production. The most common reason is not that the model got worse. It's that the harness around the m

So this weekend I spent $200 solving a $2 problem. Not because I was careless. Not because the system was broken in the old way. It happened because the tool was powerful, fast, confident, and wrong for just long enough. That is the strange thing about AI systems. They do not always fail loudly. A cloud server goes down, an alert fires, a dashboard turns red, someone opens an incident bridge, and the team knows what kind of movie they are in. AI failure is softer. The answer looks useful. The workflow keeps moving. The agent tries another path. The model explains itself beautifully. The bill keeps climbing. With cloud reliability, we learned how to survive machines failing. We built retries, failover, backups, autoscaling, health checks, runbooks, and incident reviews. The cloud taught us that infrastructure is never perfect, so systems must be designed to bend without breaking. AI is teaching us something different. The machine may be running perfectly and still produce the wrong result. The API may be healthy, the latency may be fine, the token stream may complete, and the business outcome may still be bad. That is why AI Site Reliability is going to become its own serious discipline. It will not be enough to ask, “Is the model available?” We will have to ask, “Is the model still useful?” “Is it drifting?” “Is it spending too much?” “Is it using the right tools?” “Is it looping?” “Is it making the same mistake with more confidence?” “Is a human needed before this continues?” In the cloud world, uptime was the king metric. In the AI world, usefulness will matter just as much. A model that is always available but often wrong is not reliable. An agent that finishes every task but spends 100 times more than needed is not reliable. A chatbot that gives answers with perfect grammar but poor judgment is not reliable. The next generation of reliability engineering will care about cost, correctness, context, and control. Cost matters because AI turns thinking into metered

This guide covers three core areas of daily GBase 8a operations: tracking data loads and collecting error details, configuring audit logs and analysing slow queries, and hierarchically tuning memory parameters. It also provides a standard daily and weekly inspection checklist for your gbase database . 1. Data Load Monitoring 1.1 Load Methods GBase 8a supports two main load methods: gload for large‑scale offline imports (recommended), and LOAD DATA INFILE for single‑file loads with MySQL‑like syntax. 1.2 Checking Load Progress Monitor running and historical loads through system tables: -- Currently executing load tasks SELECT task_id , table_name , status , start_time , loaded_rows , error_rows , TIMESTAMPDIFF ( SECOND , start_time , NOW ()) AS elapsed_sec FROM gclusterdb . load_task WHERE status IN ( 'RUNNING' , 'PENDING' ) ORDER BY start_time DESC ; -- Last 50 load history records SELECT task_id , table_name , status , start_time , end_time , loaded_rows , error_rows , TIMESTAMPDIFF ( SECOND , start_time , end_time ) AS duration_sec FROM gclusterdb . load_task ORDER BY start_time DESC LIMIT 50 ; 1.3 Retrieving the Last Load Task ID SELECT @@ gbase_loader_last_task_id ; Then query error details with that ID: SELECT * FROM gclusterdb . load_error_log WHERE task_id = 'your_task_id' LIMIT 100 ; 1.4 Error Data Collection Enable error collection in the gcluster configuration file ( gbase.cnf ) for production: gbase_loader_logs_collect = ON 1.5 Load Performance Parameters Parameter Scope Description Recommended gcluster_loader_max_data_processors gcluster Max concurrent load processing threads CPU cores / 2 gcluster_loader_min_chunk_size gcluster Chunk size sent to gnode (bytes) 64 MB gbase_loader_parallel_degree gnode Parallel write threads on gnode 4 – 8 gbase_loader_buffer_count gnode Number of load buffers 4 2. Audit Log Configuration and Analysis 2.1 Enabling Audit Logs Configure in both gcluster and gnode gbase.cnf files: audit_log = ON log_output = FILE # or TABLE

Your runbooks are out of date. I don't know your team, but I'd bet money on it. Most teams write runbooks once, in a panic after an outage, and then never touch them again until the next outage proves them wrong. How runbooks rot Steps reference deprecated tools. The grafana dashboard moved, the CLI command was renamed, the bastion host got retired. Nobody updated the runbook because nobody re-ran it during the calm months. The team that owned the system left. Three of the five engineers who wrote it are gone. The remaining two haven't actually run the runbook in 18 months because the outage type it covers stopped happening. Half-truths from the start. The original author skipped the obvious steps (because they were obvious to them) and the new on-call engineer can't reproduce the recovery. The result: at 2 AM during the actual incident, the runbook sends you down a dead end. Now you're improvising under stress. What working runbooks have in common I've seen exactly two patterns work: 1. Runbooks live with the code. Put them in the repo of the system they document. When the code changes, the runbook PR is part of the same review. Out-of-repo wikis die first because the cognitive distance is too great. 2. Runbooks are reviewed by people who weren't there. Have a junior engineer run through the runbook quarterly on a non-incident day. Every place they get stuck is a real bug in the document. Fix it then. The author will be too close to see the gaps. The three sections that matter A useful runbook has exactly three sections: Symptom : how do I know this is the right runbook? Concrete signals, with example screenshots if visual. First 5 minutes : what to do RIGHT NOW to stop the bleed. Not the root cause investigation, just the triage actions. Investigation : where to look, what queries to run, what to escalate. That's it. Anything else (architecture diagrams, history, philosophy) goes in a separate doc that the runbook links to. The cultural part The hardest part isn't

During maintenance tasks like backup or inspection, you often need to switch a gbase database cluster to READONLY mode and back to NORMAL afterwards. This post shows how to perform the switch online with gcadmin — no downtime required. Tools and Prerequisites Tool : gcadmin , located on any Coordinator node. User : Must be the cluster installation user (default gbase ). Pre‑check : Ensure gcware services are running ( gcware_services status ). Scope : The mode change applies cluster‑wide; no need to repeat per node. Step‑by‑Step (READONLY → NORMAL) Step 1: Check the Current Mode # Cluster‑wide gcadmin showcluster # Specific VC in a multi‑VC environment gcadmin showcluster vc vc1 Look at the VIRTUAL CLUSTER MODE field. Proceed only if it shows READONLY . Step 2: Switch to Normal Read/Write Mode # Single VC or whole cluster gcadmin switchmode normal # Specific VC gcadmin switchmode normal vc vc1 The switch takes effect in seconds. The cluster synchronises the new mode to all nodes automatically — no process restart is needed. Step 3: Verify the Change Run gcadmin showcluster again. When VIRTUAL CLUSTER MODE shows NORMAL , the cluster is fully writable again. Production Notes Business impact : An online switch does not interrupt running read queries. Pending write operations queued during READONLY mode are executed automatically after the switch. Node to run on : Any single Coordinator node is enough. Privileges : Only the gbase user can switch modes. Reverse switch : To return to read‑only mode, use gcadmin switchmode readonly [vc vc_name] . Troubleshooting : If the command fails, check gcware service health and node status first, then retry. Companion Commands # Check all cluster processes gcluster_services all info gcware_services all info # View detailed VC information gcadmin showvc The mode‑switch mechanism in GBase 8a is built for high availability. Following the "verify → switch → re‑verify" workflow lets you change cluster modes safely and transparently in a g

The Principle of Least Privilege: Operational Speed's Security Cost While developing a production ERP, delayed shipment reports were always a headache. One of the main reasons behind incomplete reports was the complexity of privilege layers in the system and, often, excessive permissions granted. In this post, I will delve into the costs we pay when we stretch security boundaries in an effort to gain operational speed. The principle of least privilege is more than just a security concept; it's critically important for operational efficiency and system stability. In this article, I will explain the impact of the principle of least privilege on operational speed, the security risks it entails, and how I've tried to strike this balance with concrete examples from my practical experience. My goal is to move beyond superficial definitions and dive deep into this topic based on my real-world field experiences, providing actionable insights to readers. Why Does the Principle of Least Privilege Seem to Hinder Operational Speed? The general tendency is to provide instant access to all relevant tools and data to speed up a task. This can be appealing, especially in an emergency or before a critical delivery. However, the Principle of Least Privilege (PoLP) advocates the opposite: a user or system component should have the absolute minimum privileges required to perform its task. This might initially seem to slow down operational processes. For example, a development team having unlimited SELECT rights to a production database might facilitate running an urgent query. However, the same developer could accidentally run UPDATE or DELETE commands, causing serious damage to the system. Such an incident, instead of speeding up a query in the short term, could lead to hours of downtime and data loss. This is where the long-term risk posed by operational speed, which PoLP is thought to hinder, becomes apparent. Another example is a system administrator frequently using the sudo su co