标签：#Engineering

When we started working on Krenalis , we spent a lot of time reviewing how customer data typically flows through a modern data stack. One pattern kept showing up often enough that we started questioning it. In many modern stacks, customer data already lands in a warehouse. Yet we often copy that same data into a CDP before we can start building customer profiles. During one of those discussions, someone asked a question that sounded almost naive: Why are we moving all this data in the first place? Nobody had a particularly strong answer ready. The answer was mostly: Because that's how CDPs work. We expected the question to have an obvious answer. It didn't. The warehouse is no longer just for analytics Over the last few years, the role of the data warehouse has changed significantly. Warehouses are no longer just analytical systems. They're increasingly becoming the place where organizations centralize the context used by applications, AI agents, copilots, and business processes. Customer data from systems like Shopify, Stripe, CRMs, support platforms, and internal applications often ends up there long before anyone starts thinking about segmentation or activation. In many organizations, the warehouse is already the place where teams answer questions about customers, revenue, retention, and product usage. That made us wonder: If the warehouse is already becoming the operational center of the data stack, why does customer identity usually live somewhere else? Consider a customer who buys through Shopify, pays through Stripe, opens support tickets in Zendesk, and uses the product under a different email address. In many organizations, all of those records already end up in the warehouse. Yet building a unified profile often requires exporting that same data into another platform before identity can be resolved. The cost of another copy To be clear, data duplication is not inherently bad. Most software systems rely on some form of replication, caching, or denormalizati

The Term Everyone Uses and Nobody Defines Your CTO came back from a conference and said the team needs to "go agentic." A vendor pitched you an "agentic data platform" last week. LinkedIn is full of posts about agentic workflows transforming everything from customer support to supply chain management. And yet, when you ask three people what "agentic" actually means for your data operations, you get four answers. This is not a vocabulary problem. It is a strategy problem. Organizations are making six-figure decisions about agentic AI without a shared definition of what they are buying, building, or hiring for. That gap between the buzzword and the architecture is where most projects fail -- not because the technology does not work, but because nobody agreed on what it was supposed to do. This guide is a practitioner's attempt to close that gap. No vendor pitch, no hand-waving. Just a clear definition, a real example, and a framework for deciding whether agentic workflow consulting is something your team actually needs. What "Agentic" Actually Means (In Plain Language) Traditional data pipelines are deterministic. You define steps, connect them in order, and run them. Step A feeds step B, which feeds step C. If the input changes shape, the pipeline breaks and a human fixes it. The pipeline does not adapt, reason, or make decisions -- it executes. Robotic process automation (RPA) is slightly smarter but still scripted. It records human actions and replays them. Click here, type there, move this file. When the UI changes or an edge case appears, the bot breaks the same way a pipeline breaks: it stops and waits for a human. Agentic workflows are fundamentally different. An agentic system has components that can reason about their task, make decisions based on context, and take actions without a pre-scripted path for every scenario. Instead of "if X then Y," an agentic node can evaluate ambiguous input, choose between approaches, validate its own output, and route work to

Most engineers build distributed systems hoping nothing breaks. Amazon S3 was engineered under the opposite assumption: that something is already broken, right now, and the system needs to be fine with that. That one mindset shift explains almost everything about how S3 works — and why it's one of the most reliable pieces of infrastructure on the planet. I went through a deep-dive conversation with Mai-Lan Tomsen Bukovec, VP of Data and Analytics at AWS, and extracted the engineering philosophy underneath the product. Not the marketing version. The real one. Here's what actually matters. 1. Hardware failure is not an emergency. It's Tuesday. S3 manages hundreds of exabytes of data across tens of millions of hard drives, spread across 120 Availability Zones in 38 AWS Regions. It currently stores over 500 trillion objects. At that scale, something is always failing. A disk here. A rack there. An availability zone every now and then. The math is unforgiving. So the S3 team made a deliberate architectural decision early: stop treating failure as an exception. Design it into the system as the baseline state. This means dedicated auditor and repair microservices run continuously in the background — not when something goes wrong, but always. They scan the entire fleet, inspect every byte of data, detect discrepancies, and trigger repairs automatically. No human in the loop. No incident ticket. No war room. There's also a specific property they engineer for called crash consistency — the system is designed so that after any fail-stop event, it automatically returns to a valid state without manual intervention. The failure happens. The system continues. Those two things are not in conflict. The system heals itself because it was designed to assume it's already sick. If you're building distributed systems and your failure handling is reactive — you only respond after something breaks — you've already lost. Design the repair loop as a first-class citizen, not an afterthought.

İlk yazıda Event Driven Architecture’ın temel kavramlarını, Kafka üzerinde topic/channel tasarımını, event-command ayrımını, schema contract’ları ve producer-consumer ilişkisini ele aldık. Bu yazıda odağı bir adım ileri taşıyıp event’in platform içindeki yaşam döngüsüne bakacağız. Çünkü EDA tasarımında asıl zorluk yalnızca event üretmek değildir. Asıl mesele, üretilen event’in güvenilir, izlenebilir, tekrar işlenebilir, zenginleştirilebilir ve farklı tüketiciler tarafından kullanılabilir hale gelmesidir. Bu yazıda şu sorulara odaklanacağız: Ham event platforma geldiğinde ne olur? Event nasıl doğrulanır, zenginleştirilir ve tüketilebilir hale gelir? Raw, validated, enriched ve curated topic’ler nasıl konumlandırılmalıdır? Bu yapı modern lakehouse mimarilerindeki Medallion yaklaşımıyla nasıl ilişkilendirilebilir? DLQ ve alert topic’leri ne zaman devreye girer? Replay, idempotency, monitoring, security ve governance nasıl düşünülmelidir? Event Pipeline Nedir? EDA mimarilerinde özellikle data platform projelerinde event’ler genellikle bir yaşam döngüsünden geçer. Bu yaşam döngüsü şöyle modellenebilir: raw -> validated -> enriched -> curated | | v v dlq alert Bu yapı, veri akışının aşama aşama olgunlaşmasını sağlar. Raw topic kaynaktan gelen ham event’i taşır. Validated topic schema ve temel kalite kontrollerinden geçmiş event’leri içerir. Enriched topic event’in referans veriler veya başka veri kaynaklarıyla zenginleştirilmiş halidir. Curated topic ise tüketiciler için güvenilir, normalize edilmiş ve iş anlamı netleşmiş event’leri temsil eder. Event Pipeline ve Medallion Architecture İlişkisi Bu yapı, modern lakehouse mimarilerinde sık kullanılan Medallion yaklaşımıyla doğal bir benzerlik taşır. Lakehouse tarafında Bronze katmanı ham veriyi, Silver katmanı temizlenmiş ve zenginleştirilmiş veriyi, Gold katmanı ise iş tüketimine hazır veri ürünlerini temsil eder. Kafka üzerindeki raw, validated, enriched ve curated topic’leri de benzer bir olgunlaşma mantığını akan veri ü

Anthropic published a thoughtful guide to making skills. It is worth reading, but it's a map of work you should not have to do. The Claude Code team wrote a piece on how they use agent skills . If you make skills, read it. It is honest and tells you something important: making a good skill is real work. Here's what the guide covers. It sorts skills into nine categories. It explains progressive disclosure, where the agent knows which files to load and when. It covers scripts, config files, combining skills together, and writing the description so the model reaches for the skill at the right moment. All of that is true and useful. It is also a lot to learn. And most of it exists only because you are doing the work by hand. We're SkillsCake . We make and score agent skills all day. So we read this guide a little differently than someone meeting skills for the first time. Here's what we think. Skills are infinite The guide splits skills into types: library reference, verification, and so on. That is a helpful way to teach a class. It is not what a skill actually is. A skill is prose that tells an agent how to do one thing, sometimes with scripts attached. The set of possible skills is not nine boxes. It is every job you could describe in writing; it's infinite. Categories are how a person gets a handle on something that open-ended. They are scaffolding for learning, not the shape of the thing. This matters because the moment you think in categories, you start bending your skill to look like the example in its bucket. Your real job rarely fits the bucket. The best engineered skill is the one written for your exact task, by an expert. Doing it yourself might not be worth it Progressive disclosure, scripts, config, descriptions tuned for the model, gotchas earned by failing, and eval loops: none of that is busywork. It's how a good skill gets built by hand. The guide is not overcomplicating anything. It is being honest about what the manual path costs. But that is the poin

If you've been calling yourself a "prompt engineer" for the past two years, it's time to update your vocabulary — and your mental model. In 2026, the real leverage when building LLM-powered systems isn't in crafting the perfect sentence. It's in context engineering : designing everything an LLM sees before it ever generates a response. Andrej Karpathy coined the term in mid-2025, and it's since taken over serious AI engineering discussions. This article breaks down what context engineering actually is, why it matters more than prompt writing, and gives you concrete techniques you can apply today. What Is Context Engineering? Context engineering is the discipline of systematically designing the information environment that surrounds a prompt. Where prompt engineering asks "what should I tell the model to do?", context engineering asks "what does the model need to know to do it well?" Think of it this way: a doctor doesn't just answer the question you ask on the spot. They look at your chart, your history, your vitals, and then respond. Context engineering is building that chart for your LLM. The context window is the LLM's working memory — everything it can "see" at once. In 2026, these windows are massive: Claude Opus 4.x : 200K tokens GPT-4o : 128K tokens Gemini 2.5 Flash : Up to 1M tokens But bigger isn't automatically better. More tokens = more cost, more latency, and a real risk of what researchers call the "lost-in-the-middle" problem — where models process information at the beginning and end of the context more reliably than content buried in the middle. Why This Matters for Data Engineers Data engineers are increasingly building pipelines that feed LLMs: RAG systems, AI copilots for data quality, agents that write and review SQL, tools that summarize data lineage. In every one of these systems, the quality of what lands in the context window directly determines output quality. A poorly designed context is like feeding a senior analyst a jumbled mess of raw l

To provide SRE as a service, a team built a center of excellence, introducing Federated SREs and roles like production manager and technical tribe lead. They created a culture of data-driven conversations where SLOs and SLAs were democratised. Surviving growing cognitive load meant continuously simplifying architecture and embedding sovereignty and resilience into platform design decisions. By Ben Linders

The speakers discuss the architectural challenges of executing safe data deletion across distributed datastores. Balancing durability, availability & correctness, they explain how to orchestrate multi-system deletion propagation without impacting live traffic. They share lessons on controlling tombstone accumulation, building continuous audit loops, and gaining trust with a centralized platform. By Vidhya Arvind, Shawn Liu

Last week, Toronto hosted Tech Week. A city-wide celebration filled with events and workshops...

The hidden cost of AI isn't generating code. It's understanding your codebase. For a long time, I assumed AI coding tools became expensive because they generated a lot of code. These tools can produce components, tests, SQL queries, documentation, and sometimes entire features on demand. If costs were climbing, the output volume must be the reason. The more I used these tools, the more I realized I was measuring the wrong thing. The expensive part isn't writing code. The expensive part is understanding what code should be written — and that work is mostly invisible. That realization changed how I think about AI-assisted development entirely. Two Prompts, Two Very Different Problems Consider these two requests: "Create a utility function that formats dates" and "Review this feature and suggest improvements." At first glance, both look ordinary. Both might even produce short answers. But they require completely different levels of understanding. The first is narrow and well-defined. The AI needs very little information before it can produce a useful answer. The second is open-ended. Before suggesting a single improvement, the AI may need to read multiple files, understand dependencies, follow existing patterns, compare implementations, and build a mental model of why the feature exists at all. The output might still be small. The work required to reach it is not. Why Agent Workflows Feel Different From Autocomplete This became much clearer when I started using AI agents. Traditional autocomplete is predictive — you type, the AI guesses what comes next. It's fast, cheap, and deliberately context-light. Agents behave differently. When you ask one to improve a feature or review a workflow, it doesn't immediately start generating code. It starts reading. It follows imports, finds related files, and tries to understand the system before touching it. That is exactly what makes agent workflows feel slower and more resource-intensive than autocomplete: they are spending effor

Sepehr Khosravi discusses the evolution of developer productivity tools. Evaluating the strengths of tools like Cursor and Claude Code, he explains actionable techniques for senior engineers - including context engineering, custom rules, and Model Context Protocol (MCP) integrations. He shares real-world benchmarks and strategic frameworks for balancing AI adoption with clean code quality. By Sepehr Khosravi

CAP Theorem Explained: Choosing Between Consistency, Availability, and Partition Tolerance in Databases Imagine you're trying to book a flight online, and just as you're about to pay, the website crashes. When you try to book again, you find that the flight is now sold out, even though the website initially showed available seats. This frustrating experience is a classic example of a database trade-off between consistency, availability, and partition tolerance. The CAP theorem, first introduced by Eric Brewer in 2000, states that it's impossible for a distributed data store to simultaneously guarantee more than two out of these three principles. In this post, we'll delve into the world of CAP theorem, exploring its fundamentals, real-world database examples, and design implications. Introduction to CAP Theorem Understanding the Basics of CAP Theorem The CAP theorem is based on three primary principles: Consistency : Every read operation will see the most recent write or an error. Availability : Every request receives a response, without guarantee that it contains the most recent version of the information. Partition Tolerance : The system continues to function and make progress even when network partitions (i.e., splits or failures) occur. Importance of CAP Theorem in Distributed Systems In distributed systems, where data is spread across multiple nodes, the CAP theorem plays a crucial role in understanding the trade-offs between these principles. By grasping the CAP theorem, developers can design more resilient and scalable databases that meet the specific needs of their applications. Brief Overview of the Blog Post This post will explore the CAP theorem in depth, using real-world database examples to illustrate the trade-offs between consistency, availability, and partition tolerance. We'll discuss the fundamentals of CAP theorem, examine CA, CP, and AP systems, and provide guidance on designing for each combination. By the end of this post, you'll have a solid un

After migrating Spark pipelines to Azure Kubernetes Service, two infrastructure settings interacted destructively: spark.kubernetes.local.dirs.tmpfs=true backed shuffle spill with RAM instead of disk, and a hard podAffinity rule forced all executors onto one node. Together, they caused repeated OOM kills invisible to standard diagnostics. By Pranav Bhasker

Most API teams treat the contract as an afterthought: write code, generate a spec, then watch the two drift apart. Git-native API design reverses that flow. You treat the API contract as source code, version it in Git, and review every change the same way you review application logic. Try Apidog today This guide focuses on implementation discipline, not a single tool. You’ll design contracts in branches, review them in pull requests, and turn a committed spec into mocks, tests, and docs. The goal is simple: your Git history should also be your API history. If you already know what Spec-First tooling looks like and want the product walkthrough, read the companion piece on the git-native API workflow . This article stays focused on practice. What “git-native” means for API work Git-native means your API definition lives in your repository as a plain text file. Not in a proprietary cloud database. Not behind a vendor login. A .yaml or .json file sits next to your code and is tracked by the same version control system your team already uses. In many cloud-locked API design tools, the contract lives in the vendor’s backend. You edit through a web UI, and your repository only contains an export. That export can become stale, and your Git history no longer explains how the API evolved. The git-native model inverts that relationship: The file in main is the contract. Any GUI is a view onto that file. Branches, commits, pull requests, blame, and rollback all apply to your API surface. Mocks, docs, tests, and generated clients derive from the committed spec. A git-native setup has three core properties: The spec is a text file in the repo. Changes flow through normal Git operations: branch, commit, PR, merge. Downstream artifacts derive from the committed file, not from a separate database. Why design and develop APIs in Git You already trust Git with your code. Your API contract deserves the same treatment. 1. History When someone asks, “When did we add the cursor pagination

T-SQL on Microsoft Fabric - Episode 1: Mastering Data Retrieval with SELECT, WHERE, and ORDER BY Learning Goals In this lesson, you will learn how to: Read data from tables using SELECT Filter rows with WHERE Sort query results with ORDER BY Get familiar with standard T-SQL syntax Practice directly in Microsoft Fabric Warehouse 1. Understanding Database and Schema In Fabric Warehouse, objects are commonly organized like this: Warehouse | |-- sales | |-- Customers | |-- Orders | |-- hr | |-- Employees | |-- finance |-- Transactions Schemas help you: Group related tables Manage permissions Organize large systems more effectively 2. Create a Schema Create a schema for the sales dataset: CREATE SCHEMA sales ; Check existing schemas: SELECT * FROM sys . schemas ; 3. Create Tables Create the Customers table: CREATE TABLE sales . Customers ( CustomerID INT , CustomerName VARCHAR ( 100 ), City VARCHAR ( 50 ), Country VARCHAR ( 50 ) ); Create the Orders table: CREATE TABLE sales . Orders ( OrderID INT , CustomerID INT , OrderDate DATE , Amount DECIMAL ( 10 , 2 ) ); 4. Insert Sample Data Customers INSERT INTO sales . Customers VALUES ( 1 , 'John Smith' , 'New York' , 'USA' ), ( 2 , 'Emma Brown' , 'Chicago' , 'USA' ), ( 3 , 'David Wilson' , 'London' , 'UK' ), ( 4 , 'Sophia Taylor' , 'Manchester' , 'UK' ), ( 5 , 'Michael Lee' , 'Singapore' , 'Singapore' ); Orders INSERT INTO sales . Orders VALUES ( 101 , 1 , '2026-01-10' , 1200 . 00 ), ( 102 , 1 , '2026-01-15' , 800 . 00 ), ( 103 , 2 , '2026-01-20' , 2500 . 00 ), ( 104 , 3 , '2026-02-01' , 500 . 00 ), ( 105 , 5 , '2026-02-05' , 3200 . 00 ); 5. SELECT Get all columns: SELECT * FROM sales . Customers ; Get specific columns: SELECT CustomerName , Country FROM sales . Customers ; 6. Alias Rename columns in the output: SELECT CustomerName AS Customer , Country AS Nation FROM sales . Customers ; 7. WHERE Filter rows using conditions. Customers in the USA: SELECT * FROM sales . Customers WHERE Country = 'USA' ; Orders greater than 100

There is a version of organizational life that feels, from inside, like maturity. Meetings reach conclusions. Decisions move forward without extended debate. The leadership team operates with visible coherence. Escalations are rare. When concerns are raised, they are quickly absorbed into the existing framework and resolved without disruption. Everything functions exactly as designed. That is what makes it difficult to notice that something essential has stopped occurring. The more stable the system appears, the more completely it has eliminated the conditions under which instability would be visible. The Epistemic Function of Conflict Conflict in organizations is not primarily a social problem. It is an epistemic mechanism. When a decision is challenged, something precise occurs: the decision's internal logic is forced into the open. Its assumptions are made explicit. Its evidence is tested against contrary interpretation. The challenge does not guarantee a better outcome — but it generates information the unchallenged decision never produces. Conflict is not disruption of the system. It is how the system verifies itself against reality. Remove the disagreement, and the system continues deciding. It simply stops testing whether its decisions are sound. The absence of challenge feels like confidence. It is blindness — a blindness that is, from inside, indistinguishable from clarity. What Fills the Vacuum When legitimate conflict disappears, the space does not remain empty. It fills with the performance of conflict. Meetings still contain discussion. Questions are still asked. Concerns are occasionally raised. But the texture has changed in ways that experienced practitioners feel before they can articulate. Questions are asked to signal engagement rather than to probe assumptions. Concerns are framed to demonstrate awareness rather than to force resolution. Debate occurs within the boundaries of what the system has already decided is acceptable to debate. The ritual

In this article, author Aaditya Chauhan discusses the limitations of RAG pipelines based purely on vector search and how an internal omni-search application using Reciprocal Rank Fusion (RRF) that combines BM25 and vector results, can enhance the search solution. By Aaditya Chauhan

Switching focus from Frontend development to Data Engineering means shifting from building user interfaces to architecting robust data pipelines. It’s a completely different mindset, and the learning curve is exciting! As I dive deeper into the world of Data, these are the 4 essential tools and concepts that have become the absolute backbone of my daily learning roadmap: 1️⃣ Python (The Swiss Army Knife): Coming from JavaScript/TypeScript, picking up Python has been a breath of fresh air. From writing custom ETL scripts to data manipulation with Pandas, it's the ultimate language for data manipulation. 2️⃣ Advanced SQL (The Core): It's not just about simple SELECT statements anymore. Mastering Window Functions, CTEs (Common Table Expressions), and query optimization is where the real magic happens when interacting with Data Warehouses. 3️⃣ ETL/ELT Pipelines: Understanding how to efficiently Extract, Transform, and Load data without breaking downstream analytics. Moving from UI state management to Data state management is a game-changer. 4️⃣ Cloud Ecosystems & Modern Stack: Exploring how data flows through modern cloud environments and learning how big data tools manage scale. The transition requires patience, but applying my previous engineering background to these new tools makes the journey incredibly rewarding. 💡 To the Data Engineers in my network: What is the one tool or concept you believe is a "must-have" for someone transitioning into the field today? Drop your advice below!

A security reviewer finds a critical issue a day or two before the release of an application. While it's an important issue, it sets the team back weeks, frustrating their product management partners and customers. The review came at the most expensive time in the process. There are many examples of how work items move through different processes to deliver software in large companies. While GenAI has allowed us to rapidly create code, it also moved and exposed the bottlenecks in our processes. It has also caused us to re-examine where it is most effective to make certain decisions. This is the challenge, and a deliberate blend of automated, programmatic, and human judgment is well suited to help you solve it. We can borrow from the well-trodden path of value stream mapping here. It is useful for spotting bottlenecks and waste in a given process, but it's also valuable to ask the deeper question of who or what should own each step. Each option earns its place differently. Is there an earlier step that may reduce costs with an agent where it was previously limited by human availability? Or is the stronger determinism of a programmatic step more important for a critical piece of the flow? Some decisions should stay with human judgment, where confidence without context is a liability. The opportunity for security teams and other stakeholders is to scale their impact across these options rather than scaling headcount. Workflow-as-code is not a new idea. There are a number of existing engines where the workflow definition is its own entity, separate from the work itself. GitHub Actions defines pipelines in version-controlled files, while the execution happens on separate runners. Airflow and Temporal follow a similar pattern for data and application workflows. Because the definition lives on its own, a team can change how a given step runs without rebuilding the whole flow. That separation is what makes it practical to adjust who or what owns each step over time. Rather

Anthropic introduced Dynamic Workflows, a new capability for Claude Code designed to handle complex software engineering tasks by coordinating large numbers of AI agents within a single workflow. The feature allows Claude to dynamically create orchestration scripts, break work into subtasks, run them in parallel, and validate results before presenting a final answer. By Robert Krzaczyński