标签：#Engineering

Hardwood, the project Gunnar Morling kick-started handling of Parquet files in Java, reached version 1. Its multi-threaded approach and zero mandatory external dependencies promise a simpler, more efficient alternative to the Apache Parquet Java implementation. For now, the library supports just reading; writing support is expected in the upcoming versions. By Olimpiu Pop

Introduction As data volumes continue to grow, running aggregation queries directly on raw datasets becomes increasingly expensive. Business dashboards, analytics platforms, and reporting systems often execute the same calculations repeatedly—such as total sales, daily active users, page views, or revenue trends. While ClickHouse® is designed to process analytical workloads at remarkable speed, repeatedly scanning billions of records still consumes valuable CPU, memory, and storage resources. This is where AggregatingMergeTree proves its value. Rather than calculating aggregates every time a query is executed, AggregatingMergeTree stores intermediate aggregation states that are merged automatically in the background. This approach allows analytical queries to read compact, pre-aggregated datasets, resulting in dramatically faster response times and reduced infrastructure costs. In this guide, you'll learn how AggregatingMergeTree works, why aggregate states matter, how to build an automated aggregation pipeline using Materialized Views, and when this engine is the right choice for your ClickHouse® workloads. What is AggregatingMergeTree? AggregatingMergeTree is a specialized ClickHouse® table engine designed to store aggregate function states instead of raw records. Unlike the standard MergeTree engine, which stores every inserted row, AggregatingMergeTree keeps partially aggregated values that ClickHouse combines during background merge operations. This significantly reduces the amount of data that must be processed when generating analytical reports. Because much of the computational work happens during data ingestion, dashboards and reporting applications can retrieve summarized information much more efficiently. Typical scenarios include: Sales reporting Website traffic analytics Financial summaries IoT sensor monitoring Business KPI dashboards Application observability metrics Why Use AggregatingMergeTree? Imagine an online marketplace processing millions of tr

The speakers discuss Agent RFT, OpenAI’s platform for fine-tuning reasoning models via real-time tool interactions and custom reward signals. They explain how reinforcement learning solves complex credit assignment challenges within the context window. They share enterprise success stories, showing how Agent RFT eliminates long-tail token loops and drives extreme efficiency. By Wenjie Zi, Will Hang

One question I get surprisingly often is: "How do you manage thousands of AI prompts without losing track of them?" The answer is simple. I don't treat prompts as conversations. I treat them as reusable software assets. Over the years, I've created prompt libraries across multiple AI projects, books, research initiatives, and client work. That means managing well over 10,000 prompts covering everything from Python development and AI agents to content generation and workflow automation. If you're still storing prompts in random ChatGPT conversations, you're making life much harder than it needs to be. Here's the system that works for me. Stop Thinking of Prompts as Temporary Most people write a prompt, get an answer, and move on. That's fine for casual use. But builders rarely solve the same problem only once. If you find yourself writing: API documentation SQL queries FastAPI endpoints Docker configurations Code reviews Git commit messages ...you're probably solving recurring problems. Recurring problems deserve reusable prompts. My Folder Structure Instead of organizing prompts by AI tool, I organize them by purpose. For example: AI-Prompts/ │ ├── Python/ │ ├── FastAPI │ ├── Django │ ├── Flask │ └── Automation │ ├── JavaScript/ │ ├── React │ ├── Node.js │ └── TypeScript │ ├── DevOps/ │ ├── Docker │ ├── Kubernetes │ └── GitHub Actions │ ├── AI/ │ ├── RAG │ ├── Agents │ ├── MCP │ └── Prompt Engineering │ └── Documentation/ This mirrors how software projects are organized. Finding a prompt takes seconds. Every Prompt Has Metadata A prompt isn't just text. It's documentation. Each prompt in my library includes: Category: Purpose: Model: Input: Expected Output: Version: Last Updated: For example: Category: FastAPI Purpose: Generate CRUD endpoints Model: GPT-4o Expected Output: Production-ready FastAPI code Six months later, I know exactly why that prompt exists. I Version My Prompts Developers version code. Why not prompts? For example: FastAPI_CRUD_v1.md FastAPI_CRUD_v

For decades, developers were judged by one thing: How much code they could write. The best programmers wrote faster. Debugged faster. Built faster. That era is ending. The next generation of developers won't spend most of their time writing code. They'll spend it directing AI. Welcome to the age of the AI Orchestrator. The Evolution of Software Development Software development has always evolved. First, developers wrote machine code. Then came assembly. Then high-level languages. Then frameworks. Then cloud platforms. Then DevOps. Each evolution removed repetitive work and let developers focus on bigger problems. AI is simply the next step. But this time, it isn't replacing a tool. It's becoming a teammate. Coding Is Becoming a Smaller Part of the Job Building software isn't just writing code. A typical project includes: Understanding requirements Researching documentation Designing architecture Writing code Reviewing code Debugging Testing Writing documentation Deploying applications Monitoring production Fixing incidents Only one of those is coding. Everything else is coordination and decision-making. That's where AI is changing the game. From Programmer to Orchestrator Think about how modern teams work. A tech lead rarely writes every line of code. Instead, they: Assign work. Review solutions. Provide feedback. Make architectural decisions. Remove blockers. Developers are beginning to work with AI in much the same way. Instead of writing every function, they'll: Define the goal. Provide the right context. Choose the right tools. Review AI-generated code. Run tests. Improve weak areas. Approve the final result. The value shifts from typing code to guiding its creation. What Does an AI Orchestrator Do? An AI orchestrator doesn't ask one question and accept one answer. They manage a workflow. For example: Break a large project into smaller tasks. Give each AI the context it needs. Decide when to retrieve documentation. Decide when to search the codebase. Ask AI to g

You merge the PR. The build passes. The code does what you expected it to do. You move on. That is review for most engineers. A final read. A feeling that things looked right before the branch closed. I did it the same way for years. Three phases had already run before this one. Think had scoped the work, Plan had written the requirements, Build had shipped a diff that matched the plan exactly. I trusted that the chain held. I had never actually checked. Then I ran the Review phase, and checking turned out to mean something specific: not does this work, but does this requirement hold up, and what is my evidence. I went in expecting to approve it or send it back. The phase gave me three answers instead: covered, partial, missing. I found out what they meant one requirement at a time, starting with the one I almost got wrong. I had been giving impressions, not verdicts The notification scheduler used a queue to manage dispatch. Every call to the external provider went through it. The provider was never exposed directly. The requirement said the provider must be notified. It was notified, exactly the way I had pictured it. I almost called it covered and moved to the next line. The Review phase stopped me there. But the requirement said must be notified , not how. The queue had introduced a call order and a timing the requirement never anticipated. Nothing was broken. Something had changed shape, quietly, and nobody had written that shape down. I sat with that for longer than I expected to. Not because the code was wrong. Because I could not immediately tell you whether the change mattered. The same pass gave the shim from Plan a different verdict on the same page: covered. Mapped to the requirement it existed to satisfy, no gap between what was promised and what was in the diff. One requirement held exactly the shape it was given. The other had quietly grown a new one. Same review. Same pass. Two verdicts. Partial is not a softer word for broken. It is the verdict for

A company shifted from project- to product-thinking after their platform outgrew single-team use. The limitations that they felt with their platform were one-off deliveries, lack of product vision, and weak feedback loops. They have moved toward a self-service, API-driven, multi-tenant infrastructure with clearer ownership and better abstractions. By Ben Linders

I used to waste hours tweaking and re-tweaking my AI model prompts. It was like trying to find a needle in a haystack—I'd make a change, run the code, wait for the results, and then... nothing. The output would be inconsistent, unhelpful, or just plain wrong. I'd try again with tiny modifications, rinse and repeat, until I was about to pull my hair out. It wasn't until I stumbled upon the concept of reusable prompt templates that everything changed. It was like a switch had flipped—my code started producing consistent results, and I finally understood why. No more guesswork, no more frustration. Just good old-fashioned productivity. A simple shift from writing one-off prompt strings to using reusable templates is the key to reducing prompt overhead, increasing consistency, and getting back to doing what we love—building amazing, AI-driven applications. From Chaos to Control: A Simple Example Let's make this tangible. Imagine you're building a feature to generate a short story, but for different characters. Before: The Inconsistent, One-Off Way Without a template, you'd likely write a new prompt each time, introducing small, unintentional differences that lead to wildly different results. Two separate prompts = inconsistent, unpredictable output prompt_for_alex = "Write a short story about a character named Alex who is trying to get to work on time, but keeps getting delayed in a busy city." prompt_for_jordan = "Generate a story about someone named Jordan. They're late for work and stuck in traffic in a big city." See the problem? The tone, wording, and details are different. You have no control over the consistency of the output. After: The Clean, Templated Way Now, let's use a single template. We define the core structure once and simply pass in the parts that change. Now, let's use a single template. We define the core structure once and simply pass in the parts that change. One template = consistent, predictable output story_template = "Write a short story about

The speakers discuss Netflix’s architecture for surviving extreme traffic spikes. They explain the mechanics of prioritized load shedding embedded in their Envoy sidecar proxy, allowing user-initiated requests to steal capacity from non-critical traffic. They share automated platform strategies for continuous chaos load testing, config generation, and retry storm mitigation. By Anirudh Mendiratta, Benjamin Fedorka

📚 Series: Databricks on AWS (Part 0, prologue) The Target Architecture ← you are here Building a Databricks AI Platform on AWS RBAC with Function-Role Groups Compute Governance: Pools, Policies, Clusters The BOOTSTRAP_TIMEOUT Mystery Fixing It with AWS PrivateLink How We Structure the Terraform Before the build story, here's the destination. This is the target-state data architecture we designed the whole platform toward — the three principles that shaped every later decision, and the Unity Catalog governance model that keeps production data safe from human hands. The rest of this series is a build log: workspaces, RBAC, compute, the networking rabbit hole, the Terraform layout. But every one of those decisions was made in service of a target picture we drew first . This post is that picture — the "to-be" architecture, not the scaffolding we happened to have up on any given week. It's built on three things Databricks basically hands you if you lean into them: the Lakehouse (one store, ACID tables, no separate warehouse to sync), the Medallion architecture (raw → cleaned → integrated → business, each layer a promotion), and Unity Catalog as the single governance plane across all of it. The interesting part isn't reciting those three buzzwords — it's the specific way we wire them so that prod, dev, and analyst sandboxes never step on each other. Three principles, and everything follows Almost every concrete rule later in this series is a consequence of one of these three. 1. Nobody touches production by hand. Create, update, delete in prod data happens only through an automated, code-reviewed pipeline running as a service principal. Human accounts don't get write on prod — not analysts, not engineers, not admins. The blast radius of a bad afternoon is capped at whatever a person can do with read-only. This one principle is why the whole "promote" flow later exists. 2. Never copy production to look at it. If an analyst wants to explore the gold layer, they read it in p

Pressing enter to accept model suggestions now takes less effort than scrolling past it. One keystroke, and the code is yours. Reading it, understanding it, deciding if it's actually right, that part hasn't gotten any faster. That gap, between how fast we can accept code and how fast we can actually understand it, is where things start to go wrong. The new shape of technical debt We used to know where technical debt came from. Tight deadline, cut corner, # TODO: comment that nobody ever revisits. Rushing was the cause, and we could at least point to it. Now you can build up the same kind of debt on a calm Tuesday afternoon, no deadline in sight, just six suggestions in a row accepted because they looked fine and the flow felt good. Nobody rushed you, and the code still ended up just as unexamined. Same debt, just a different excuse. "It works" is not the same as "I understand why it works" Everyone knows that debugging is twice as hard as writing a program in the first place. So if you're as clever as you can be when you write it, how will you ever debug it? — Brian Kernighan, 1974 Fifty years later, the gap got wider. Kernighan was talking about code you wrote. At least you understood it once. A suggestion that compiles, passes the linter, survives code review and even comes with passing tests can still be standing on a wrong assumption that nobody caught, because nobody was reading it as code. They were reading it as output, and output that makes sense tends to get approved. Compiling is a low bar. Passing tests is a slightly higher one, depending on whether you wrote the tests, or its suggestion shaped or created those too. If it's the second, it's like grading its homework with its own answers. None of it tells you the logic is sound, that the edge cases are covered, or that it does what you actually needed, something we already learned every time we trusted code we didn't write. Somehow it's easy to forget it the moment the code appears inline, in our own edito

CodeTrace-AI v1.0.1 — Stop Reading Code. Start Understanding It. Every developer has experienced this. You clone a repository, open it, and suddenly you're staring at thousands of files. You spend hours answering questions like: Where is this function called? Which files depend on this module? What happens if I modify this class? Is this code even used anymore? Traditional tools like grep , IDE search, or AI chat assistants can help you find code. They don't help you understand the architecture . That's why I built CodeTrace-AI . What is CodeTrace-AI? CodeTrace-AI is an AI-powered code intelligence tool that transforms your repository into a searchable structural knowledge graph. Instead of treating your project as plain text, it understands your codebase structurally by analyzing: 📂 Folder hierarchy 📄 Files 🏛 Classes ⚙ Functions 📦 Imports 🔗 Function calls 🌐 Cross-file dependencies Think of it as having an AI Software Architect that understands your entire repository. 🚀 What's New in v1.0.1 This release focuses on speed, privacy, and understanding large repositories. 🕸 Interactive Code Graph One of the biggest additions is the interactive repository graph. Instead of reading hundreds of files manually, you can visualize relationships between: Folders Files Classes Functions Imports Function calls Understanding a new project becomes dramatically easier. ⚡ SHA-256 Delta Sync Engine One feature I'm particularly proud of is the new Incremental Indexing Engine. Most code intelligence tools rebuild their entire index every time. CodeTrace-AI doesn't. It computes a SHA-256 fingerprint for every tracked file and detects: ✅ Modified files ➕ Newly added files ❌ Deleted files Only those files are: Re-parsed Re-embedded Re-added to the knowledge graph Everything else is skipped. This makes repeated indexing dramatically faster, especially for large repositories where only a few files change between runs. Under the hood The sync engine includes: SHA-256 fingerprinting Parallel f

Tree of Thoughts was a genuine leap. Instead of reasoning in one straight line, it branches into several lines, scores them, prunes the dead ends, and searches for the best path — so a puzzle that would sink a single chain of thought becomes solvable. But a tree has one restriction baked right into its shape, and once you see it you can't unsee it: every node has exactly one parent. A branch can be extended or abandoned. It can never be combined with another branch. That matters more than it sounds. Real problems decompose, and when they do, different branches each get part of the answer right. Branch A nails the first half; branch B nails the second half; neither is fully correct on its own. A tree is forced to pick one and throw the other's good half away. Graph of Thoughts (GoT) removes exactly that restriction. 🕸️ Interactive demo (a real merge-sort that branches, merges, and refines — with live-verified scores): https://dev48v.infy.uk/prompt/day21-graph-of-thoughts.html The core idea: thoughts are nodes in a graph GoT reframes reasoning as building a graph . Each vertex is a thought — a partial solution or intermediate result. Each edge is an operation that produced one thought from one or more others. Because it's a general graph and not a tree, a thought is allowed to have several parents, and edges can even loop back on themselves. That single change in the data structure is the entire conceptual leap. Everything else is just the operations you're now free to run on that network. The four operations generate (branch) — the familiar move, straight from Tree of Thoughts. From one thought, produce several different next thoughts. This can also be a split : break the input into independent sub-problems solved on separate branches. Diversity matters here — near-duplicates waste budget. score / rank — turn each thought into a number so the controller can compare them. Objective scorers win: a validator, a test, a metric. In the demo, the scorer is deliberately con

Cassie Shum discusses the architectural evolution of GraphRAG and why data foundations are critical for advanced AI workflows. She explains how traditional vector RAG falls short when addressing global context, multi-hop reasoning, and provenance. She shares enterprise strategies for building semantically structured knowledge graphs that shift raw orchestrating logic down to the data layer. By Cassie Shum

We all use AIs today - From a 5th grader to a retired pensioner, from a small-time business owner to a multimillionaire businessman, from a software engineer to a medical expert. AI exists everywhere! And to be honest its making our lives very simple. Yes, it does!. Response in no time, flexibility, reliability - yes, AI gives all and even more And as Software Engineers, we are getting more inclined towards AI. Back in the days, we used to rely on Stackoverflow to get our queries resolved. Sometimes it did, sometimes it didn't. But, AI changed that landscape completely - asking a query, retrieving data, asking follow-ups and so and on so forth. But, honestly, how many of us have thought - Wow this looks amazing! But how does it actually work! Let's say I type this in Chat GPT or Gemini or Claude etc: "Hi, how is the weather today?". The AI assistant takes the input and processes it and returns the response. But , there is a lot of processing and workflow happening under the hood. As a Software Architect, I struggled a lot to get these answers. Different sources, different suggestions. And the suggestions at some point seemed too overwhelming for me. So, I decided to break it down and start a series which will enable people to understand AI. I want to make people understand AI in the simplest way possible and make every developer leverage AI - not just to get their job done, but also to help in upskilling, so that they don't get lost in the overwhelming world of AI as I did initially! Follow me for more updates!

As AI agents work under increasingly less human supervision, the need for a trustworthy, secure work...

Security Team: “I have a major Grype...with what I Syfted out of your provided image." Developer: “Well your Grype is slowing me down...let’s tone it down a notch.” While deploying bookstack into my local environment, this issue surfaced. It is true for many organizations today deploying images and packages in their environment. How can this noise fatigue in the software supply chain be remedied? Add a .gype.yaml file to the root directory of your project. This will allow grype to ignore certain CVE's that do not execute or pose a threat in your environment. The yaml config can be as simple as below: Linux Environment # grype.yaml ignore : - vulnerability : CVE-2026-32631 reason : " Platform-specific false positive. Git for Windows only; not applicable to this Linux-based image." - vulnerability : CVE-2016-2781 reason : " Chroot escape via ioctl. Containers rely on namespaces/cgroups, not chroot, so this path isn't exploitable here." OR # grype.yaml ignore : - vulnerability : CVE-2026-32631 - vulnerability : CVE-2016-2781 This will help developers and security engineers get along better. 😃 Grype config reference: https://oss.anchore.com/docs/reference/grype/configuration/

When most people hear Bitcoin , the conversation usually starts with price. But for developers, Bitcoin is much more than a chart. Bitcoin is a distributed system operating without a central authority. It combines networking, cryptography, game theory, economics, and software engineering into a protocol that has remained operational for years while processing value globally. As a software developer, what fascinates me most is not speculation it’s the architecture. Some concepts every developer can appreciate: ⚡ Distributed Consensus Thousands of nodes independently verify the same rules without trusting each other. 🔐 Cryptography in Practice Digital signatures make ownership verifiable without revealing private keys. ⛏️ Proof of Work A mechanism that converts computation into security and coordination. 🌍 Open Source at Global Scale Anyone can inspect the code, run a node, contribute, or build on top of the ecosystem. 📦 Immutability Through Design Data integrity is achieved through incentives, validation rules, and chained blocks. Studying Bitcoin changes how you think about: System reliability Security models Network design Incentive structures Building software that survives failure Whether you plan to build in blockchain or not, Bitcoin is worth studying because it teaches principles that extend far beyond finance. Curious to hear from other developers: What concept in Bitcoin architecture changed the way you think about software systems?

Elastic open-sourced Atlas, a system built on Elasticsearch that maintains three categories of memory for agents. Atlas integrates with agents via MCP and maintains per-user isolation of memories. When evaluated on question-answering capability, it scored 0.89 Recall@10. By Anthony Alford

TL;DR Anthropic recently published When AI Builds Itself, an essay explaining how AI is...