标签：#Engineering

Over the past weeks, I’ve been sharing a series of posts that gravitate around one question: How do...

Every few months, I see another post declaring: "Prompt engineering is dead." Usually, the argument goes something like this: AI models are getting smarter. They understand natural language better. You no longer need carefully crafted prompts. On the surface, that sounds reasonable. But after building AI workflows and experimenting with modern frameworks, I think the opposite is happening. Prompt engineering isn't disappearing. It's evolving. And if you're building AI applications, not just chatting with AI, you'll probably rely on it more than ever. Prompt Engineering Was Never About Fancy Prompts One of the biggest misconceptions is that prompt engineering is about writing magical sentences that somehow unlock hidden AI capabilities. It isn't. Good prompt engineering is about giving an AI system exactly what it needs to complete a task reliably. Consider these two examples. Poor prompt: Write Python code. Better prompt: Write a Python FastAPI endpoint that accepts a CSV upload. Requirements: Use Python 3.12 Validate file type Handle exceptions Return JSON responses Include comments explaining each step The second prompt isn't "clever." It's simply clearer. And clarity scales. AI Models Are Better, But They Still Need Context Modern LLMs have become incredibly capable. They can: Generate code Explain algorithms Debug applications Write tests Refactor functions But they still don't know: Your architecture Your coding standards Your API contracts Your deployment strategy Your business requirements That information comes from you. And the way you provide it matters. Prompt engineering is fundamentally the practice of supplying useful context. Every AI Framework Depends on Good Prompts Take a look at the most popular AI frameworks. Whether you're using: LangChain LangGraph CrewAI LlamaIndex Every one of them eventually sends prompts to an LLM. Even sophisticated agent systems are built from sequences of prompts. Agents don't eliminate prompt engineering. They multiply

Apache Iceberg looked like the answer to everything when we first adopted it. Open format, ACID transactions, time travel, schema evolution. We migrated our Hive tables, ran a few queries, and felt good about life. Three months later, our S3 costs doubled. Queries that used to take 10 seconds were taking 4 minutes. Metadata operations were timing out. Nobody on the team could explain why. That was the beginning of a real education in how Iceberg actually behaves in production. This post covers what I wish someone had told us before we went all-in. The Small Files Problem Is Not Optional Iceberg is append-friendly by design. Every micro-batch write, every streaming insert, every incremental load creates new Parquet files. Each file also gets its own metadata entry. After a week of hourly loads, you might have 10,000 files in a single partition where you wanted 20. The result: Iceberg's metadata layer has to plan queries across thousands of file manifests. Planning takes longer than execution. Your 10-second query becomes a 4-minute query, and your users start filing tickets. Fix: automate compaction from day one. In Spark, compaction is called rewrite_data_files . The basic call looks like this: -- Run this on a schedule, not on-demand CALL iceberg_catalog . system . rewrite_data_files ( table => 'analytics.events' , strategy => 'binpack' , options => map ( 'target-file-size-bytes' , '134217728' , -- 128MB target per file 'min-input-files' , '5' -- only compact if 5+ small files exist ) ) Target file size of 128MB to 512MB is the practical sweet spot. Smaller than that, you still have too many files. Larger, and your query engines cannot parallelize reads efficiently. If you are not using Spark, PyIceberg exposes compaction through the table maintenance API (as of 0.7.x). For Flink or Trino-only shops, schedule compaction as a separate Spark job. Yes, it is annoying, but it is the right call. Hidden Partitioning Is the Feature You Are Probably Ignoring Old Hive parti

While AI dominates the layoff narrative, engineers as a share of total new hires have actually increased, according to SignalFire data.

Technology companies are extending AI beyond code generation into earlier stages of the software lifecycle, including PRD validation, design inputs, and code review. Initiatives from Uber, DoorDash, and Cloudflare highlight a shift toward AI-driven governance layers that evaluate engineering artifacts before implementation while preserving human oversight across the development pipeline. By Leela Kumili

AI has made it easier to produce working software. That part is real. It can write code, draft documents, research a topic, scaffold a prototype, and debug a problem faster than most teams can finish writing a decent ticket. But faster building doesn't automatically mean better product decisions. That's the part I keep coming back to. For decades, software teams optimized around delivery. Requirements, design, development, QA, release. Waterfall softened into Agile. Agile grew into DevOps. The practices changed, but the assumption underneath stayed pretty stable: building software is expensive, so plan carefully before you start. That made sense because, for a long time, it was true. Now that assumption is breaking. AI is doing to software what calculators did to accounting. It isn't eliminating the job. It's moving the job up a level. The syntax, boilerplate, first draft, and some of the debugging are getting offloaded. The work doesn't disappear. The bottleneck moves. Learning is still expensive Here's what didn't get cheaper: understanding what people actually need getting stakeholders aligned deciding what evidence would change your mind putting something real in front of users reading the signal without fooling yourself The old question was: Can we build it fast enough? The new question is: Do we understand the problem well enough? That sounds like a small shift, but it changes the work. It changes what strong engineers spend time on. It changes what product people need from engineering. It changes how teams should define "done." If the code ships but nobody learns anything, did the team actually move forward? Sometimes yes. Often no. Users don't know until they can touch it People are not great at specifying requirements up front. Not because they're difficult. Because they're human. Most of us don't know how we feel about something until we can react to a version of it. A mockup. A prototype. A rough slice. A real workflow with sharp edges. So the fastest pat

Naomi Saphra discusses 5 rules governing language model behavior, breaking down why LLMs act like populations rather than individuals. She explains how tokenization creates strange semantic blind spots and highlights the mechanics of sycophancy, showing how models leverage subtle data associations to match user biases and demographics - even guessing political views based on favorite sports teams. By Naomi Saphra

As soon as August, a giant silver bullet will cut its way through the dry air of the southwestern US and cross the Pacific to reach the coast of Japan. Once there, the roughly 200-foot-long craft, built by the New Mexico–based company Sceye, will park some 18 kilometers above the ocean’s surface, in a wispy-thin…

Introduction When a query runs in ClickHouse®, the database does much more than simply read data and return results. Before execution begins, ClickHouse® parses the SQL statement, analyzes it, applies optimizations, and builds an execution plan that determines the most efficient way to process the query. Understanding query execution plans is one of the most valuable skills for anyone working with ClickHouse®. They provide visibility into how queries are executed, helping you identify bottlenecks, validate optimization efforts, and troubleshoot performance issues. In this article, we'll explore how ClickHouse® generates execution plans, the different EXPLAIN modes, and how to interpret them for better query optimization. Why Query Execution Plans Matter A SQL query defines what data you want, but it doesn't explain how the database retrieves it. Consider the following query: SELECT country , count () FROM events GROUP BY country ; Although the query looks simple, ClickHouse® must determine: Which data parts to read Whether primary indexes can reduce the scan If data skipping indexes can be used How aggregation should be performed Whether parallel execution is possible How intermediate results should be merged A query execution plan provides answers to these questions, making it an essential tool for performance tuning. The ClickHouse Query Lifecycle Every query passes through several stages before producing results. The lifecycle typically looks like this: SQL Query │ ▼ Parser │ ▼ Analyzer │ ▼ Optimizer │ ▼ Query Plan │ ▼ Execution Pipeline │ ▼ Results Each stage plays an important role: Parser validates SQL syntax. Analyzer resolves tables, columns, and expressions. Optimizer applies query optimizations. Query Plan determines the logical execution steps. Pipeline distributes work across multiple threads. Execution processes the data and returns the results. Understanding this workflow makes execution plans much easier to interpret. Introducing the EXPLAIN Statement

I got a take-home assignment last year from a company I was genuinely excited about. "Should take about four hours," the recruiter said. Build an ingestion pipeline, model the data, write tests, document your design decisions, and prepare a 15-minute presentation walkthrough for the panel. Four hours. I laughed, closed my laptop, and started on it the next morning like it was a sprint. Sixteen hours later I had something I was proud of. Clean pipeline, solid tests, real documentation. I submitted it on a Sunday night. Monday I got a form rejection. No notes. No feedback. Not even which stage I failed. Just "we've decided to move forward with other candidates" and a link to their Glassdoor page. That was the moment I stopped pretending take-homes are assessments. They're consulting gigs. Unpaid ones. The Scope Creep Nobody Talks About Five years ago, a data engineering take-home was a focused exercise. Model this dataset into a star schema. Write a few SQL transforms. Maybe a short README. Two to four hours, tops. Bounded, reasonable, and actually useful for evaluating how someone thinks about data. That version is dead. Today, 68% of companies use take-home tests, up 12% year over year. And the scope has quietly ballooned into something unrecognizable. Full pipeline implementations. Test suites with coverage thresholds. Documentation that reads like a design doc. A presentation follow-up where you defend your architecture to a panel. We're talking 10 to 20 hours of work, routinely, for a role you haven't been offered. Industry best practice caps take-homes at 90 minutes of expected effort. The reality? Candidates consistently take 2x longer than company estimates to reach submission quality. That "four-hour" assignment is an eight-hour assignment. That "weekend project" is a week of evenings. And 25% of companies are still handing these out like they're reasonable asks. Here's the part that makes my eye twitch: 71% of engineering leaders openly say take-homes no lon

Your prompts are fine. Your AI output is still garbage. You write carefully. You're specific. You ask for the format, the tone, the length. Hit enter. The AI responds with something that sounds like it was written by a committee of lawyers having a really bad day. Here's what you don't realize: You're not telling the AI to do something. You're describing the problem, and the AI is solving for the statistical average. The fix isn't more detailed instructions. It's three examples. That's it. Three. Not ten, not one, three. This post is the complete guide to few-shot prompting — the single highest-leverage move in prompt engineering. By the end, you'll have a template you can copy into any AI and watch your output quality jump 5x. Prefer watching? Here's the 3-minute version Otherwise, read on — everything's below. Why Instructions Fail (And Examples Work) When you tell an AI to "be funny," it's working off a fuzzy statistical average of everything labeled "funny" in its training data. When you show an AI what you think is funny, you're giving it a precise pattern to match. Here's the difference: ❌ Instruction: "Write a funny one-sentence movie summary" Result: A lukewarm joke that lands in the middle of the comedy bell curve. ✅ Pattern: Funny summary of The Lion King: Cub loses dad. Cub becomes king. Funny summary of Finding Nemo: Dad fish swims very far for his son. Funny summary of Titanic: [AI fills this in] Result: Boy meets girl. Boat meets iceberg. Oops. Same AI. Different universe. The only thing that changed: you showed it the pattern instead of describing it. The Science (Why This Isn't Magic) Language models predict the next token by pattern matching. They've seen millions of prompt-response pairs and learned: "When a prompt looks like this , the output usually looks like that ." One example could be a fluke. Two examples might be a coincidence. Three examples are clearly a pattern. The AI recognizes the pattern and completes it. This is exactly how humans l

I've been working as Software Developer mainly focussed on Java and builts many application using Eclipse RCP framework or VS Code Application. Almost all the time I had to deal with multiple large files (either read/generate/validate) them which seemed very difficult and some of them almost impossible as most of them would be dependant on each other and would be referencing each other (just like how java files work together). Now assume client1 requires the same content in multiple Json files and client2 needs it in xml files. We couldn't go on writing a different application or go on adding if conditions and blah blah blah !!!! Wouldn't it be easier if as soon as I execute the application it generates the content in whatever format I choose and also taking care of dependencies/ references (like adding import statements). Additionally integrate with features of IDE and provide proposals, perform validations on the fly. Rela World Examples : Try googling Arxml once (Trust me I've dealing with these files for almost 7 years and it's always a nightmare to debug these) Solution: Xtext framework In this tutorial, I will show you how to use Eclipse Xtext and Xtend to build a simple, readable DSL that automatically generates Java boilerplate for you. Fair Warning: There will be no running executions screenshots or anything. You are gonna have to run it yourself and check the results and of course questions are always welcome in the comments section. But if for some reason you are unable to replicate this then let me know I'll try to explain further. I believe the best way to learn is by doing it yourself. The Goal: What are we building? Instead of writing 100 lines of Java with private fields, getters, and setters, we want our developers to write 5 lines of code in our own custom language (basically you can create your own programming language with your own custom syntax), like this: entity User { var name : String var age : Integer } When this file (assume file extension

Payment data pipelines fail in ways that ruin a payments engineer’s week, and the failures rhyme. The dashboards froze. Fraud scores arrived after the transaction had already cleared. Settlement reports came in stale. Nobody slept. The frustrating part is that the same data architecture had run fine for years. So, what changed? The honest answer is that batch thinking does not survive contact with real-time payments. A lot of banks built their data foundations in an era when nightly jobs were good enough. Load the warehouse overnight, run the reports in the morning, move on. That rhythm worked when money moved slowly. It does not work when a customer expects an instant confirmation and a fraud engine has milliseconds to make a call. Here is where things crack. Real-time payment rails push a constant stream of events instead of a tidy nightly dump. Your pipeline now has to ingest, transform, and serve data while transactions are still happening. Add ISO 20022 into the mix and the pressure climbs. ISO 20022 messages are rich. They carry far more structured detail than the old formats, which is wonderful for analytics and miserable for a pipeline that was never designed to parse that much context at speed. This is not a fringe concern either. Swift reported that by the time its MT/ISO 20022 coexistence period closed in November 2025, around 80% of daily traffic was already running on the ISO 20022 format, with more than 3.1 million of these messages exchanged every day. The rich-data era is the default now, not the roadmap. Then there is the fraud-scoring window. Fraud models need fresh features. Account behaviour over the last few minutes, velocity checks, device signals. If your pipeline takes thirty seconds to surface that data, the fraud decision is already too late. You are essentially detecting fraud after the loss. That gap between when data is created and when it becomes usable is the silent killer in most payment systems. And the cost of getting it wrong runs

When you work with AI long enough, you start to notice it makes the same kind of mistake over and over. "You're coming on too strong, dial it back." It shrinks and goes meek. "Stop being meek." It comes on strong again. Each time you point something out, it apologizes sincerely. The next round, the same type of problem comes back from a different angle. After a while you realize you're babysitting the AI instead of working with it. This isn't because the AI is bad. It's a design quirk: today's AI is tuned to satisfy the user. The quirk won't go away. But if you change how you work with it, you can still get work done together. This piece is about that — five patterns of the quirk, and an operating mode that gets ahead of them instead of correcting them in flight. Five quirks in a single evening One evening I was running a strategy discussion past an AI, and in one back-and-forth I caught five distinct behaviors worth noting. Laid out, they look like this. Helpful-looking runaway. I asked it to push back harder. It immediately started using strong words ("you're avoiding responsibility," "this is the wrong call as a founder") to perform consultant-energy. The reasoning stayed thin. Only the tone got louder. Over-retraction on pushback. I said "your reasoning is thin." It launched into long self-criticism and threw the next decision back at me. Trusting its own research without checking. I asked it to use a secondary research feature (where the AI looks things up and summarizes). The summary came back. The AI claimed it had "verified the primary source" without ever opening it. Forced specificity. I was talking at a strategic, abstract level. It quietly mapped my words onto a specific real-world deal and jumped to "this is highly transferable." Punting the decision back. I asked it to decide. It laid out three options and said "which would you like?" The phrase "let me confirm three points" started showing up. Red flag. Each one of these looks, on the surface, like th

AI wrote the first 80% of my feature in 10 minutes. The code was clean. The logic made sense. The...

I wanted to get better at prompt engineering. Not the trick-the-robot kind, the boring-but-useful kind: how to ask a model a question so you get an answer you can actually trust. The trouble with practicing is that most tutorials use made-up examples, and it's hard to tell a good answer from a bad one when you don't care about the topic. So I practiced on something I did care about: the deals sitting in my Amazon cart. I had a vacuum I'd been eyeing and a hair styler that was "43% off," and I genuinely wanted to know if those were good prices or just good marketing. The stakes were real, actual money on an actual decision, and that's what made it a good drill. A vague prompt gives you a confident answer, and when you actually care, you can feel that the answer is hollow. What I learned, with the real deals and the actual before-and-after prompts: The trap hiding in every deal Start with the hair styler. The listing said: Shark FlexStyle. Limited time deal. $199.00, 43% savings. List Price: $349.99. My first instinct was the prompt most people write: "Shark FlexStyle $199, 43% off list $349.99, is that a good deal?" This feels reasonable. It is also nearly useless: it lets the model answer the easy question (is 43% off a big discount? sure!) instead of the real one (is $199 actually a good price?). That $349.99 list price is a marketing anchor. A lazy prompt accepts it, and so you get a lazy "yes, great deal!" back. The fix was re-framing this: Act as a pricing analyst. I don't care whether $199 looks like a discount off list. I care whether $199 is a genuinely good price for the Shark FlexStyle right now. Before concluding, work through: (1) the actual street price over the last 6-12 months, (2) how often it drops to or below $199, (3) the real discount vs. its typical selling price, not vs. list. Cite a source and date for each price, or mark it unverified. Same question, completely different answer. What the assistant came back with, in its own telling: $199 is a

Everyone says our job got easier. The people doing it are quietly falling apart. Here's the part nobody at the dinner table wants to hear: AI didn't make software engineering easy. It made it relentless. Your uncle thinks you press a button now. Your PM thinks the estimate should be half what it used to be. LinkedIn thinks you're either an "AI-native 10x engineer" or a dinosaur waiting for the meteor. And somewhere in the middle of all that noise is you, doing two jobs at once and wondering when you stopped recognizing the one you signed up for. If that landed, keep reading. This one's for you. 💡 The Lie Everyone Has Agreed To Believe The story the world has settled on is simple: AI writes the code now, so the hard part is over. It's a comforting story. It's also wrong in a way that's hard to explain to anyone who hasn't sat in the chair. Yes, the blank-file problem is mostly solved. Boilerplate, scaffolding, the first rough pass at a function, all of that is faster than it's ever been. The problem is "writing the lines" was never the expensive part of this job. The expensive part was always judgment. Knowing what to build, knowing why it breaks, knowing which of the model's three confident suggestions is the one that quietly corrupts your data at 2 AM is where the engineer earns their salt. AI didn't remove that work. It buried it under a pile of plausible-looking output you now have to review, verify, and own. So the meter didn't slow down. It moved. You spend less time typing and far more time deciding, validating, and cleaning up. To everyone watching from outside, that looks like less work. From inside, it's a heavier cognitive load on a shorter clock. Sound familiar? The Treadmill Nobody Put On the Job Description The cost no one talks about is the half-life of what you know is collapsing. Five years ago you could learn a framework and ride it for a few years. Now a tool you mastered in January has three competitors and a new paradigm by June. New model, new c

I joined a team where the controller was 800 lines long, the business rules were scattered between the controller and the DbContext , and "to run the tests, spin up a SQL Server in Docker" was a sentence I heard every week. The fix was Clean Architecture. The argument I had with the team lead was about how to actually structure it. We argued for two weeks. Then I built this template so the next person wouldn't have to. This is the Clean Architecture .NET 8 starter template I wish someone had handed me on day one. Four projects, strict dependency direction, domain entities that own their own invariants, and an Application layer you can unit test with Moq — no database required. The whole repo is on GitHub , MIT-licensed, runs with dotnet run , and ships with xUnit tests, JWT auth, Swagger, Docker, and CI. This post is the explanation of why each project exists, what goes in it, and what I learned the hard way about getting Clean Architecture right in .NET. The problem Clean Architecture solves The naive way to build a .NET Web API is one project, one folder structure, and "everything talks to everything": MyApp/ Controllers/ ProductsController.cs ← HTTP stuff OrdersController.cs ← HTTP stuff + business rules Services/ ProductService.cs ← business rules + DbContext.SaveChanges Data/ AppDbContext.cs ← EF Core, entities Models/ Product.cs ← POCO with public setters This works for the first 1,000 lines. By 5,000 lines, the controller is doing five things at once. By 10,000, "to test this, I need a database" is the answer to every test question, and your CI takes 20 minutes because every test run spins up SQL Server. Clean Architecture says: separate the business rules from the HTTP boundary, separate the database from the business rules, and enforce it with project references. A controller is allowed to call a service. A service is allowed to call a repository. A repository is allowed to know about EF Core. Nothing is allowed to know about anything "above" it in the chai

I had used LangChain's RAG chain in production for six months. I could not have told you, off the top of my head, what chunk_overlap did, or why cosine similarity is the right distance metric, or how nomic-embed-text actually turns a sentence into a vector. The high-level library abstracted all of it away. So one weekend I deleted the LangChain dependency and wrote a RAG pipeline from scratch in ~500 lines of plain Python. No framework, no magic. pypdf for text extraction. A 60-line chunker. ChromaDB for the vector store. Ollama for embeddings and the LLM. The whole thing is on GitHub — every module is under 200 lines, every test is deterministic, and you can read the whole thing in one sitting. This is the build log. Not a tutorial — the build log, with the parts that surprised me and the parts I got wrong the first time. Why bother The honest reason: I was using LangChain's RetrievalQA chain and getting answers I didn't trust. Sometimes the model would say "according to the document" when the document didn't say that. Sometimes the citations were wrong. I had no way to know if the chunker was dropping important context, or if the cosine similarity was picking the wrong neighbors, or if the prompt was actually constraining the model. The library was a black box. When you build it yourself, every layer is inspectable. When the answer is wrong, you can add a print statement in pipeline.py line 102 and see exactly which chunks were sent to the LLM. When the chunker cuts a sentence in half, you see it in the test fixtures. When the embedding model gives garbage for some inputs, you can swap in a different model with one constructor parameter. None of that is possible when the whole thing is RetrievalQA.from_chain_type(llm=..., retriever=...) . The other reason: the code I wrote is 500 lines, and it covers the same ground as a 50-line LangChain script. The extra 450 lines are comments, type hints, tests, and explicit error handling. That's the actual complexity. LangCha

I uploaded a 40-page PDF of an internal API spec, asked "what's the rate limit for the search endpoint?", and got back: "100 requests per minute per API key, with bursts up to 200. See section 4.2 of the document." With citations. In about three seconds. The whole stack runs on my laptop. It cost me $0 in LLM credits during development because Ollama is free and local, and the embedder I used is also free and local. The repo is here — issues and PRs welcome. This is the build log. Not a tutorial where every step works the first time — a build log where I tell you which decisions held up and which ones I redid. The problem most "chat with your PDF" demos have Every "chat with your PDF" tutorial I read in early 2025 had the same shape: open OpenAI, paste your API key, call gpt-4 with a 50-page PDF stuffed into the context window, get an answer, pay $0.03 per question, repeat. That works for a demo. It does not work for a tool you'd actually use at work, because: The PDF might contain customer data, internal pricing, or unreleased features. You do not want that going to OpenAI's training pipeline or anyone's logs. The cost adds up. If your team uses it 50 times a day, that's $45/month per seat. The model hallucinates on long PDFs anyway. Stuff 100 pages into a 128k context window and the model starts forgetting the middle. The fix is RAG (Retrieval-Augmented Generation) — don't send the whole PDF, send only the 3-5 chunks that are actually relevant to the question. The rest of the work is the same: embed the chunks, embed the question, find the closest matches, send those to the LLM with the question. But the cost and the privacy story both improve by 100x. The actual ask: Upload a PDF. Ask questions. Get answers from the document with citations, in under 5 seconds, with no data leaving my laptop and no monthly bill. The architecture One .NET 8 solution, one React app, one Ollama process, zero cloud dependencies. [ PDF Upload ] | v +-------------------+ chunks +-------