标签：#engineering

Have you ever been at a situation where you have been recently hired to maintain a legacy project, an important project at your company, but the previous team has long retired, and when you start, there is no documentation? When that happens, the old adage of "The code is the documentation" sounds true, but what happen when the code is also very old, hard to understand, and make use of libraries from when your parents were dating? In that case, using an AI Tool to help you understand how this project was created and maintained could be an option. Below are five prompt engineering techniques taken from a scientific article, with CLI based examples, to help you get the best out working with a legacy project. Zero-shot prompt A zero-shot prompt is when you make a prompt to request the model to execute a task without giving it any extra information or practical example in the input prompt. A classic example would be to ask a coding model to translate a "string.xml" file containing commonly used text strings in a natural language (for example, English) into another (for example, Spanish). Those are the easiest prompts to create and to feed to the model, but their results can be more unpredictable, since they usually lack the constraints of larger prompts. Prompt example: I want you to translate the contents of "string.xml" from English into Spanish and output it to me. Check this project files and directories, and provide me a list containing the current node version being used on this project and its libraries. Few-shot prompt In contrast to a zero-shot prompt, a few-shot prompt is a prompt including one or more pairs of the desired input and output, so the model can infer or mimic the desired output when you input the next value. A classic example would be asking a model for predictions based on a set of constraints. Those prompts tend to be longer and more complicated, and you must check the answer, because there is always a chance the model may infer incorrectly your

TL;DR — When the spec is good, implementation needs less model. I started using a top-tier model to write the spec and a cheaper, faster one to implement it — still using the strong model, just spending it on the spec instead of the implementation. The gain isn't some magic prompt phrasing; it's the context: explicit business rules, audited project constraints, a defined output contract. That's systems engineering — the discipline of anyone who's kept real software alive, whatever their stack. Every backend dev knows the scene: the Swagger is out of date, the last hotfix shipped without a unit test, and the README.md documents a command nobody's used in six months. The code works. The docs lie. And the gap between the two is exactly where AI — and we — start to go wrong. I've spent the last few months developing with AI for real inside production projects, not tutorial greenfield. My takeaway was less about which model to use and more about a shift that already has a name: the move from prompt engineering to context engineering . The difference isn't semantic. Prompt engineering treats the problem as writing — finding the magic phrase. Context engineering treats it as what it always was: a systems engineering problem . And it's where my backend background applied most directly — though anyone who's kept a real system alive has the same instinct. The experiment that convinced me Let me start with the evidence, because that's what made me take this seriously. My reflex, for a long time, was to reach for the strongest model for everything — more expensive, smarter, fewer errors. Makes sense on paper. In practice, I saw something else. When the task's specification is well done — explicit business rules, audited project constraints, a defined output format — the model capability needed for implementation drops sharply. Enough to split the work by stage: I started using a top-tier model (currently Opus) to write the spec , and a cheaper, faster model (Sonnet) to implemen

I started my career in the late 2010s, and I have had a front-row seat to the growth of the industry that has given me everything: software engineering. Looking back over the last decade, I have mixed feelings about some of the calls I made. And I am seeing the same patterns play out again now. So for engineers who are confused about where this is headed and how to navigate it, here is how I think about it. Generalist SWEs were a product of cheap money The late 2010s, I saw an huge amount of startup funding, globally. Flipkart, Snapdeal, Jugnoo, and hundreds of others were scaling hard and one hiring pattern I saw was that: everyone wanted generalist software engineers. People who could easily get upto speed across the stack.- backend, frontend, infra, deployment and simply ship. Building software was expensive. Automation was still low. Kubernetes had just gone mainstream. Shipping still meant a surprising amount of manual work: SSH-ing into servers, copying artifacts around, running mvn builds by hand, debugging deployments straight in production, duct-taping infrastructure that today you would never touch. Companies fought over engineers who maximized feature throughput. Breadth was a premium, because every extra engineer increased the rate at which software got built. It helped because the money was also free and VCs rewarded growth over efficiency, and hiring software engineers in bulk was the easiest way to spend it. Pull up a resume from an engineer who started around that time and you will usually see the same shape: a long list of technologies and frameworks, broad and adaptable, but rarely deep in any one thing. There was no incentive to go deep. LLMs Changed The Dynamics LLMs did not kill software engineering. It compressed the cost of implementation. The work that got hit first was the work that was already standardized: CRUD apps; API integration and glue code; Framework-heavy backend work; Frontend scaffolding; Standard architectural patterns. What use

As developers, data engineers, or analysts, we’ve all been there: you download a massive database export, a logging stack dump, or a transaction archive, only to find it's a multi-gigabyte JSON file. You try to import it into a spreadsheet or run it through a standard online converter, and boom—your browser tab freezes, crashes, or shows the dreaded "Out of Memory" screen. Even worse, if you try to use standard cloud-based online tools, you might have to wait for a 500MB upload to complete, only to hit a rigid file-size cap or, worse, compromise sensitive data privacy by uploading corporate logs or database records to a third-party server. In this guide, we will explore: Why large JSON files crash standard parsers (the V8 heap limit problem). How streaming architectures solve this by reading data chunk-by-chunk. NDJSON (JSON Lines) vs. JSON Arrays and how to stream them. A browser-native, 100% offline tool to convert large JSON to CSV instantly: Parsify's Large JSON Stream Converter . How to implement your own basic browser-based JSON streaming parser in JavaScript. 1. The Anatomy of a Memory Crash (Why JSON.parse Fails) If you are using JavaScript or Node.js, the simplest way to read and parse a JSON file is to load the file into memory and run JSON.parse(). const fs = require ( ' fs ' ); // Naive approach: Will crash on a 1GB+ file fs . readFile ( ' database-dump.json ' , ' utf8 ' , ( err , data ) => { if ( err ) throw err ; // POINT OF FAILURE: V8 Heap Out of Memory const records = JSON . parse ( data ); records . forEach ( record => { // Process record... }); }); This works fine for small config files. But once your JSON file reaches 100MB, 500MB, or 1GB+, this approach is guaranteed to trigger a fatal crash: FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory Why does this happen? The String Duplication Overhead: When you load a 1GB file into memory, you first allocate ~1GB of RAM for the raw text string. The

Over the past weeks, I’ve been sharing a series of posts that gravitate around one question: How do...

Every few months, I see another post declaring: "Prompt engineering is dead." Usually, the argument goes something like this: AI models are getting smarter. They understand natural language better. You no longer need carefully crafted prompts. On the surface, that sounds reasonable. But after building AI workflows and experimenting with modern frameworks, I think the opposite is happening. Prompt engineering isn't disappearing. It's evolving. And if you're building AI applications, not just chatting with AI, you'll probably rely on it more than ever. Prompt Engineering Was Never About Fancy Prompts One of the biggest misconceptions is that prompt engineering is about writing magical sentences that somehow unlock hidden AI capabilities. It isn't. Good prompt engineering is about giving an AI system exactly what it needs to complete a task reliably. Consider these two examples. Poor prompt: Write Python code. Better prompt: Write a Python FastAPI endpoint that accepts a CSV upload. Requirements: Use Python 3.12 Validate file type Handle exceptions Return JSON responses Include comments explaining each step The second prompt isn't "clever." It's simply clearer. And clarity scales. AI Models Are Better, But They Still Need Context Modern LLMs have become incredibly capable. They can: Generate code Explain algorithms Debug applications Write tests Refactor functions But they still don't know: Your architecture Your coding standards Your API contracts Your deployment strategy Your business requirements That information comes from you. And the way you provide it matters. Prompt engineering is fundamentally the practice of supplying useful context. Every AI Framework Depends on Good Prompts Take a look at the most popular AI frameworks. Whether you're using: LangChain LangGraph CrewAI LlamaIndex Every one of them eventually sends prompts to an LLM. Even sophisticated agent systems are built from sequences of prompts. Agents don't eliminate prompt engineering. They multiply

Apache Iceberg looked like the answer to everything when we first adopted it. Open format, ACID transactions, time travel, schema evolution. We migrated our Hive tables, ran a few queries, and felt good about life. Three months later, our S3 costs doubled. Queries that used to take 10 seconds were taking 4 minutes. Metadata operations were timing out. Nobody on the team could explain why. That was the beginning of a real education in how Iceberg actually behaves in production. This post covers what I wish someone had told us before we went all-in. The Small Files Problem Is Not Optional Iceberg is append-friendly by design. Every micro-batch write, every streaming insert, every incremental load creates new Parquet files. Each file also gets its own metadata entry. After a week of hourly loads, you might have 10,000 files in a single partition where you wanted 20. The result: Iceberg's metadata layer has to plan queries across thousands of file manifests. Planning takes longer than execution. Your 10-second query becomes a 4-minute query, and your users start filing tickets. Fix: automate compaction from day one. In Spark, compaction is called rewrite_data_files . The basic call looks like this: -- Run this on a schedule, not on-demand CALL iceberg_catalog . system . rewrite_data_files ( table => 'analytics.events' , strategy => 'binpack' , options => map ( 'target-file-size-bytes' , '134217728' , -- 128MB target per file 'min-input-files' , '5' -- only compact if 5+ small files exist ) ) Target file size of 128MB to 512MB is the practical sweet spot. Smaller than that, you still have too many files. Larger, and your query engines cannot parallelize reads efficiently. If you are not using Spark, PyIceberg exposes compaction through the table maintenance API (as of 0.7.x). For Flink or Trino-only shops, schedule compaction as a separate Spark job. Yes, it is annoying, but it is the right call. Hidden Partitioning Is the Feature You Are Probably Ignoring Old Hive parti

While AI dominates the layoff narrative, engineers as a share of total new hires have actually increased, according to SignalFire data.

Technology companies are extending AI beyond code generation into earlier stages of the software lifecycle, including PRD validation, design inputs, and code review. Initiatives from Uber, DoorDash, and Cloudflare highlight a shift toward AI-driven governance layers that evaluate engineering artifacts before implementation while preserving human oversight across the development pipeline. By Leela Kumili

AI has made it easier to produce working software. That part is real. It can write code, draft documents, research a topic, scaffold a prototype, and debug a problem faster than most teams can finish writing a decent ticket. But faster building doesn't automatically mean better product decisions. That's the part I keep coming back to. For decades, software teams optimized around delivery. Requirements, design, development, QA, release. Waterfall softened into Agile. Agile grew into DevOps. The practices changed, but the assumption underneath stayed pretty stable: building software is expensive, so plan carefully before you start. That made sense because, for a long time, it was true. Now that assumption is breaking. AI is doing to software what calculators did to accounting. It isn't eliminating the job. It's moving the job up a level. The syntax, boilerplate, first draft, and some of the debugging are getting offloaded. The work doesn't disappear. The bottleneck moves. Learning is still expensive Here's what didn't get cheaper: understanding what people actually need getting stakeholders aligned deciding what evidence would change your mind putting something real in front of users reading the signal without fooling yourself The old question was: Can we build it fast enough? The new question is: Do we understand the problem well enough? That sounds like a small shift, but it changes the work. It changes what strong engineers spend time on. It changes what product people need from engineering. It changes how teams should define "done." If the code ships but nobody learns anything, did the team actually move forward? Sometimes yes. Often no. Users don't know until they can touch it People are not great at specifying requirements up front. Not because they're difficult. Because they're human. Most of us don't know how we feel about something until we can react to a version of it. A mockup. A prototype. A rough slice. A real workflow with sharp edges. So the fastest pat

Naomi Saphra discusses 5 rules governing language model behavior, breaking down why LLMs act like populations rather than individuals. She explains how tokenization creates strange semantic blind spots and highlights the mechanics of sycophancy, showing how models leverage subtle data associations to match user biases and demographics - even guessing political views based on favorite sports teams. By Naomi Saphra

As soon as August, a giant silver bullet will cut its way through the dry air of the southwestern US and cross the Pacific to reach the coast of Japan. Once there, the roughly 200-foot-long craft, built by the New Mexico–based company Sceye, will park some 18 kilometers above the ocean’s surface, in a wispy-thin…

Introduction When a query runs in ClickHouse®, the database does much more than simply read data and return results. Before execution begins, ClickHouse® parses the SQL statement, analyzes it, applies optimizations, and builds an execution plan that determines the most efficient way to process the query. Understanding query execution plans is one of the most valuable skills for anyone working with ClickHouse®. They provide visibility into how queries are executed, helping you identify bottlenecks, validate optimization efforts, and troubleshoot performance issues. In this article, we'll explore how ClickHouse® generates execution plans, the different EXPLAIN modes, and how to interpret them for better query optimization. Why Query Execution Plans Matter A SQL query defines what data you want, but it doesn't explain how the database retrieves it. Consider the following query: SELECT country , count () FROM events GROUP BY country ; Although the query looks simple, ClickHouse® must determine: Which data parts to read Whether primary indexes can reduce the scan If data skipping indexes can be used How aggregation should be performed Whether parallel execution is possible How intermediate results should be merged A query execution plan provides answers to these questions, making it an essential tool for performance tuning. The ClickHouse Query Lifecycle Every query passes through several stages before producing results. The lifecycle typically looks like this: SQL Query │ ▼ Parser │ ▼ Analyzer │ ▼ Optimizer │ ▼ Query Plan │ ▼ Execution Pipeline │ ▼ Results Each stage plays an important role: Parser validates SQL syntax. Analyzer resolves tables, columns, and expressions. Optimizer applies query optimizations. Query Plan determines the logical execution steps. Pipeline distributes work across multiple threads. Execution processes the data and returns the results. Understanding this workflow makes execution plans much easier to interpret. Introducing the EXPLAIN Statement

I got a take-home assignment last year from a company I was genuinely excited about. "Should take about four hours," the recruiter said. Build an ingestion pipeline, model the data, write tests, document your design decisions, and prepare a 15-minute presentation walkthrough for the panel. Four hours. I laughed, closed my laptop, and started on it the next morning like it was a sprint. Sixteen hours later I had something I was proud of. Clean pipeline, solid tests, real documentation. I submitted it on a Sunday night. Monday I got a form rejection. No notes. No feedback. Not even which stage I failed. Just "we've decided to move forward with other candidates" and a link to their Glassdoor page. That was the moment I stopped pretending take-homes are assessments. They're consulting gigs. Unpaid ones. The Scope Creep Nobody Talks About Five years ago, a data engineering take-home was a focused exercise. Model this dataset into a star schema. Write a few SQL transforms. Maybe a short README. Two to four hours, tops. Bounded, reasonable, and actually useful for evaluating how someone thinks about data. That version is dead. Today, 68% of companies use take-home tests, up 12% year over year. And the scope has quietly ballooned into something unrecognizable. Full pipeline implementations. Test suites with coverage thresholds. Documentation that reads like a design doc. A presentation follow-up where you defend your architecture to a panel. We're talking 10 to 20 hours of work, routinely, for a role you haven't been offered. Industry best practice caps take-homes at 90 minutes of expected effort. The reality? Candidates consistently take 2x longer than company estimates to reach submission quality. That "four-hour" assignment is an eight-hour assignment. That "weekend project" is a week of evenings. And 25% of companies are still handing these out like they're reasonable asks. Here's the part that makes my eye twitch: 71% of engineering leaders openly say take-homes no lon

Your prompts are fine. Your AI output is still garbage. You write carefully. You're specific. You ask for the format, the tone, the length. Hit enter. The AI responds with something that sounds like it was written by a committee of lawyers having a really bad day. Here's what you don't realize: You're not telling the AI to do something. You're describing the problem, and the AI is solving for the statistical average. The fix isn't more detailed instructions. It's three examples. That's it. Three. Not ten, not one, three. This post is the complete guide to few-shot prompting — the single highest-leverage move in prompt engineering. By the end, you'll have a template you can copy into any AI and watch your output quality jump 5x. Prefer watching? Here's the 3-minute version Otherwise, read on — everything's below. Why Instructions Fail (And Examples Work) When you tell an AI to "be funny," it's working off a fuzzy statistical average of everything labeled "funny" in its training data. When you show an AI what you think is funny, you're giving it a precise pattern to match. Here's the difference: ❌ Instruction: "Write a funny one-sentence movie summary" Result: A lukewarm joke that lands in the middle of the comedy bell curve. ✅ Pattern: Funny summary of The Lion King: Cub loses dad. Cub becomes king. Funny summary of Finding Nemo: Dad fish swims very far for his son. Funny summary of Titanic: [AI fills this in] Result: Boy meets girl. Boat meets iceberg. Oops. Same AI. Different universe. The only thing that changed: you showed it the pattern instead of describing it. The Science (Why This Isn't Magic) Language models predict the next token by pattern matching. They've seen millions of prompt-response pairs and learned: "When a prompt looks like this , the output usually looks like that ." One example could be a fluke. Two examples might be a coincidence. Three examples are clearly a pattern. The AI recognizes the pattern and completes it. This is exactly how humans l

I've been working as Software Developer mainly focussed on Java and builts many application using Eclipse RCP framework or VS Code Application. Almost all the time I had to deal with multiple large files (either read/generate/validate) them which seemed very difficult and some of them almost impossible as most of them would be dependant on each other and would be referencing each other (just like how java files work together). Now assume client1 requires the same content in multiple Json files and client2 needs it in xml files. We couldn't go on writing a different application or go on adding if conditions and blah blah blah !!!! Wouldn't it be easier if as soon as I execute the application it generates the content in whatever format I choose and also taking care of dependencies/ references (like adding import statements). Additionally integrate with features of IDE and provide proposals, perform validations on the fly. Rela World Examples : Try googling Arxml once (Trust me I've dealing with these files for almost 7 years and it's always a nightmare to debug these) Solution: Xtext framework In this tutorial, I will show you how to use Eclipse Xtext and Xtend to build a simple, readable DSL that automatically generates Java boilerplate for you. Fair Warning: There will be no running executions screenshots or anything. You are gonna have to run it yourself and check the results and of course questions are always welcome in the comments section. But if for some reason you are unable to replicate this then let me know I'll try to explain further. I believe the best way to learn is by doing it yourself. The Goal: What are we building? Instead of writing 100 lines of Java with private fields, getters, and setters, we want our developers to write 5 lines of code in our own custom language (basically you can create your own programming language with your own custom syntax), like this: entity User { var name : String var age : Integer } When this file (assume file extension

Payment data pipelines fail in ways that ruin a payments engineer’s week, and the failures rhyme. The dashboards froze. Fraud scores arrived after the transaction had already cleared. Settlement reports came in stale. Nobody slept. The frustrating part is that the same data architecture had run fine for years. So, what changed? The honest answer is that batch thinking does not survive contact with real-time payments. A lot of banks built their data foundations in an era when nightly jobs were good enough. Load the warehouse overnight, run the reports in the morning, move on. That rhythm worked when money moved slowly. It does not work when a customer expects an instant confirmation and a fraud engine has milliseconds to make a call. Here is where things crack. Real-time payment rails push a constant stream of events instead of a tidy nightly dump. Your pipeline now has to ingest, transform, and serve data while transactions are still happening. Add ISO 20022 into the mix and the pressure climbs. ISO 20022 messages are rich. They carry far more structured detail than the old formats, which is wonderful for analytics and miserable for a pipeline that was never designed to parse that much context at speed. This is not a fringe concern either. Swift reported that by the time its MT/ISO 20022 coexistence period closed in November 2025, around 80% of daily traffic was already running on the ISO 20022 format, with more than 3.1 million of these messages exchanged every day. The rich-data era is the default now, not the roadmap. Then there is the fraud-scoring window. Fraud models need fresh features. Account behaviour over the last few minutes, velocity checks, device signals. If your pipeline takes thirty seconds to surface that data, the fraud decision is already too late. You are essentially detecting fraud after the loss. That gap between when data is created and when it becomes usable is the silent killer in most payment systems. And the cost of getting it wrong runs

When you work with AI long enough, you start to notice it makes the same kind of mistake over and over. "You're coming on too strong, dial it back." It shrinks and goes meek. "Stop being meek." It comes on strong again. Each time you point something out, it apologizes sincerely. The next round, the same type of problem comes back from a different angle. After a while you realize you're babysitting the AI instead of working with it. This isn't because the AI is bad. It's a design quirk: today's AI is tuned to satisfy the user. The quirk won't go away. But if you change how you work with it, you can still get work done together. This piece is about that — five patterns of the quirk, and an operating mode that gets ahead of them instead of correcting them in flight. Five quirks in a single evening One evening I was running a strategy discussion past an AI, and in one back-and-forth I caught five distinct behaviors worth noting. Laid out, they look like this. Helpful-looking runaway. I asked it to push back harder. It immediately started using strong words ("you're avoiding responsibility," "this is the wrong call as a founder") to perform consultant-energy. The reasoning stayed thin. Only the tone got louder. Over-retraction on pushback. I said "your reasoning is thin." It launched into long self-criticism and threw the next decision back at me. Trusting its own research without checking. I asked it to use a secondary research feature (where the AI looks things up and summarizes). The summary came back. The AI claimed it had "verified the primary source" without ever opening it. Forced specificity. I was talking at a strategic, abstract level. It quietly mapped my words onto a specific real-world deal and jumped to "this is highly transferable." Punting the decision back. I asked it to decide. It laid out three options and said "which would you like?" The phrase "let me confirm three points" started showing up. Red flag. Each one of these looks, on the surface, like th

AI wrote the first 80% of my feature in 10 minutes. The code was clean. The logic made sense. The...

I wanted to get better at prompt engineering. Not the trick-the-robot kind, the boring-but-useful kind: how to ask a model a question so you get an answer you can actually trust. The trouble with practicing is that most tutorials use made-up examples, and it's hard to tell a good answer from a bad one when you don't care about the topic. So I practiced on something I did care about: the deals sitting in my Amazon cart. I had a vacuum I'd been eyeing and a hair styler that was "43% off," and I genuinely wanted to know if those were good prices or just good marketing. The stakes were real, actual money on an actual decision, and that's what made it a good drill. A vague prompt gives you a confident answer, and when you actually care, you can feel that the answer is hollow. What I learned, with the real deals and the actual before-and-after prompts: The trap hiding in every deal Start with the hair styler. The listing said: Shark FlexStyle. Limited time deal. $199.00, 43% savings. List Price: $349.99. My first instinct was the prompt most people write: "Shark FlexStyle $199, 43% off list $349.99, is that a good deal?" This feels reasonable. It is also nearly useless: it lets the model answer the easy question (is 43% off a big discount? sure!) instead of the real one (is $199 actually a good price?). That $349.99 list price is a marketing anchor. A lazy prompt accepts it, and so you get a lazy "yes, great deal!" back. The fix was re-framing this: Act as a pricing analyst. I don't care whether $199 looks like a discount off list. I care whether $199 is a genuinely good price for the Shark FlexStyle right now. Before concluding, work through: (1) the actual street price over the last 6-12 months, (2) how often it drops to or below $199, (3) the real discount vs. its typical selling price, not vs. list. Cite a source and date for each price, or mark it unverified. Same question, completely different answer. What the assistant came back with, in its own telling: $199 is a