AI 资讯
Prompt Caching in LLMs: The Hidden Optimization Saving Millions of GPU Hours
Hello, I'm Shrijith Venkatramana. I'm building git-lrc, an AI code reviewer that runs on every commit. Star Us to help devs discover the project. Do give it a try and share your feedback for improving the product. Every developer eventually discovers the same frustrating pattern. Your application sends a 20,000-token prompt to an LLM. The first request takes 2 seconds. The next request contains the exact same 20,000 tokens plus a tiny user message at the end. And somehow the model processes the entire thing again. At least, that's what many developers assume. Modern LLM systems have a trick called prompt caching that can dramatically reduce latency and cost by reusing work from previous requests. But unlike traditional application caches, prompt caching isn't storing generated text. It's storing something much deeper inside the model. To understand how prompt caching works, we need to follow a prompt all the way through the transformer itself. The Expensive Part of Processing a Prompt When a prompt enters a transformer model, it isn't immediately generating text. First, the model must process every input token through every layer of the network. Imagine a prompt like: System: You are a helpful coding assistant. Project Documentation: [20,000 tokens of documentation] User: How does authentication work? Before generating a single output token, the model performs: Tokenization Embedding lookup Multi-head attention Feed-forward networks Layer normalization ...across dozens or even hundreds of transformer layers. For a large model, this preprocessing is often more expensive than generating a short answer. If another user asks: System: You are a helpful coding assistant. Project Documentation: [Same 20,000 tokens] User: Explain the database schema. Most of the prompt is identical. Without caching, the model would recompute everything from scratch. Prompt caching exists to avoid that waste. The Key Insight: Cache Internal Transformer State, Not Text A common misconception
开发者
A Wild Register Appears: Hunting the 30-Year-Old World of Xeen MT-32 Crash
submitted by /u/TheBigCore [link] [留言]
开发者
Formal methods and the future of programming
submitted by /u/swe129 [link] [留言]
AI 资讯
Every job interview I had in my life assumed that OOP was invented by Stroustrup
Here is how interview goes every single time. Interviewer asks a question: what are the basic principles of OOP? Interviewer always assumes that the answer is "inheritance, encapsulation, polymorphism". These concepts were popularized and have direct connection to C++. C++ was designed by Bjarne Stroustrup. If I try to say that Stroustrup himself said that he didn't invent OOP [1] and try to talk about Simula, Smalltalk and Alan Kay's definition of OOP I always have a concerned look from the interviewer. I'm tired. I don't even have deep understanding of history of programming languages. It's just a one google search that leads to Wikipedia article[2]. It's common knowledge. Why it's a problem every time a have an interview? --- [1] Wired speaks to Bjarne Stroustup : Please note that my claim to fame is not to have invented OOP. I did not - that honour belongs to the designers of Simula: Ole-Johan Dahl and Kristen Nygaard - but I did have a major hand in making it mainstream. [2] https://en.wikipedia.org/wiki/Object-oriented_programming#History submitted by /u/FG3149 [link] [留言]
开发者
Developing a pkg.go.dev TUI explorer
submitted by /u/der_gopher [link] [留言]
产品设计
[video] Search Autocomplete - System Design
submitted by /u/lucian-12 [link] [留言]
AI 资讯
How to enjoy programming in a world of AI
It's widely accepted that as AI writes more and more of our code, opportunities decrease for newcomers to gain the skills needed to become professional programmers. At the same time, experienced people bemoan the way AI is causing them to gradually lose the skills they took so long and worked so hard to gain. Programming, like mathematics and perhaps music, sits in the middle between science and art. As science, it's all about capturing and organising knowledge, and as art, it's the application of creativity to problem-solving. Enjoyment - deep satisfaction or an emotional rush - can be had from either or both of these. But the perception is that AI is depriving us of opportunity, both to learn and to be creative. Code written by a human reveals a lot about the coder. Their depth of knowledge of the coding language, their creative use of variable and function names, and even the visual layout of a script. You can often tell at a glance whether the writer truly cared about the code they wrote, or whether it was just a means to an end, to ensure their next pay packet arrives on time. As AI improves, it makes fewer mistakes, but it also produces code that is increasingly hard for any human to read and truly understand. Since the job of a human in the coding loop is increasingly not to write but to validate code, this is becoming a problem. So here's my recent story, a series of happy accidents. I've been using agentic AI for about a year. First with Copilot in VS Code, then more recently with Claude Code at the command line. At the start, I found Copilot to make a lot of mistakes and to produce rather repetitive, poorly-structured code, but looking back, a lot of that was probably down to my own ignorance of how to control the process. The biggest problem I had was in validating JS or Python code. There was just so much of it, often using features of both languages I never fully got to grips with. I've never been a first-class programmer and most of my own code is pure
AI 资讯
Email Data Normalization for Automation
submitted by /u/PGurskis [link] [留言]
开发者
How a TCP Load Balancer works under the hood.
submitted by /u/Sushant098123 [link] [留言]
开发者
Git merges can be better
submitted by /u/agentvenom1 [link] [留言]
开发者
Type Theory Forall #62 - Dependent Haskell - Vladislav Zavialov
submitted by /u/mttd [link] [留言]
AI 资讯
Struct Embedding in Go: Composition That Bites When You Reach for Inheritance
Book: The Complete Guide to Go Programming Also by me: Thinking in Go (2-book series) — Complete Guide to Go Programming + Hexagonal Architecture in Go My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools Me: xgabriel.com | GitHub You come to Go from a language with classes. You see struct embedding for the first time, and it reads like inheritance. A field with no name, methods that "carry over" to the outer type, a base struct that your type extends. So you write code the way you always have, and most of it works. Then a method does something you did not ask for, a type satisfies an interface you never meant to implement, or two embedded types fight over a name and the compiler shrugs until the exact line that calls it. Embedding is not inheritance. It is composition with a syntax that promotes methods and fields up one level. Once you hold that distinction, the surprises stop being surprises. Here is where they come from. Embedding promotes, it does not subclass Write an embedded field by giving a type with no field name: type Engine struct { Horsepower int } func ( e Engine ) Start () string { return "vroom" } type Car struct { Engine // embedded Brand string } Car now has a Start method and a Horsepower field, both promoted from Engine . You can write car.Start() and car.Horsepower as if they were declared on Car . car := Car { Engine : Engine { Horsepower : 300 }, Brand : "Fiat" } fmt . Println ( car . Start ()) // vroom fmt . Println ( car . Horsepower ) // 300 This is where the inheritance illusion starts. car.Start() is sugar. The compiler rewrites it to car.Engine.Start() . The receiver of Start is still an Engine , never a Car . There is no base class, no super , no virtual dispatch. Engine does not know Car exists. That last point is the one that bites. A promoted method runs against the embedded value, not the outer struct. The method that ignores the outer struct Say you want a stringer on the embe
创业投融资
Typescritp: Sobrecarga de Construtor
Introdução Assim como funções, construtores podem ter múltiplas assinaturas: O problema class Evento { constructor ( id : string , tipo : string , competencia : string ) { ... } // como aceitar também só id e tipo, sem competencia? } Solução — overload signatures class Evento { id : string ; tipo : string ; competencia : string ; // assinaturas constructor ( id : string , tipo : string ); constructor ( id : string , tipo : string , competencia : string ); // implementação constructor ( id : string , tipo : string , competencia : string = " nao-definida " ) { this . id = id ; this . tipo = tipo ; this . competencia = competencia ; } } new Evento ( " 1 " , " R-2010 " ); // ✅ primeira assinatura new Evento ( " 1 " , " R-2010 " , " 2024-01 " ); // ✅ segunda assinatura
AI 资讯
defer in Loops: The Resource Leak Go Still Lets You Write
Book: The Complete Guide to Go Programming Also by me: Thinking in Go (2-book series) — Complete Guide to Go Programming + Hexagonal Architecture in Go My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools Me: xgabriel.com | GitHub You wrote a function that walks a directory and reads every file. It opened cleanly in review. It passed tests on a fixture folder with three files. Then a customer pointed it at a folder with 20,000 files, and the logs filled with: open /data/batch/img-08431.png: too many open files The code never closed anything early. It deferred every close. And that is exactly the problem. defer runs at function return, not end of iteration defer schedules a call to run when the surrounding function returns. Not when the current block ends. Not when the loop iteration ends. When the function returns. Most of the time that distinction does not matter, because most deferred calls live in short functions that return quickly. Put a defer inside a for loop, though, and every iteration adds one more deferred call to a stack that does not unwind until the whole loop is done and the function exits. Here is the version that ships: func processFiles ( paths [] string ) error { for _ , p := range paths { f , err := os . Open ( p ) if err != nil { return err } defer f . Close () // runs at RETURN, not here if err := handle ( f ); err != nil { return err } } return nil } Read it the way the language reads it. On the first iteration, os.Open returns a file handle and you defer its close. On the second iteration, another handle, another deferred close. By the time the loop has touched 20,000 files, you are holding 20,000 open descriptors, and not one of them closes until processFiles returns. Your process has a file-descriptor limit. On Linux the soft default is often 1024. You blow through it somewhere around the 1024th file, and the error you get back says nothing about defer . It says too many open files , wh
AI 资讯
Types of loops in JS
Programming is all about solving problems efficiently. Two concepts that play a major role in writing reusable and efficient programs are loops and functions . Loops help us perform repetitive tasks without writing the same code again and again, whereas functions help us organize code into reusable blocks. Let's understand these concepts in detail. Why Do We Need Loops? Suppose we want to print "Hello" five times. Without loops, we would write: console . log ( " Hello " ); console . log ( " Hello " ); console . log ( " Hello " ); console . log ( " Hello " ); console . log ( " Hello " ); Although this works, it violates one of the fundamental principles of programming: Don't Repeat Yourself (DRY) Repeating code: Increases the number of lines. Makes maintenance difficult. Introduces more chances for errors. Loops solve this problem by allowing us to execute the same block of code multiple times. Types of Loops in JavaScript JavaScript provides three looping statements: Loop Type Category while Entry-Check Loop for Entry-Check Loop do...while Exit-Check Loop Entry-Check Loop / Entry-Controlled Loop In entry-Check loops, the condition is checked before executing the loop body. If the condition is false initially, the loop body never executes. Examples: while loop for loop Exit-Check Loop / Exit-Controlled Loop In an exit-Check loop, the loop body executes first and then checks the condition. Therefore, the body executes at least once. Example: do...while loop Components of Every Loop Every loop generally consists of three parts: 1. Initialization Determines where the loop starts. let i = 1 ; 2. Condition Determines whether the loop should continue executing. i <= 5 3. Increment or Decrement Updates the loop variable after each iteration. i ++ ; or i -- ; 1. while Loop The while loop repeatedly executes a block of code as long as the condition remains true. Syntax while ( condition ) { // statements } Example: Print Numbers from 1 to 5 let i = 1 ; while ( i <= 5 ) { cons
AI 资讯
Reading a Paginated API Without Holding the Whole Thing in Memory
Your API hands out 50 records at a time across 400 pages. You need all of them. You do not need them all at once. Here's a very familiar situation that shows up constantly on the backend. Some API returns data in pages, 50 or 100 records at a time, and you need to walk every page: sync them to your database, export them to a file, run a report. The endpoint gives you a cursor or a page number and you keep asking until there's nothing left. The way most of us write it the first time looks like this: async function getAllRecords () { const all = []; let cursor = 0 ; while ( cursor !== null ) { const { records , nextCursor } = await fetchPage ( cursor ); all . push (... records ); cursor = nextCursor ; } return all ; } const everything = await getAllRecords (); for ( const record of everything ) { process ( record ); } It works. At four hundred records it's fine. The trouble starts when the dataset grows, and it has three separate problems hiding in it. It holds the entire dataset in memory before you touch a single record. It's all or nothing: if page 380 fails, you've thrown away the 19,000 records you already fetched . And it's eager. You can't start processing record one until the very last page has landed , even if all you wanted was the first ten. There's a shape in JavaScript built for exactly this, and if you read the first two posts in this series you already have both halves of it. Two ideas you've already seen In the CSV post , we pulled rows out of a huge file one at a time with a generator, so the file never fully loaded into memory. Lazy. Pull-based. You ask for the next row, you get the next row, nothing more. In the async/await post , we saw that a generator can pause at a yield and resume later.A generator can hold its place across an asynchronous gap. Put those together. A generator that pulls data lazily, and can pause to await something between pulls. That's an async generator, and it's the natural tool for walking a paginated API. You pull records
AI 资讯
What Happened When I Told Codex to Calm Down
I have been doing a lot of work lately tightening up my diagnostic suite: the mechanics, the workflow, the way it runs against target repos, the way it helps narrow a repair instead of letting everything turn into a fog machine. And because I work with Codex as my coding agent, I have also become very familiar with a specific kind of AI-agent behavior. The “I am helping so hard I am about to make this worse” behavior. If you work with coding agents, you probably know the vibe. You ask for one thing. The agent does that thing. Then it also adjusts a helper. Then it updates a fixture. Then it “notices” a nearby pattern. Then it starts explaining three other improvements you never asked for. And now you’re staring at the diff like: “Why are you in that file?” “I did not tell you to touch that.” “That was not the repair lane.” “Please stop being useful for one second.” I am not proud of how many times I have verbally threatened a language model. But here we are. The funny thing is, I am building Scarab partly because I already expect this kind of drift. I know that when an AI coding agent is given too much uncertainty, it tries to solve the uncertainty itself. Sometimes that is useful. Sometimes it is a raccoon with a soldering iron. The challenge is that while I am developing the diagnostic system, I cannot always use the diagnostic system to supervise itself. So there are moments where I have to manually hold the line. That means a lot of conversations with Codex that sound like: “Do not widen the patch.” “Do not change the diagnostic output to make the diagnostic pass.” “Do not fix the test by changing what the test means.” “Do not touch SDS mechanics while repairing the target repo.” “Stay in the target.” “Stay in the lane.” “Why are you like this?” Very normal. Very calm. Very professional. Then something changed At some point, after a lot of tightening, the workflow started to feel different. Scarab had enough of the diagnostic work under control that I could tell
开发者
System and game performance monitoring with Python
submitted by /u/riklaunim [link] [留言]
开发者
How does DynamoDB figure out which keys are out of sync across replicas ?
submitted by /u/Normal-Tangelo-7120 [link] [留言]
开发者
Signals, the push-pull based algorithm
submitted by /u/fagnerbrack [link] [留言]