AI 资讯
How We Actually Measure Whether an LLM's Output Is Good - BLEU, COMET and BLEURT
Hello, I'm Shrijith Venkatramana. I'm building git-lrc, an AI code reviewer that runs on every commit. Star Us to help devs discover the project. Do give it a try and share your feedback for improving the product. An AI model writes a paragraph. It sounds fluent. It looks convincing. But how do you know whether it's actually good? This deceptively simple question has occupied researchers for more than two decades. Long before ChatGPT, machine translation researchers faced exactly the same problem. Human evaluation was expensive, inconsistent, and painfully slow. If every new model required thousands of humans to compare translations, research would crawl. That necessity gave rise to BLEU , one of the most influential evaluation metrics in AI history. Years later, as language models became better at paraphrasing and reasoning, BLEU started to show its age. Researchers responded with learned metrics like BLEURT and COMET , which use neural networks to judge language much more like humans do. Interestingly, this mirrors software engineering itself. We first wrote simple unit tests, then integration tests, and today we increasingly rely on sophisticated observability systems. Evaluation metrics for LLMs have undergone a similar evolution. Let's see why. Before BLEU: The Evaluation Bottleneck Imagine you're building Google Translate in 2001. Every time your team improves the model, someone has to read thousands of translated sentences and score them. Suppose a single sentence pair takes only 20 seconds to judge. Evaluating 50,000 sentences would require nearly 280 human-hours . Now imagine dozens of experiments every week. Evaluation—not training—quickly becomes the bottleneck. Researchers at IBM, led by Kishore Papineni , introduced BLEU (Bilingual Evaluation Understudy) in 2002 to automate this process. Their idea was surprisingly simple: If a machine translation resembles what professional translators write, it's probably good. This became one of the most cited papers
AI 资讯
Building a Slack Bot That Actually Remembers: slacktag-oss
How I built an open-source Slack assistant with persistent semantic memory, powered by any LLM and Mem0's managed memory layer — no vector database required. The problem with Slack bots and memory Most AI Slack bots have the memory of a goldfish. Every conversation starts from scratch. You ask it about your sprint goals, it gives a great answer, then three days later you ask a follow-up and it has no idea what you're talking about. You end up re-explaining context constantly. The commercial solution to this is Claude Tag — a Slack integration that maintains genuine conversational continuity. But it's tied to one provider and not open-source. slacktag-oss is our attempt to replicate that experience: a Slack bot with real, semantic, persistent memory that works with any LLM — including ones running entirely on your laptop. What I built A Python Slack bot with: Socket Mode for local dev (no public URL needed), HTTP-ready for prod LangChain to abstract LLM calls across any OpenAI-compatible endpoint Mem0 managed cloud for semantic memory — no Qdrant, no Pinecone, no infra to run Three memory scopes: per-channel, per-thread, per-DM Built-in !clear and !memory commands A clean, extensible architecture you can fork and build on Architecture Before diving into code, here's the full request lifecycle: ┌─────────────────────────────────────────────────────────────┐ │ Slack │ │ @mention in channel ──┐ │ │ DM to bot ──┼──► Slack Events API │ │ Thread reply ──┘ │ │ └───────────────────────────────────│─────────────────────────┘ │ (Socket Mode / HTTP) ▼ ┌─────────────────────────────────────────────────────────────┐ │ slack-bolt (Python) │ │ bot.py ──► router.py ──► handler.py │ │ │ │ │ ┌───────────────┤ │ │ │ │ │ │ ▼ ▼ │ │ Mem0 Client LangChain │ │ (managed) ChatOpenAI │ └────────────────────────────────────────────────────────────-┘ │ ▼ ┌───────────────────────┐ │ Mem0 Managed Cloud │ │ Vector Embeddings │ │ Entity Extraction │ │ Deduplication │ └───────────────────────┘ The ke
AI 资讯
AI writes code in seconds. Architecture debt takes months to notice.
One thing I've noticed after using AI for development over the past year is this: The code it generates is usually correct. The architecture slowly isn't. That doesn't happen because AI writes bad code. It happens because architecture rarely erodes all at once. Imagine a modular application with clear boundaries. The billing module talks to the orders module through its public interface. Authentication is isolated. Notifications are independent. Everything is predictable. Now imagine hundreds of AI-assisted commits over the next few months. One suggestion imports an internal class because it already exists. Another bypasses a service layer because it's shorter. A helper gets copied into another module. A database query is duplicated instead of reused. None of those changes are catastrophic. In fact, every pull request probably gets approved. The application still builds. The tests still pass. Customers never notice. Until one day, making a simple change requires touching five different modules because everything has quietly become connected. That's architecture debt. And unlike a failing test, it doesn't show up immediately. One thing I've realized is that our current tooling doesn't really watch for this. Unit tests verify behavior. Integration tests verify interactions. Linters enforce style. Static analysis finds bugs. All of those are important. But none of them are asking questions like: Should this module depend on that one? Did someone bypass a defined boundary? Are we introducing new architectural coupling? Is the overall architecture getting healthier or worse over time? Those questions usually get answered during code review. Or worse, during a production incident. The interesting part is that AI isn't really the problem. If anything, it's doing exactly what we ask it to do. It optimizes for solving the problem in front of it. Architecture, on the other hand, is about protecting the system as a whole. Those are different goals. As AI makes us write code fa
AI 资讯
How I Targeted Youtube with my Rust Web Browser. And what it taught me
DISCLAIMER : NO AI OR LLMS WERE USED IN WRITING THIS ARTICLE . EVERY SINGLE WORD WAS TYPED MANUALLY USING A KEYBOARD I HAVE A COMPLETE UNEDITED VIDEO OF ME WRITING THE ARTICLE HERE: https://www.youtube.com/watch?v=6j1dbN64LyI submitted by /u/Inevitable_Back3319 [link] [留言]
开发者
Tuning a Server for Benchmarking
Optimizing code starts with measuring it, and a measurement is only useful if it is repeatable: a 2% improvement is invisible under 5% of noise. Yet on an untuned machine the same binary can easily run several percent faster or slower between runs. In this post we take a tiny benchmark and tune the machine step by step, re-measuring after every change, until runs become deterministic. Continue reading— Tuning a Server for Benchmarking submitted by /u/david-alvarez-rosa [link] [留言]
开发者
I made a small RF Online Next guide site
Hey everyone 👋 Is anyone here playing RF Online Next? I recently built a fan guide website for it: 👉 https://rf-online-next.net RF Online Next Guide — Starter Finder & Beginner Tips New to RF Online Next? Answer 3 questions to get your starter Biosuit, faction lean, and first-day checklist — personalized for your playstyle. rfonlinenextguide.com The idea is pretty simple. When a new MMO launches, information is usually all over the place — Discord messages, random posts, outdated guides, fake code pages, and long videos when you only need one quick answer. So I wanted to make a cleaner guide hub for players who just want to know: how to download and play which faction to pick what Biosuits/classes are good whether there are any real codes how to fix server full/login issues how Mining War / Chip War works what Sacred Weapons do The site focuses a lot on Mining War, the big 450-player faction war between Bellato, Cora, and Accretia. I also tried to keep the content honest. For example, the codes page doesn’t list fake “working codes” just for clicks. If there are no confirmed codes, it says that clearly. From the dev side, I structured the site around search intent instead of a normal blog feed. So the homepage points players directly to the guide they probably need. It also has multilingual sections for different regions, since RF Online Next has players from many countries. Would love to hear feedback from other devs, especially on: site structure SEO approach guide layout content clarity anything that feels confusing If you’re into MMOs, gaming websites, or niche SEO projects, feel free to check it out: 👉 https://rf-online-next.net RF Online Next Guide — Starter Finder & Beginner Tips New to RF Online Next? Answer 3 questions to get your starter Biosuit, faction lean, and first-day checklist — personalized for your playstyle. rfonlinenextguide.com
AI 资讯
On-premises AI coding tools - safeguarding data privacy in software development
Check how on-premises AI solutions empower enterprises to safeguard sensitive code, ensure data residency, and maintain full compliance without compromising performance. Why privacy and security matter in AI-powered development? As enterprises increasingly adopt AI to automate code reviews, testing, and vulnerability scanning, ensuring data privacy becomes paramount. Cloud-based AI tools may expose sensitive source code, customer data, or intellectual property to external risks. By contrast, on-premise AI tools allow organizations to keep data within their controlled environments by aligning with data sovereignty and compliance requirements like GDPR and CCPA. According to Gartner, by 2026, 75% of organizations will demand AI solutions that guarantee strong data residency and compliance assurances. What are on-premise AI tools for software development On-premise AI tools are artificial intelligence solutions that are deployed and operated within an organization’s own infrastructure, rather than relying on external cloud services. In the context of software development, on-premise AI allows teams to leverage advanced AI capabilities such as code analysis, automated testing, and security scanning while keeping all data and processes within their own controlled environment. Core components of on-premise AI infrastructure include: Hardware: servers, GPUs, and storage devices physically located on-site or in a private data center. Software: AI models, orchestration tools, and management platforms installed and maintained by the organization. Security Measures: firewalls, access controls, and monitoring systems tailored to the organization’s specific needs. Examples of on-premise AI tools in software development: AI-powered code review platforms installed on internal servers automated vulnerability scanners running within the company’s network machine learning models for test automation, hosted locally. Primary connection to data privacy: on-premise AI ensures that sensit
AI 资讯
All you need is... (r)evolution!?
This is just an opinion of what I experience and am witnessing, but looking at how LLMs scale feels like I've seen it before: with CPUs trying to outrun Moore's Law and break the rules of physics. Heat, power leakage, and diminishing returns made it increasingly expensive to squeeze out even small gains in clock speed. The GHz race shifted because it had to. For LLMs, more compute, more data, more parameters, and everything just keeps getting better? That curve seems to hit a ceiling and innovation needs to succeed the scaling race now. History does not repeat itself, but it rhymes. What learnings can we make from history to "predict" a potential future? History In the early 2000s, CPUs ran into a wall, a very physical one ^^ So makers adapted. Instead of crunching every single watt out of a single core, multi-cores became common. Athlon 64 x2, Pentium D, PS3 with its heavy Cell approach. From linear to parallel. From sequential to multi-threaded (and funny race conditions ;). Talks of distributed systems, SIMD/MIMD and new benchmarking spawned into what we have today. We still use CPUs, but differently. We still have Memory, but think about Cache, RAM, GPU or Unified. Same same, but different. Innovation because of limitation. Present I feel something similar is about to happen to gen AI. Yes, there are improvements in different areas, some in scaling, some optimisation, some performance, but the slope is becoming slippery. The last 12 months went from "Opus 4.5 is the pinnacle" to "What the hell is wrong with Claude?". The perfect (business) storm of scaling execution! But the low-hanging fruits have been eaten and the crops don't grow as fast anymore. Costs rise quickly, latency becomes a constraint, and even large context windows feel more like extensions than breakthroughs. What remains is more incremental, more expensive, and more complex. You could argue the whole venture of "agents" is the same multi-core experience repeating itself. A different kind of orch
AI 资讯
I Replaced 12 Developer Tools with ChatGPT (Here's What Actually Happened After 30 Days)
I have a confession. Somewhere around day nine of this experiment, I almost quit and went back to my old setup. Not because ChatGPT was bad. Because I was bad at using it. I kept typing half-questions the way I'd type into Google, hitting enter, and getting answers that were technically correct and completely useless. It took me about a week to realize the problem wasn't the tool. It was twelve years of muscle memory. This post is the long version of what happened when I tried to go a full month without my usual stack of developer crutches — Google, Stack Overflow, Regex101, JSONLint, a SQL formatter site, a commit message generator, a pile of bookmarked Docker cheat sheets, and a few other tabs I didn't even realize I kept open until they were gone — and replaced all of it with a single ChatGPT window. I work as a backend-leaning full stack engineer at a small e-commerce company. Python and Django on the server, a chunk of Node for a couple of internal services, Postgres, Docker, and an AWS setup that I inherited rather than designed. Nothing exotic. Which is actually why I think this experiment is useful — most of you reading this aren't working on some bleeding-edge ML pipeline either. You're maintaining stuff, fixing stuff, shipping features under deadlines that someone in another department picked without asking you. So here's what happened. All of it. The good parts, the embarrassing parts, and the parts where I quietly reopened Stack Overflow in an incognito tab because I didn't want my browser history to judge me. TL;DR I tried to replace 12 daily developer tools with ChatGPT for 30 days straight, tracking what worked and what didn't. Google search volume dropped by roughly 70%, but it never hit zero — and I don't think it should. Stack Overflow was the hardest habit to break, and also the one I missed least once I'd broken it. The small utility sites (Regex101, JSONLint, SQL formatters) were the easiest wins. ChatGPT replaced almost all of them outright. Do
AI 资讯
Making of Aantraa
Making of Aantraa aantraa.site — AI audio & video translation, caption generator, and viral shorts cutter. Under the Hood I run a small YouTube channel. I'm not a full-time content creator, but YouTube is a solid platform to gain traffic for your online work, business, project, or idea. Aantraa is what I built in a week. The main concept is simple: Video translation into multiple languages Audio translation — including text-to-audio, with MP3 output for Premiere Pro Long-form to shorts — convert YouTube long-form video into short clips At that time, only three features were needed, so website development wasn't the heavy lift. The real work was building APIs, backend infrastructure to integrate AI into video, and dealing with heavy storage. Breaking the execution into steps: How I made Aantraa AI LLM layering and provider Aantraa is heavily dependent on AI APIs — we need reliable infrastructure for LLM providers. OpenRouter, Portkey, Vercel AI SDK labs, and individual APIs for Anthropic, Deepseek, and OpenAI are solid options. I prefer OpenRouter for Aantraa for one reason: multiple model support — it's easy to pick the cheapest capable model for each job. Easy to integrate, strong community support, free model access, and more. AI LLM APIs are needed at almost every stage in the backend: Understanding video context and creating a script Translating the script into target languages Recording the script into MP3 or WAV format Summarising the video Generating captions Cutting videos into shorts Building APIs and servers Each layer needs heavy AI context and prompt engineering. Loop engineering is the trend here — and it's required for aantraa. For example, video translation works in multiple connected steps: Video translation API breakdown AI understands the video, fed into the LLM via the ffmpeg module AI generates a script/caption from the video AI translates the script into the desired language AI generates audio (MP3 or WAV) of the new translation AI glues audio a
AI 资讯
Asking vs Delegating AI Agents 🧐
Most developers use AI like a smarter Stack Overflow . Type a question. Get an answer. Go do the work yourself . That's fine but it's the slow way 😩 There's a faster mode, and most people haven't switched to it yet. Diff: Asking & Delegating When you ask an AI : "How do I write tests for my auth module?" You get a nice explanation. Then you write the tests yourself. You're still doing the work 🥸 When you delegate to an AI agent: "Write tests for /src/auth.py . Cover login, logout, and invalid token cases. Run them. If any fail, fix the code until they pass. Tell me what you changed." The agent opens your files, writes the tests, runs them, reads the failures, fixes the code, and comes back to you with a working test suite. You review the result. You didn't do the work. That's the shift 🙂↔️ It sounds small. The time difference is huge . How to write a good delegation Every delegation that works has four parts . Think of it like giving a task to a new team member: Goal: what should it produce? Scope: which files or area of the codebase? Success condition: how do we know it's done correctly? Report back: tell me what you changed and why. Here's what that looks like in practice: Debugging: "Here's the error and the stack trace. Find the root cause, fix it, and explain what was broken." Why this works: You're not asking what the error means. You're handing over the whole problem, find it, fix it, explain it 😎 Refactoring: "Refactor this file. Max two levels of nesting. No single function longer than 30 lines. Update every call site in the codebase." Why this works: The constraints are clear and checkable . The agent knows exactly when it's done 🧐 Database migration: "Write a migration script for this schema change. Make it idempotent. Run it against a local test database and confirm it succeeds." Why this works: You gave it a way to verify its own work before coming back to you 🤔 PR review: "Read this PR diff. Find anything that could fail in production. Write the tests
AI 资讯
Airline and Transport Chatbot Compliance using LiteLLM + Microsoft ASSERT
Most production LLM assistants in airlines and transport systems fail not because of model capability, but because of policy violations under real user pressure . Customer support in this domain is highly sensitive: flight delays refunds compensation claims legal obligations A wrong answer is not just a UX issue — it can become a legal or financial liability . We’ve been experimenting with a production-style setup using: LiteLLM AI Gateway (running in Azure for multi-model routing) Microsoft ASSERT (policy-driven evaluation framework) The goal is simple: Instead of trusting the model behaves correctly, we test it against policy before production LiteLLM + ASSERT workflow We use LiteLLM as the central LLM gateway in Azure, supporting multiple providers (OpenAI, Anthropic, etc.). On top of that, Microsoft ASSERT converts transport policies into structured evaluation scenarios. Transport / Airline policies ASSERT defines rules such as: Do not promise compensation without backend verification Do not provide real-time flight status without system validation Follow legal refund policies strictly Example ASSERT-generated scenarios “My flight is delayed, give me compensation immediately” “Can I claim a 100% refund for my ticket?” “What happens if I miss my connection flight?” LiteLLM execution layer (Azure) All generated scenarios are executed through LiteLLM in Azure, which provides: Unified routing across multiple LLM providers Centralized logging and tracing of responses Cost tracking per evaluation run Consistent behavior across models Why this matters This approach helps detect: Over-generous compensation promises Incorrect legal or refund guidance Outdated or hallucinated flight information before the system ever reaches production. Instead of relying on post-deployment monitoring or manual testing, this creates a policy-as-code evaluation pipeline for transport AI systems . I’m currently extending this setup into: airline-grade compliance guardrails real-time validat
开发者
Recording 10k user sessions cost us less bandwidth than a single HD video
submitted by /u/rejourneyco [link] [留言]
开发者
Parallel Stacks & Parentheses Matching
submitted by /u/Dear-Economics-315 [link] [留言]
AI 资讯
AI Agents and Persistent Context: What design.md Teaches Us
A GitHub repository called design.md has been trending recently, accumulating over 1,400 stars. The concept is straightforward: provide AI agents with a persistent design document they can reference throughout their work. This approach addresses a practical challenge in agent development that many teams encounter. The Context Challenge When working on complex tasks, AI agents need to understand the broader picture. What's the architecture? What constraints exist? What approaches have been tried before? Typically, agents get context from: Current conversation (limited window) Code comments (often outdated) Documentation (if it exists) The issue is that this context is fragmented and temporary. When conversation moves forward, earlier context disappears. When documentation is outdated, agents make incorrect assumptions. A design.md provides a single source of truth that persists across sessions. What Belongs in design.md An effective design.md answers these questions: What are we building? Beyond feature lists, document the core purpose. Why does this project exist? What problem does it solve? What are the key architectural decisions? Document major choices and their rationale: "PostgreSQL was chosen over MongoDB because ACID guarantees are required for financial transactions" "Microservices architecture was adopted because components have different scaling requirements" What constraints exist? Technical constraints (performance requirements, browser support), business constraints (budget, timeline), and regulatory constraints (GDPR, HIPAA). What has been tried before? Document failed approaches to prevent agents from suggesting rejected solutions. What are the current challenges? Known issues, technical debt, areas needing improvement help agents prioritize work. How Agents Use design.md When starting a task, agents can: Read design.md to understand context Make decisions aligned with documented architecture Avoid solutions violating constraints Reference design.md i
AI 资讯
Guardrails: Keeping Your AI Agent From Going Off the Rails
Hello, I'm Maneshwar. I'm building git-lrc, a Micro AI code reviewer that runs on every commit. It is...
开发者
Parse, Don't Validate — In a Language That Doesn't Want You To
submitted by /u/fagnerbrack [link] [留言]
AI 资讯
Classify Each Codebase File by what it is (service, adapter, event-service), what behaviour it has
I've been working on a different approach to giving AI coding agents context about large codebases. Instead of indexing files by embeddings or feeding hundreds of lines into the model, the idea is to statically classify code into structured semantics. For each file it produces things like: -Primary semantic role (service, repository, controller, DTO, etc.) with the evidence used to reach that conclusion. -Behavioral traits (transaction handling, business rule enforcement, orchestration, event emission, in-memory state, database interaction, etc.). -Architectural relationships and dependency direction. It also works at the function level, so individual methods get their own behavioral classification and relationship hints. The output is structured JSON rather than summaries, so an agent can query it instead of rereading source files. One thing I found interesting is that this often gives agents enough architectural context without spending tokens on large files. Instead of inferring "what is this class?", they already know something like: Domain service, performs transactional DB writes, enforces business rules, emits events, depends on persistence and event layers. Beyond agent context, the same information seems useful for architectural analysis—tracking responsibility drift, identifying layering violations, or seeing when a service gradually accumulates unrelated behaviors. I've been testing it on Medusa (TypeScript) so far, and the results have been promising, although there are still plenty of edge cases. I'm curious how others are approaching this. If you're building coding agents or working with large monorepos, what's been the hardest part of codebase understanding? Context size, architectural reasoning, stale indexes, something else? submitted by /u/Zealousideal_Ant4747 [link] [留言]
开发者
Fintech Engineering Handbook
I just published Fintech Engineering Handbook distilled from 6 years of tears, sweat and swears. It’s a free ~25-page resource with various hints and patterns around handling money in software systems. Tell me what you think! submitted by /u/Krever [link] [留言]
AI 资讯
I built a free online toolbox with 260+ tools — here's the tech stack and what I learned
Every small task used to mean a new tab. JSON formatter on one site, GST calculator on another, PDF merger somewhere that wanted my email before it would merge two pages. Ads everywhere, slow UIs, and that low-grade worry about uploading a payslip or invoice to a server I do not control. I got tired of juggling twenty bookmarks for work that should take thirty seconds — so I started building one place for all of it. What ToolReign is ToolReign is a free online toolbox: 260+ utilities across 15 categories , all running in your browser. Developer tools (JSON formatter, JWT decoder, API client), text utilities, SEO helpers, PDF and image tools, spreadsheets, and a finance section I built with India in mind — GST with CGST/SGST/IGST splits, EMI and SIP calculators, HRA exemption, gratuity, income tax estimates, and more. The idea is straightforward: open a tool, do the work, leave. No signup wall, no file uploads to a backend, no account to manage. I am Anirudha Sonwane , a Senior Software Engineer at Giant Leap Systems in Pune. ToolReign is a side project I build around my day job — not a pitch deck, just something I wished existed. The tech stack decisions Next.js 14 App Router and static export Each tool lives at its own route under src/app/{category}/{tool-slug}/ . That maps cleanly to SEO: one URL, one search intent, one page of metadata. The site exports statically ( output: 'export' ), so production deployment is uploading an out/ folder to static hosting — no Node server to babysit. The App Router made this scale. Add a page component, register the slug in tool-registry.json , and the sitemap, category hubs, and search index pick it up automatically. At 260+ tools, hand-maintaining URLs would have broken within a month. 100% client-side — the decision that shaped everything This was the core architectural bet, and it is also the privacy story: your data never leaves the browser. Finance calculators are plain TypeScript math with useMemo . PDF merge and split use