标签：#rag

RAG SOTA, Agent Harnessing, and Langfuse Observability for AI Frameworks Today's Highlights Today's top stories delve into optimizing RAG performance with open-source benchmarks, designing robust AI agent systems, and implementing best practices for LLM observability in production. RAG SOTA: I Tested 7 Pipelines and Built SEQUOIA (Open Source) (Dev.to Top) Source: https://dev.to/__2ddbae6bb7d/--5cec This article presents a comprehensive benchmark of seven Retrieval-Augmented Generation (RAG) pipelines, culminating in the development and open-sourcing of SEQUOIA, a new RAG system. The author details over 20 hours of compute time spent locally to rigorously test different RAG configurations against real-world tasks, providing valuable insights into their performance characteristics. The technical deep dive includes discussions on various components like chunking strategies, embedding models, vector databases, and re-rankers, along with their impact on retrieval quality and generation coherence. Readers gain an understanding of the trade-offs involved in designing effective RAG systems and the empirical evidence supporting different architectural choices. The release of SEQUOIA as an open-source project means developers can directly implement and experiment with a battle-tested RAG pipeline, offering a tangible starting point for their own projects. Comment: This is an invaluable resource for anyone building RAG. Benchmarking 7 pipelines and open-sourcing a well-performing one provides immediate practical value and a solid foundation for further experimentation. Stop Upgrading the Model. Start Engineering the Harness. (Dev.to Top) Source: https://dev.to/tacoda/stop-upgrading-the-model-start-engineering-the-harness-194 This insightful article argues that instead of solely focusing on larger or "better" base models, teams should invest in "engineering the harness" around their AI agents to improve performance. The author highlights that the supporting architecture—compri

RAG SOTA: I Tested 7 Pipelines and Built SEQUOIA (Open Source) After 20+ hours of compute time on local hardware, I benchmarked 7 RAG configurations against real-world tasks. SEQUOIA (RAPTOR tree + step-back prompting) consistently outperformed alternatives. The Full Pipeline List Method Core Approach No-RAG Direct LLM generation Classical RAG Dense retrieval (BGE-small + FAISS) Hybrid RAG BM25 + Dense + RRF + reranker LightRAG Key-value graph + dense hybrid PageIndex Two-stage hierarchical retrieval GraphRAG Entity graph + dense fallback Agentic RAG Multi-step reasoning pipeline SEQUOIA RAPTOR tree + step-back prompting SEQUOIA Pro Multi-query + rerank + compression Why LightRAG Underperformed The hype suggested graph-based RAG would revolutionize retrieval. On real banking documents and technical manuals: Graph construction is expensive (entity extraction, relationship mapping) Retrieval quality did not justify the overhead Academic benchmarks do not equal production reality Why RAPTOR Works Recursive Abstractive Processing for Tree-Organized Retrieval: Cluster leaf nodes (individual chunks) Summarize upward (hierarchical abstraction) Retrieve at multiple levels (specific details + high-level context) This mirrors how humans organize knowledge. Step-Back Prompting: Free Performance Before retrieving, generalize the query: User asks: "What's the error rate for Q3?" Step-back: "What metrics are tracked quarterly?" Retrieve broader context first, then narrow Result: ~15% improvement in recall. Zero latency cost. SEQUOIA Architecture User Query Step-back Prompting (generalize) RAPTOR Tree Retrieval (multi-level) Context Compression (summarize long contexts) Re-ranking (cross-encoder) Local LLM Generation Local LLM Evaluation I used a local model weaker than GPT-4 for judging. Key finding: relative rankings between methods stayed consistent even with a weaker evaluator. You can prototype and compare approaches without burning API credits on GPT-4 evaluations. Productio

Problem Statement We have a misinformation problem. But more specifically, we have a speed problem. A journalist spots a suspicious claim. They search for sources. Cross-reference databases. Call experts. Write a verdict. Get it edited. Publish, maybe 6 hours later. Maybe 3 days later. Meanwhile, the original claim has been screenshot, reposted, quoted in newsletters, and cited in arguments across five platforms. I wanted to build something that closed that gap. Not a chatbot that guesses. A proper pipeline, one that retrieves real evidence, reasons from it, and tells you why it reached a verdict. That's what Sift is. What is Sift? Sift (Source Inspection & Fact-checking Tool) is an open-source multi-agent AI pipeline that takes any text, extracts every factual claim, retrieves grounded evidence, and returns auditable verdicts — TRUE, FALSE, or UNCERTAIN, with cited sources and full reasoning chains. Paste a news article. A politician's speech. A viral statistic. A WhatsApp forward. Sift breaks it into individual claims and fact-checks each one independently. Why Multi-Agent? The naive approach is to ask an LLM: "Is this claim true?" The problem: LLMs hallucinate. They have knowledge cutoffs. They're confidently wrong in ways that are hard to detect. And critically, they don't show their work. A single LLM call can't reliably handle the full pipeline of: Extracting structured claims from noisy text Retrieving dated, traceable evidence from live sources Reasoning across conflicting evidence without confabulating Adversarially reviewing its own conclusions for overconfidence Finding corrections when something is wrong Each of these is a distinct task that benefits from its own prompt, its own tools, and its own failure modes. That's why I built five separate agents, orchestrated with LangGraph. The 5-Agent Pipeline Agent 1 — Claim Extractor A single paragraph can contain 4-5 distinct factual claims. Generic LLMs miss them or conflate them. This agent uses LLaMA 3.3 70

I built a bot trained on my own X bookmarks and likes. Around 50,000 of them, accumulated over years...