How RAGScope Knows Which Chunks Your LLM Actually Used

How RAGScope Knows Which Chunks Your LLM Actually Used Your retriever fetched 10 chunks. Your LLM only used 3. RAGScope shows a precision score of 30 out of 100. The question every new user asks: how does it know? There is no OpenTelemetry attribute that says "this chunk was in the context window." RAGScope infers it — and the way it does this is the most consequential piece of engineering in the whole tool. There Is No "In Context" Attribute in OTel The OpenTelemetry semantic conventions for generative AI ( gen_ai.* ) define attributes for model, input/output tokens, and retrieved documents. They do not define anything like gen_ai.chunk.reached_llm or gen_ai.retrieval.used_document_ids . When your RETRIEVER span fires, you get a list of documents. When your LLM span fires, you get a prompt and a completion. The two spans are connected by a parent-child trace relationship — but there is no attribute that maps which retrieved documents appear in which prompt. This gap matters. A reranker might drop 7 of your 10 chunks. Your application code might apply a token budget and truncate 4 more. From the trace alone, you cannot tell. RAGScope needs this information to compute the precision sub-score — the highest-weighted metric at 40% of the overall score. Getting it wrong would make precision meaningless. The Substring Match — How assembleContext Works RAGScope's answer is in src/enrichment/pipeline.ts , in a function called assembleContext : function assembleContext ( chunks : RagChunk [], llmSpans : ParsedSpan []): RagChunk [] { const llmPrompts = llmSpans . map (( s ) => s . prompt ). filter (( p ): p is string => !! p ); if ( llmPrompts . length === 0 ) return chunks ; let position = 0 ; return chunks . map (( chunk ) => { if ( ! chunk . content ) return chunk ; const inContext = llmPrompts . some (( p ) => p . includes ( chunk . content ! )); if ( inContext ) { return { ... chunk , inContext : true , contextPosition : position ++ }; } return { ... chunk , inContext :