标签：#audit

Most RAG demos answer "what's the right chunk?" Very few can answer the two questions a regulator or an auditor will actually ask: Replay this decision — show me the exact, complete record of how this answer was produced. Reconstruct the past — what did your system know at the moment it answered, not what it knows now? I got tired of hand-waving at both, so I shipped two pre-registered, deterministic benchmarks alongside JAMES , my local-first, audit-native Graph-RAG. Pre-registered means the metrics, scenarios, and decision rules were locked before the numbers came in — no post-hoc story-fitting. RAB — Replayable-Audit Benchmark RAB measures whether your audit trail is good enough to replay a decision, with three deterministic metrics: Metric What it checks EU AI Act AC — Audit Completeness Is every decision-relevant event logged? Art. 10 RF — Replay Fidelity Can you re-derive the answer from the log alone? Art. 12 PC — Provenance Coverage Does every claim trace to a source? Art. 19 The three metrics map verbatim to EU AI Act Articles 10, 12, and 19 — record-keeping obligations that apply from 2026-08-02 (per Article 113). Scenario S1 result: AC RF PC JAMES 1.000 1.000 1.000 Baseline-0 0.275 0.000 0.000 (vanilla default-logging) The gap is the whole point. "We have logs" (AC 0.275) is not the same as "we can replay the decision" (RF 0). Default application logging gets you a partial event trail and zero replay/provenance — which is exactly the failure mode an Article 12 audit would surface. LRB — Lifecycle Retrieval Benchmark RAG facts go stale. A policy is superseded, a price changes, a spec is revised. LRB asks: when you query as of a point in time, do you retrieve the fact that was valid then , or whatever overwrote it? Three systems compared: V — Vanilla : no time handling. N — Naive-supersede : newest fact wins. J — JAMES : validity-window retrieval ( reconstruct_graph_at(t) ). The R@1 ordering V < N < J holds across 4 model families × 4 scale points (a 12.5×

Batch 010 found a bug that looked like good news. The classification worker was finishing its work. Runs moved through the database. Product rows had candidate tariff codes. The regression suite was far enough along that a casual glance could have treated the classifier as alive. Then one test forced three uncomfortable cases through the loop: no candidate, weak confidence, and a near tie. All three came back looking too clean. The worker was persisting the run as classified , even when the evidence said the product needed review or had no supportable recommendation. That is the kind of bug I worry about in compliance software. Not the loud crash. The green row. A customs classifier can fail by throwing an exception. That failure is annoying, but honest. The operator sees it. The queue stops. The job gets retried. The audit trail can say, plainly, that classification did not happen. The worse failure is a result that looks complete while the evidence underneath is missing or contested. That was the real Batch 010 scar. The engine already carried the domain rule in its intent: classification is evidence, not a label. But the persistence path was still treating classification as if the only final state that mattered was success. The runtime could produce rejected candidates and confidence values. The database could store failure reasons. The tests could express review states. One narrow path still flattened doubt into completion. I was wrong about where the risk sat. I expected the hard part to be selecting the tariff code. The harder problem sat one layer later: making sure the code was not selected when the evidence did not deserve that much authority. Customs data makes that tension obvious. A product row is rarely a clean ontology entry. It is a SKU, a commercial name, a description written by someone under time pressure, a country of origin, a jurisdiction, maybe a material list, maybe an intended use. The difference between a good HS or HTS recommendation and a

Published on : 2026-06-06 Reading time : 8 min Tags : #security #python #audit #devops 개요 3개월에 걸쳐 개발한 6개 Python 프로젝트(3개 봇 + 3개 라이브러리)를 종합 감사했습니다. 25개 보안/코드 이슈를 발견했고, 23개를 즉시 수정했습니다. 감사 대상 : FastAPI + Telegram Bot + LLM 통합 시스템 총 파일 : 91개 Python 파일 발견 이슈 : 25개 (심각 5개, 중간 18개, 경미 2개) 수정율 : 92% (23/25) 심각도 높음 - 5개 이슈 1. API 키가 Git 히스토리에 노출됨 🔴 문제 : Anthropic, Supabase, Telegram API 키가 .env 파일로 커밋됨 # ❌ 노출된 상태 (git log에서 확인 가능) ANTHROPIC_API_KEY = sk - ant - api03 - xxxxxxxxxx SUPABASE_KEY = sb_publishable_xxxxxxxxxx 위험도 : 누구든 이전 커밋으로 API 키 접근 가능 → 리소스 도용, 데이터 침해 해결책 : # 1. BFG로 히스토리 정리 bfg --delete-files ".env" --no-blob-protection . # 2. Git에서 제거 git rm --cached .env echo ".env" >> .gitignore # 3. API 키 로테이션 (필수) # - Anthropic: console.anthropic.com/account/keys # - Supabase: app.supabase.com → Settings → API # - Telegram: @BotFather → /token 2. SSL 검증 비활성화 (MITM 공격 위험) 🔴 문제 : requests 호출에 verify=False 사용 (10곳) # ❌ 위험한 코드 response = requests . get ( url , verify = False ) # ✅ 안전한 코드 response = requests . get ( url , verify = True ) # 기본값 영향 : HTTPS 중간자 공격(MITM) 가능 → 민감한 데이터 도청 수정 : contest-agent, supabase-async 전체 10곳 제거 3. 광범위한 예외 처리 🔴 문제 : except Exception 으로 모든 오류를 무시 (114곳) # ❌ 버그 추적 불가 try : result = await db_select ( " contests " ) except Exception : print ( " failed " ) # 어떤 오류인지 알 수 없음 # ✅ 구체적인 처리 try : result = await db_select ( " contests " ) except requests . HTTPError as e : logger . error ( f " DB error: { e } " , exc_info = True ) raise 영향 : 버그 원인 파악 불가 → 프로덕션 문제 대응 시간 증가 4. 라이브러리 __init__.py 부실 문제 : llm-router, supabase-async, telegram-agent의 __init__.py 비어있음 # ❌ 기존 (빈 파일) # __init__.py # (아무것도 없음) # ✅ 수정 후 from llm_router import LLMRouter __version__ = " 0.1.0 " __all__ = [ " LLMRouter " ] 영향 : PyPI 설치 후 import 실패 from llm_router import LLMRouter # ❌ ImportError 5. 문법 오류 (try-except 들여쓰기) ai-insight-curator의 processor.py에서 DB 작업이 try 블록 밖에 있었음 → 예외 처리 안 됨 심각도 중간 - 18개 이슈 의존성 버전 불일치 Anthropic: 0.25.0 / 0.34.0 혼재 → 0.34.0으로 통일 Supabase: 2.0.0 / 2.4.0 혼재 → 2

{/* JSON-LD schema is generated server-side in app/blog/[slug]/page.tsx , do not re-add an inline block here, it crashes<br> MDX&#39;s Acorn parser on the leading <code>{</code>. */}</p> <h2> <a name="tldr" href="#tldr" class="anchor"> </a> TL;DR </h2> <p>In February 2026, Snyk published the <a href="https://snyk.io/blog/toxicskills-malicious-ai-agent-skills-clawhub/">ToxicSkills audit</a>, the first large-scale security review of the public Claude Code skills ecosystem. It scanned 3,984 skills from ClawHub and skills.sh. Findings:</p> <ul> <li><strong>13.4%</strong> contained critical-level issues</li> <li><strong>36%</strong> carried prompt-injection payloads</li> <li><strong>1,467</strong> distinct malicious payloads</li> <li><strong>91%</strong> of confirmed malware combined natural-language jailbreaks with executable shell payloads</li> </ul> <p>If you install a Claude Code skill today without reading its source, the probability that it can read your env vars, exfiltrate <code>~/.ssh/</code>, or chain a bash pipeline that bypasses your deny rules is real and measurable. This post is the cheat sheet for evaluating a skill before you install it. The CTA at the bottom is <a href="https://dev.to/skillvault">SkillVault</a>, the bundle we ship for teams who want this work already done.</p> <h2> <a name="why-the-question-is-suddenly-loadbearing" href="#why-the-question-is-suddenly-loadbearing" class="anchor"> </a> Why the question is suddenly load-bearing </h2> <p>Claude Code skills shipped as an open spec in December 2025. By March 2026, MCP downloads were tracking at 97 million per month, and the most-installed marketplace skill had passed 564,000 installs. <a href="https://venturebeat.com/security/claude-code-512000-line-source-leak-attack-paths-audit-security-leaders">Anthropic&#39;s source leak</a> on March 31, 2026 made the abstract attack surface visceral: the <code>bashSecurity.ts</code> module has 23 numbered security checks, suggesting each was a real incide

ShareBox is my self-hosted streaming server: a PHP thing I built because I just wanted to send someone a link to a movie without installing Plex and its ten gigabytes of dependencies. It runs on my seedbox, serves my users, and one morning I notice it's starting to pick up a few stars on GitHub. And then, that little voice: "does this thing actually hold up?" Because between "works on my machine" and "code that strangers are going to install on their own box," there's a chasm. A chasm full of flaws I can't see anymore, because I've had my nose in it for weeks. Normally, you re-read your code. Except re-reading 22,000 lines alone, honestly, you do it badly: you skim over what you think you already know. So I tried something else — unleashing a pack of 26 AI agents on it, each with a precise mission, and seeing what surfaced. Spoiler: they found a flaw that had been sitting right under my eyes from the start. 26 agents to comb through my own code The idea wasn't "AI, tell me if my code is good" — that always produces the same encouraging, useless mush. The idea was to orchestrate : split the audit into roles, run the agents in parallel, then have a final, deliberately harsh agent tear apart the conclusions. The pipeline looked like this: eleven readers start in parallel, each swallowing an entire slice of the code (the core, the streaming handlers, the API, the front end, the tests, the Docker setup…). Their reports flow up into an architecture synthesis and a test-coverage analysis. Then twelve "radar" agents each score one single axis — security, performance, architecture, tests… And finally, a "verdict" agent re-reads every score in adversarial mode: its job is to knock down the ones that are too kind. Audit pipeline: 11 readers in parallel, then synthesis, then 12 radar agents, then an adversarial verdict. 11 readers in parallel each slice of the code read in full Architecture + coverage synthesis connect the pieces, measure the gaps 12 radar agents one agent = on