标签：#python

Python Redis: Caching and Fast Data Structures Redis is an in-memory data store used for caching, session storage, pub/sub messaging, leaderboards, rate limiting, and more. With redis-py 's async client, it integrates cleanly into any asyncio application. Installation pip install redis[hiredis] # hiredis is a C parser — 2-5× faster protocol parsing Connect and Verify import asyncio import redis.asyncio as aioredis from datetime import timedelta import json REDIS_URL = " redis://localhost:6379/0 " async def get_redis () -> aioredis . Redis : client = aioredis . from_url ( REDIS_URL , encoding = " utf-8 " , decode_responses = True , socket_connect_timeout = 5 , socket_timeout = 5 , retry_on_timeout = True , ) pong = await client . ping () print ( f " Redis connected: { pong } " ) return client Strings — Basic Cache with TTL async def cache_set ( r : aioredis . Redis , key : str , value : str , ttl : int = 300 ) -> None : await r . set ( key , value , ex = ttl ) async def cache_get ( r : aioredis . Redis , key : str ) -> str | None : return await r . get ( key ) # Cache-aside pattern async def get_user_profile ( r : aioredis . Redis , user_id : int , db ) -> dict : cache_key = f " user:profile: { user_id } " cached = await r . get ( cache_key ) if cached : print ( f " Cache HIT for user { user_id } " ) return json . loads ( cached ) print ( f " Cache MISS for user { user_id } — querying DB " ) user = await db . fetch_user ( user_id ) # your DB call if user : await r . set ( cache_key , json . dumps ( user ), ex = 600 ) return user or {} # Atomic counter async def increment_page_views ( r : aioredis . Redis , page : str ) -> int : key = f " views: { page } " count = await r . incr ( key ) await r . expire ( key , 86400 ) # reset counter after 24 h return count Hashes — Structured Objects async def save_session ( r : aioredis . Redis , session_id : str , data : dict , ttl : int = 3600 ) -> None : key = f " session: { session_id } " await r . hset ( key , mapping = data )

Builder Journal · ARC Prize 2026 There is a specific horror in a detective story when you realize the witness everyone trusted has been lying, or just wrong, the whole time, and every conclusion built on their testimony has to come down with them. I had that moment with my own notes this month. The unreliable witness was me. Context, if you are new to this thread : I'm competing in the ARC Prize 2026, building an agent that has to win games it has never seen. It had been stuck, underperforming on the hidden test in a way I could see on the scoreboard but could not explain, and I had been hunting the cause across several sessions. The two comforting facts In two earlier work sessions I had written down, as settled conclusions, two things about why the agent was failing. One: the failure was a kind that only happens on the hidden online games, so it could not be taken apart and studied on my own machine. Two: the practice games I did have were useless for investigating it anyway, because they scored a flat zero on the relevant measure. Notice what those two beliefs do when you put them together. They say, in a calm and reasonable voice, that there is nothing to be done here. The problem is unreachable, the practice data is a dead end, the smart move is to spend your energy elsewhere. They were not just facts. They were permission to stop looking. So I stopped looking. Twice. The hour that knocked it all down Eventually I made myself do the one thing I had been quietly avoiding. Instead of rereading my own notes for the third time, I went and checked. I wrote small probes and ran them against the real artifacts, the actual code and the actual game data, rather than against my memory of what they did. Both beliefs collapsed inside an hour. The failure was not unreachable. It came apart cleanly, deterministically, on the games I already had sitting on my disk. And the "dead end" practice data was not a dead end at all. It showed the problem plainly the moment I asked it

Module caching, execution order, and circular imports explained by tracing what actually happens. How Python's Import System Works and Why It Matters for Debugging The import system is one of the least understood parts of Python and one of the most practically important for debugging production issues. Circular import errors, unexpected code execution, and module state bugs all stem from not understanding what happens when Python encounters an import statement. What Happens on the First Import When Python executes import mymodule for the first time: Python checks sys.modules for mymodule . If found, returns the cached module object immediately. If not found, Python locates the module file. Python creates a new module object and adds it to sys.modules under the module name. Python executes the module file's code in the new module's namespace. The name mymodule in the importing module is bound to the module object. Step 3 happens before step 4. This is critical for understanding circular imports. The Module Cache import sys import os print ( " os " in sys . modules ) print ( sys . modules [ " os " ] is os ) Output: True True Every imported module is cached in sys.modules . Subsequent imports return the cached object without re-executing the module code. Module-Level Code Executes on Import # config.py print ( " config module loading " ) DEBUG = True print ( f " DEBUG is { DEBUG } " ) # main.py import config import config # second import print ( config . DEBUG ) Output: config module loading DEBUG is True True The print statements in config.py run exactly once — when the module is first imported. The second import returns the cached module object without re-executing the code. Circular Import Behavior # module_a.py print ( " loading module_a " ) from module_b import b_function def a_function (): return " from a " # module_b.py print ( " loading module_b " ) from module_a import a_function def b_function (): return " from b " When you import module_a , Python starts exe

I spent a chunk of last year around legacy modernization work — the kind of project where a bank or an insurer is taking twenty years of accumulated code and rebuilding it as modern services, one system at a time. Every one of those systems starts the same way: a PRD or a requirements document says what the business needs, that gets translated into a spec precise enough for an AI to implement, and eventually someone tests what came out. What struck me, watching this happen at scale, wasn't that the code was bad. It was that nobody was testing the thing that actually determined whether the code would be bad: the spec itself — the technical description handed to the model, not the PRD that motivated it. Every security tool I looked at — SAST scanners, DAST tools, even the AI coding assistants themselves — waited until an implementation existed before doing anything adversarial. Attack the code, once it's there. That's the whole industry's model, and it's worked fine for forty years because the volume was always survivable. A team ships a handful of PRs a week, a human reviews them, and eventually a pentest catches whatever slipped through. That math falls apart at modernization scale. When you're regenerating a few million lines of code, you're also generating a few thousand specs, faster than any review process was ever built to absorb. Testing after the fact doesn't just get slower under that load — it quietly stops happening, spec by spec, until the aggregate exposure is enormous and nobody can point to when it happened. So I built GAUNTLEX to test the thing that happens before the code does: the spec. This is also where I want to be precise about a word that gets overloaded. "Spec-driven development" — the broader industry shift toward writing structured, agent-facing specs instead of prompting an AI free-form — is exactly the world GAUNTLEX lives in. But a spec (what to build, precise enough for a model to implement) and a PRD or requirements doc (why it's needed

FROST周报 | 为什么智能体需要「谱系」？从生物学隐喻看AI治理新范式 作者按 ：本文是 FROST 开源项目的每日推广系列文章，周一深度篇。 一、一个被忽视的根本问题 当我们谈论 AI Agent 时，大多数讨论都聚焦于「能力」：能不能写代码？能不能调用工具？能不能规划任务？ 但有一个根本问题很少被触及： 当一个 Agent 执行了错误的决策时，谁来负责？当它消亡后，它的经验能否被传承？ 就像一个没有记忆的人，每次醒来都是白纸一张——这不叫智能体，这叫复读机。 FROST 正是为了解决这个「治理真空」而诞生的。 二、从细胞分裂到 Agent 家族 FROST 的核心哲学只有一句话： 细胞会死，但谱系会存续。Agent 会消亡，但宪法会传承。资产会永存。 这不是文学修辞，而是一套完整的技术架构。 四个原子：最小可行集合 FROST 只定义了四个原子，却能构建任意复杂度的智能体系统： 原子 职责 生物类比 Store 记忆容器，只做 save/load/delete 细胞核 Skill 纯能力单元，无状态无副作用 蛋白质 Agent 膜包裹的细胞，拥有 Store + Skills 神经细胞 SOP 有序步骤列表，可教学、校验、优化 宪法文本 from core import Store , Agent , skill_set , skill_get # 创建一个最小 Agent store = Store () agent = Agent ( " cell " , store , skills = { " set_context " : skill_set , " get_context " : skill_get }) # 执行任务 result = agent . run ( sop_steps = [ " set_context " , " get_context " ], initial_context = { " key " : " message " , " value " : " FROST is alive " } ) # result["_result"] == "FROST is alive" 关键洞察 ：Store、Skill、Agent、SOP 这四个概念彼此正交，可以自由组合。就像乐高积木，从简单到复杂，始终保持可解释性。 三、家族治理：超越扁平架构 传统的多 Agent 系统通常是扁平的：所有 Agent 平等对话，没有层级，没有记忆，没有责任边界。 FROST 引入了「家族治理模型」——一个三层递归结构： 祖辈 (Ancestor) ：定义不可违背的宪法与长期目标 父辈 (Parent) ：领域协调者，可递归委托 孙辈 (Leaf) ：执行具体原子任务，瞬态存在 四个协议保障治理闭环 ： 层级 Store 继承 ：祖先记忆只读，后代自动继承 SOP 宪法校验 ：祖辈审核后代 SOP，拒绝违规执行 编排层级限制 ： max_spawn_generation 硬编码，禁止越级 spawn 选择性持久化 ：父辈收割有价值产出，淘汰冗余 Agent 四、V5.0 五维元模型：多维治理架构 2026年7月发布的 V5.0 引入了一个重大升级—— 五维元模型 ： 维度 模块 核心职责 武器注册表 Armory 能力的元数据管理与发现 任务注册表 TaskRegistry DAG 任务编排与图谱 SOP 事件编目 EventCatalog + Strategist 态势感知与双模式事件分析 平台注册表 PlatformRegistry 外部能力的发现、调用与健康检查 规则注册表 RuleRegistry 可版本化的治理约束与合规检查 197 个测试用例 保障了每个维度的质量。 五、与现有框架的差异 维度 LangChain CrewAI FROST 状态管理 链式传递 角色记忆 层级 Store 权限边界 无 提示词软约束 代码强制只读 治理可审计 无 对话日志 结构化执行历史 架构无关 ✅ ✅ ✅ FROST 不重复造轮子。它填补的是「治理」这个空白地带： 让多智能体系统真正可控制、可追溯、可进化 。 六、快速体验 # 克隆仓库 git clone https://gitee.com/liao_liang_7514/frost.git cd frost # 运行测试 python -m pytest # 查看示例 python frost_run.py 完整文档： https://gitee.com/liao_liang_7514/frost 七、写在最后 AI Agent 的下一阶段，不是更强的模型，而是 更好的治理 。 当我们把 100 个 Agent 放在一起时，如果没有宪法、没有层级、没有记忆传承

I Tested Direct Provider APIs vs Aggregators — Here's the Truth Six months ago I was staring at a $48,000 invoice from an AI provider that shall not be named. We had committed to a six-month contract because the sales rep promised "priority routing" and "negotiated rates." What we got instead was a rate hike, an outage during our biggest product launch, and a support team that took 72 hours to respond. That was the moment I decided to stop signing contracts with AI providers entirely. This is the playbook I wish someone had handed me on day one — the architecture decisions, the math, and the code that lets a small team punch way above its weight class without betting the company on a single vendor. The Trap Most Startups Fall Into When I started my last company, I did what every founder does. I read the docs, got an API key, shipped a feature. The model worked, the demo went well, the investors nodded. Then we hit production traffic and the bills started arriving like clockwork. Here's what nobody tells you about going direct to a model provider as a startup: The pricing page you see on the website is the retail price. The actual cost of running production workloads includes rate limits you didn't anticipate, caching you forgot to implement, context windows that blow up your token count, and prompt engineering iterations that look cheap per call but compound fast. I watched one team burn $20K in a single weekend because they were streaming completions without setting a max_tokens guardrail. Direct providers also lock you into their ecosystem. Their SDK, their tools, their prompt format, their authentication scheme. The moment you want to A/B test a different model — which you will, probably next quarter — you're rewriting integration code instead of shipping features. And then there's the geopolitical mess. Some of the best models in 2026 come from providers that don't accept US credit cards. I've personally lost an afternoon trying to sign up for an account that re

Python is an excellent language for automating cloud infrastructure, but the official Kubernetes Python client leaves developers with an important architectural decision: Where should Kubernetes manifests live? Should they be constructed directly with Python objects? Embedded as multiline strings? Or stored as external files and rendered at runtime? Each approach works, but they have very different implications for readability, maintainability, and long-term operational cost. The key is recognizing that deployment logic and platform configuration evolve on different lifecycles. Your deployment code, the part that authenticates to Kubernetes, renders templates, and applies resources, may remain unchanged for months. Your manifests, however, often change weekly as applications evolve, resource limits are tuned, cloud-provider annotations are added, or networking requirements change. When those two concerns are tightly coupled, even a configuration, only change forces you to modify, test, and redeploy the delivery or application code itself. Over time, this increases maintenance costs, slows platform changes, and makes configuration drift and production mistakes more likely. This is a familiar software engineering principle: separate concerns that evolve independently. The same thinking that keeps application configuration separate from executable code also applies to Kubernetes manifests. Treating manifests as first-class configuration artifacts allows them to evolve independently from the Python code that delivers them. In this article we'll compare three ways of deploying Kubernetes resources with the official kubernetes-python-client , ranging from tightly coupled implementations to a design that cleanly separates deployment logic from platform configuration. The Landscape at a Glance The comparison below assumes a common application deployment scenario, where the desired state is largely known ahead of time. Controllers and Operators have fundamentally different r

"AI agent" gets thrown around so much I figured I should just build one instead of reading more threads about it. The core idea turned out to be small: you put Claude in a loop and hand it some tools. It picks a tool, you run it, you hand back the result, and it keeps going until it has an answer. Code is here if you want to skip ahead: claude-research-agent . What it can do Give it a task and it works out the steps on its own. Mine can: search the web (no API key for this part, it just hits DuckDuckGo's HTML page) open a URL and pull the readable text out do math without me trusting eval run read-only SQL against a SQLite file read local files, but only inside the project folder save findings to a notes file The loop is basically the whole thing Honestly this is most of it: messages = [{ " role " : " user " , " content " : user_message }] for _ in range ( MAX_STEPS ): response = client . messages . create ( model = " claude-sonnet-5 " , max_tokens = 2048 , tools = TOOL_SCHEMAS , messages = messages , ) messages . append ({ " role " : " assistant " , " content " : response . content }) if response . stop_reason != " tool_use " : return final_text ( response ) tool_results = [] for block in response . content : if block . type == " tool_use " : result = run_tool ( block . name , block . input ) tool_results . append ({ " type " : " tool_result " , " tool_use_id " : block . id , " content " : result , }) messages . append ({ " role " : " user " , " content " : tool_results }) The thing to watch is stop_reason . If Claude says tool_use , it wants you to run something. You run it, drop the result back into the conversation as a tool_result , and loop. When it stops asking for tools, you're done. The MAX_STEPS cap is just there so a confused agent can't spin forever. Tools are just functions Each tool is a Python function plus a little JSON schema telling Claude when to reach for it. Want a new capability? Write a function, add its schema. The loop never changes, which w

Hi guys, I've been working on Regdrift, my first open-source project. It's a CLI and GitHub Action that compares two CMSIS-SVD files to check whether there are any register-map changes that could affect firmware functionality. It catches changes such as moved registers, interrupt renumbering, access changes, and altered read/write behavior. It then classifies those changes as BREAKING , WARNING , or SAFE so the tool can act as a CI gate. I'm looking for feedback from people who maintain SVDs, HALs, PACs, SDKs, or firmware repositories. If possible, I'd like to test it against real old/new SVD pairs and learn where the classifications produce false positives, miss important changes, or are unclear. For people who work frequently with CMSIS-SVD files: which types of register-map changes are most detrimental to firmware or cause the most difficult code-related problems? Resources GitHub: https://github.com/Pranav-s79/regdrift Install pip install regdrift Usage regdrift check old.svd new.svd

skip에서 partition overwrite로: business_date 재처리를 Iceberg로 다시 표현하기 이전 글에서는 같은 source_hash 가 다시 들어왔을 때 기존 successful run을 재사용하는 idempotency를 다뤘다. 하지만 재처리에는 두 종류가 있다. 1. 같은 입력이 다시 들어온 경우 -> skip이 맞다. 2. 같은 business_date의 정정 입력이 들어온 경우 -> skip하면 안 된다. -> 같은 날짜의 gold 결과를 중복 없이 교체해야 한다. manufacturing-data-platform-mini 의 B5 slice는 두 번째 문제를 아주 작게 다룬다. 전체 Spark pipeline을 만든 것이 아니다. gold_daily_metrics Iceberg table 하나를 local Spark에서 만들고, business_date partition overwrite와 snapshot evidence만 검증했다. Scenario 이미 아래 gold row가 있다. business_date=2026-06-29 plant-a / line-1 / gearbox-a units_produced=120 defect_count=3 나중에 같은 business_date=2026-06-29 에 대한 정정 source가 들어온다. 운영자가 원하는 것은 append가 아니다. 원하지 않는 상태: 2026-06-29 old row 2026-06-29 corrected row -> 같은 날짜 결과가 중복됨 원하는 상태: 2026-06-29 corrected row만 남음 2026-06-30 같은 다른 날짜 partition은 그대로 유지됨 재처리 전후 snapshot evidence가 남음 그래서 이 slice의 질문은 이렇다. 같은 business_date의 정정 source를 처리할 때, gold table에서 해당 날짜 partition만 중복 없이 교체하고, 어떤 run이 어떤 Iceberg snapshot을 만들었는지 남길 수 있는가? Decision Pressure Slice1의 CSV pipeline은 already-successful source를 안전하게 skip할 수 있다. dataset_id + business_date + source_hash 이 key가 같으면 같은 입력이다. 다시 계산해도 같은 결과이므로 기존 run을 재사용한다. 하지만 source_hash 가 달라졌다면 의미가 다르다. same business_date different source_hash 이건 retry가 아니라 correction이다. CSV run-folder 방식에서는 새 run output을 만들 수는 있지만, "현재 gold table에서 해당 날짜를 원자적으로 교체한다"는 table-level 의미가 약하다. Iceberg를 붙이는 이유는 여기 있다. source_hash -> 같은 입력인지 판단하는 idempotency key business_date partition -> 정정 시 교체할 gold table 범위 snapshot_id -> table commit의 evidence 즉 Spark/Iceberg는 도구 이름을 추가하려고 붙인 것이 아니라, 재처리 상태 전이를 더 명확히 표현하기 위해 붙였다. Options Option 장점 문제 판단 same source면 항상 재계산 단순함 retry 때 불필요한 commit이 계속 생김 제외 corrected source를 append 구현 쉬움 같은 날짜 gold row가 중복될 수 있음 제외 whole-table overwrite 단순함 다른 날짜 partition까지 지울 위험 제외 business_date partition overwrite correction 범위가 명확함 Spark/Iceberg 설정과 test가 필요 선택 MERGE/upsert 강력함 이번 skeleton에 과함 backlog 이번 구현은 DataFrameWriterV2.overwritePartitions() 를 사용했다. corrected_d

wide CSV 여러 개를 EAV로 모아 gold mart 만들기 현실의 데이터 소스는 한 가지 모양으로 오지 않는다. 같은 의미의 값도 어떤 파일에서는 생산수량 , 다른 파일에서는 units , 또 다른 파일에서는 made 로 올 수 있다. 온도도 어떤 곳은 섭씨, 어떤 곳은 화씨일 수 있다. 이걸 매번 pipeline code에 if source == ... 로 박기 시작하면 source가 늘 때마다 코드가 지저분해진다. manufacturing-data-platform-mini 의 EAV mini slice는 이 문제를 작게 다룬다. 여러 wide CSV를 mapping config로 표준 attribute에 맞춘 뒤, EAV long format으로 모으고, 다시 gold metric mart로 pivot/aggregate한다. 데이터는 모두 synthetic이고, 회사 코드·고객 데이터·실제 schema는 쓰지 않았다. 1. Scenario 서로 다른 공장/라인/벤더에서 비슷한 제조 지표 파일이 들어온다. 예: plant_a.csv: 설비ID, 생산수량, 불량수, 온도C, 압력kPa plant_b.csv: machine_id, output_units, defects, temp_f, pressure_bar vendor_d.csv: unit_name, made, scrap, deg_c, kpa 비즈니스적으로는 같은 지표를 보고 싶다. units_produced defect_count temperature_c pressure_kpa 문제는 source마다 column name과 unit이 다르다는 점이다. 2. Decision Pressure 단순 구현은 source마다 코드를 늘린다. if source == "plant_a": 생산수량을 units_produced로 읽는다 if source == "plant_b": output_units를 units_produced로 읽는다 temp_f를 섭씨로 변환한다 if source == "vendor_d": made를 units_produced로 읽는다 이 방식은 작게는 빨라 보이지만 source가 늘수록 문제가 된다. 새 파일 형식마다 pipeline code를 고쳐야 한다. column mapping과 transform logic이 섞인다. unit conversion이 흩어진다. quality check가 source별로 갈라진다. gold mart grain을 설명하기 어려워진다. 그래서 mapping은 config로 빼고, pipeline은 표준 attribute를 처리하게 만들었다. 3. Options option result risk source별 hard-coded parser 처음엔 빠름 source가 늘 때 code change 반복 모든 source를 wide table 하나로 합치기 보기 쉬움 sparse/heterogeneous column 폭발 EAV long format 이종 attribute를 표준 형태로 모음 pivot/quality 설계가 필요 full mapping DSL/rules engine 유연함 mini project에는 과함 이 프로젝트의 선택은 단순한 JSON mapping + EAV long + gold pivot이다. 4. Decision 각 source는 JSON config로 자신의 column을 표준 attribute에 매핑한다. source column -> standard attribute output_units -> units_produced temp_f -> temperature_c with f_to_c pressure_bar -> pressure_kpa with bar_to_kpa pipeline 흐름: wide CSVs -> mapping configs -> EAV long rows -> gold entity_daily_metrics -> quality checks -> catalog/lineage EAV row의 핵심 shape: entity_id business_date attribute value v

schema drift를 fail이 아니라 warn으로 둔 이유 데이터 파이프라인에서 source schema가 바뀌는 순간은 애매하다. 무조건 무시하면 운영자는 입력 구조가 바뀐 사실을 모른다. 반대로 모든 schema 변화를 실패로 처리하면, 정상적인 컬럼 추가까지 daily run을 막아버린다. manufacturing-data-platform-mini 에서는 이 문제를 작게 다뤘다. synthetic manufacturing CSV의 실제 header를 기준으로 schema_hash 를 만들고, previous successful run과 비교해 달라졌으면 schema_drift quality check를 warn 으로 남긴다. 단, required column이 빠져 silver/gold contract를 만들 수 없는 경우는 현재 ValueError 로 빠르게 실패한다. 1. Scenario 어느 날 source CSV에 새 컬럼이 추가된다. 기존 header: event_time,plant_id,line_id,work_order_id,machine_id,product_code, operation,units_produced,defect_count,cycle_time_ms,business_date 새 header: event_time,plant_id,line_id,work_order_id,machine_id,product_code, operation,units_produced,defect_count,cycle_time_ms,business_date,operator_id operator_id 는 아직 silver/gold mart에서 쓰지 않는다. 하지만 source 구조가 바뀐 사실은 기록되어야 한다. 2. Decision Pressure schema drift에서 중요한 질문은 단순히 "바뀌었나?"가 아니다. 바뀐 것을 운영자가 알 수 있는가? 정상적인 컬럼 추가 때문에 pipeline을 멈춰야 하는가? downstream gold mart contract가 조용히 바뀌지는 않는가? 이전 successful run과 지금 run의 schema identity를 비교할 수 있는가? 초기 구현에서는 한 가지 실제 버그가 있었다. schema_hash 가 고정된 required column 목록에 너무 묶여 있어서, 추가 컬럼이 들어와도 hash가 바뀌지 않았다. 즉 operator_id 가 추가되어도 drift가 보이지 않았다. 이 문제를 고치기 위해 read_rows 가 실제 CSV header를 반환하고, 그 실제 header 기준으로 schema_hash 를 계산하도록 바꿨다. 3. Options option result risk ignore drift pipeline은 계속 돈다 source 변화가 보이지 않음 fail every drift 변화에 강하게 반응 정상적인 컬럼 추가도 막음 warn and continue 변화가 보이고 run도 계속됨 warning을 inspect해야 함 auto-evolve silver/gold 새 컬럼을 바로 사용 가능 downstream contract가 조용히 바뀔 수 있음 full schema registry production에 가까움 mini slice에는 무거움 이 프로젝트의 선택은 warn and continue 다. 4. Decision 현재 contract는 이렇다. previous successful run이 없으면: schema_drift = pass baseline schema established current schema_hash == previous successful schema_hash: schema_drift = pass current schema_hash != previous successful schema_hash: schema_drift = warn quality_passed는 true 유지 run/lineage record에 previous/current schema_hash 저장 required column missing: V

source_hash로 같은 입력 재처리를 안전하게 skip하기 작은 데이터 파이프라인도 한 번만 실행된다고 가정하면 금방 거짓말이 된다. 실제로는 같은 파일을 다시 실행할 수 있다. 실패한 run을 재시도할 수도 있고, 과거 날짜를 backfill할 수도 있고, 운영자가 실수로 같은 입력을 다시 넣을 수도 있다. 이때 결과가 중복되면 gold metric은 더 이상 믿을 수 없다. 이 글은 개인 포트폴리오 프로젝트 manufacturing-data-platform-mini 에서 source_hash 를 이용해 같은 입력 재처리를 안전하게 skip하도록 만든 작은 설계 판단을 정리한 글이다. 데이터는 모두 synthetic이며, production platform이 아니라 검증 가능한 mini slice다. 1. Scenario 같은 business_date 의 제조/로봇 이벤트 파일을 다시 처리해야 하는 상황이 있다. 예: retry: 앞 run이 중간에 실패해서 다시 실행한다. backfill: 과거 날짜를 다시 채운다. operator mistake: 같은 파일을 실수로 다시 실행한다. 단순히 매번 append하면 같은 날짜의 gold metric이 중복될 수 있다. 2. Decision Pressure 단순 CSV pipeline은 보통 이렇게 끝난다. CSV 읽기 -> silver 만들기 -> gold 집계 -> 결과 저장 하지만 운영 관점에서는 질문이 생긴다. 이 입력은 전에 처리한 파일과 같은가? 같은 파일을 다시 돌리면 중복 output이 생기나? 다른 파일로 같은 날짜를 다시 돌리면 어떻게 구분하나? 어떤 run이 어떤 source에서 만들어졌나? 그래서 재실행을 판단할 identity가 필요했다. 3. Options option result problem always append 모든 run 결과를 계속 추가 같은 입력 재실행 시 중복 always overwrite 결과를 항상 덮어씀 이전 결과/원인 추적이 약함 skip by business_date only 같은 날짜면 무조건 skip 정정 파일을 반영할 수 없음 skip by dataset_id + business_date + source_hash 같은 입력만 no-op 정정 파일은 새 run으로 처리 가능 이 프로젝트의 Slice1은 마지막 선택지를 쓴다. 4. Decision 현재 mini pipeline은 입력 파일의 content hash를 source_hash 로 계산한다. idempotency key: dataset_id + business_date + source_hash 이미 성공한 run이 있으면 새로 처리하지 않고 기존 run을 재사용한다. same dataset_id same business_date same source_hash prior successful run exists => status = skipped 이 선택은 작지만 중요하다. 같은 파일 재실행: skip -> 중복 없음 같은 날짜의 정정 파일: source_hash가 다름 -> skip하지 않고 새 run으로 처리 가능 단, 여기서 조심해야 할 경계가 있다. Slice1은 다른 source_hash 를 새 run으로 처리할 수 있지만, 이전 gold partition을 원자적으로 교체하는 Iceberg-style overwrite까지 구현한 것은 아니다. 그 문제는 다음 Slice2의 business_date partition overwrite 주제다. 5. Evidence 관련 코드와 검증 evidence: src/manufacturing_data_platform/pipeline/lakehouse.py tests/test_lakehouse_pipeline.py VERIFICATION_LOG.md README.md 검증 로그: 2026-07-08 publication readiness check: pytest: 33 passed lakehouse JSON CLI: passed, status=processed, quality_passed=true EAV JS

GitHub热门项目 | Zapret (Запрет обход блокировки Дискорда и Ютуба) | Stars: 1,335 | 18 stars today | 语言: Python

GitHub热门项目 | 100+ AI Agent & RAG apps you can actually run — clone, customize, ship. | Stars: 118,237 | 549 stars today | 语言: Python

GitHub热门项目 | Prefect is a workflow orchestration framework for building resilient data pipelines in Python. | Stars: 23,014 | 55 stars today | 语言: Python

I built a voice assistant that controls 45 AI tools. I say "Hey Jarvis" and it executes. What It Does Command Action "generate content" Creates YouTube scripts for 9 channels "research quantum computing" Deep research via Tavily + AI "write email about meeting" Drafts email, copies to clipboard "start focus" Starts Pomodoro + blocks apps "code review" Reviews git diff with AI "summarize" Summarizes clipboard content "find file tax PDF" Natural language file search Architecture Mic → Whisper (offline) → Intent Classify → Router → Ollama → say (TTS) Key Features Offline speech (Whisper local) Wake word: "Hey Jarvis" Global hotkey: Ctrl+Space Command chaining: "research AI then write blog" Memory across conversations Hindi + English Setup brew install portaudio pip install SpeechRecognition pyaudio openai-whisper python voice_commander_pro.py 🔗 github.com/amrendramishra/ai-tools 🌐 amrendranmishra.dev

Checkpoint-skip gate: a multi-agent pipeline can finish with task_success: true while the mandatory confirmation checkpoint never ran. checkpoint_skip_gate.py replays a recorded JSONL trajectory against a declarative spec of mandatory checkpoints and handoff contracts, offline, and blocks when the road was wrong. The verdict never consults the final metric. That is the point. AI disclosure: I wrote checkpoint_skip_gate.py with an AI assistant and ran it myself, offline, on Python 3.13.5, standard library only, no network. Every number, exit code, and hash in the output blocks below is pasted from a real local run. I ran each scenario twice to confirm STDOUT is byte-for-byte identical, and the tool prints a sha256 of its own report so you can reproduce the exact bytes. The Alberta write-up and the arXiv paper I cite are other people's work, attributed inline, and their numbers stay out of my fixtures. In short: task_success=true proves the pipeline arrived. It does not prove the mandatory steps happened, happened in order, or that each agent-to-agent handoff delivered what the next agent assumed. A trajectory can be perfectly green and structurally wrong. The gate replays a recorded trajectory against a spec you declare: checkpoints that must precede specific actions, plus contracts for each handoff (required fields, verified flags). The final metric is printed for contrast and ignored for the verdict. The demo that matters: two trajectories identical except one JSONL line, the confirm_with_user checkpoint event. Both end task_success: true . Delete that line and the verdict flips from PASS exit 0 to BLOCK exit 1 checkpoint-skipped . It also tracks unverified values across handoffs. A number that travelled a connected chain of two handoffs with no hop verifying it blocks as unverified-claim-propagated-2-hops . Everyone shared the number. Nobody verified it. Offline, keyless, zero network, fail-closed: broken input exits 2, never a silent green. The whole 8-fixture sw

周日慢读：如果细胞会写日记——FROST家族的记忆传承 作者 ：FROST Team 日期 ：2026-07-12 主题 ：轻量科普 | 周日轮换 阅读时间 ：5分钟 一封来自细胞的日记 想象一下，如果你是一个细胞，有一天你突然有了自我意识，会发生什么？ 2026年7月12日 晴 今天是我诞生的第0天。 细胞核对我说："这是你的记忆存储区， 所有的经验都必须记录在这里。" 我第一次理解了什么叫"生而有根"。 这不是科幻小说。这是一段真实的代码注释，出自FROST——一个用Python写成的AI Agent家族。 为什么Agent需要"记忆"？ 大多数Agent框架都在解决一个问题： "Agent能做什么" 。 搜索Agent能搜索、写作Agent能写作、代码Agent能写代码。打开框架，创建实例，调用方法，任务完成。 但FROST问了一个不同的问题： 当Agent完成一个任务后，它学到了什么？ 这不是哲学问题。这是工程问题。 类比：人类 vs Agent 的记忆 人类 Agent FROST的解决方案 记忆存储在大脑 记忆存储在Store Store 原子 记忆需要整理归档 记忆需要结构化 Lineage 族谱 师徒传承经验 Agent继承父辈能力 代际继承协议 忘记教训会重复犯错 没有记忆会重复失败 历史可追溯 人类的记忆是分散的、模糊的、容易遗忘的。 Agent的记忆可以是精确的、可查询的、永不丢失的。 关键是 设计好存储结构 。 一段代码：Store原子 FROST的Store是记忆存储的最小单元。它的设计哲学是 简单到极致 ： class Store : """ FROST的Store：记忆存储的原子单元 只有三个操作： - save(key, value): 存入记忆 - load(key): 取出记忆 - delete(key): 删除记忆 简单到极致，但足够强大。 因为记忆的本质就是 " 存取 " 。 """ def __init__ ( self ): self . _memory = {} def save ( self , key : str , value : any ) -> None : """ 存入记忆 """ self . _memory [ key ] = value print ( f " 💾 记忆已存储: { key } " ) def load ( self , key : str ) -> any : """ 取出记忆 """ value = self . _memory . get ( key , None ) if value : print ( f " 📖 读取记忆: { key } " ) else : print ( f " ❓ 记忆不存在: { key } " ) return value def delete ( self , key : str ) -> None : """ 删除记忆 """ if key in self . _memory : del self . _memory [ key ] print ( f " 🗑️ 记忆已删除: { key } " ) # 使用示例 store = Store () store . save ( " 用户偏好 " , " 喜欢简洁的回复 " ) store . save ( " 对话历史 " , " 讨论了Agent的记忆问题 " ) store . load ( " 用户偏好 " ) # → "喜欢简洁的回复" 三个方法，解决Agent的记忆问题。 族谱：记忆的传承 单个Agent的记忆只是"点"。族谱把记忆连成"线"。 在FROST中，每个Agent都有自己的"父辈"： ┌─────────────┐ │ 祖辈Store │ ← 家族宪法，不可篡改 │ (根节点) │ └──────┬──────┘ │ 继承 ┌───────────────┼───────────────┐ ▼ ▼ ▼ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ 父辈Agent │ │ 父辈Agent │ │ 父辈Agent │ │ (Branch A) │ │ (Branch B) │ │ (Branch C) │ └──────┬──────┘ └──────┬──────┘ └──────┬──────┘ │ 继承 │ 继承 │ 继承 ▼ ▼ ▼ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ 孙辈Agent │ │ 孙辈Agent │ │ 孙辈Agent │ │ (执行任务) │ │ (执行任务) │ │ (执行任务) │ └──────

Detecta si tu modelo de materiales hace trampa con la "huella bibliográfica" Un modelo de ML puede predecir la propiedad de un material sin entender la química: basta con que "aprenda" qué autores, revistas o años suelen ir con cada resultado. Esta herramienta aplica el test de falsificación de Clever Materials para descubrirlo. El problema: cuando el modelo lee el membrete, no la ciencia Imagina que entrenas un modelo para predecir si un material es estable. El modelo no mira la química: descubre que los artículos del grupo X (publicados en la revista Y, en torno al año Z) casi siempre reportan "estable". Así que aprende a clasificar por el membrete bibliográfico , no por la estructura. Funciona en el papel y se rompe en la práctica. A esto se le llama confounding bibliográfico (o leakage por metadata). No es un error de código: es una señal espuria que el modelo aprovecha. El paper Clever Materials (Jablonka et al., 2026) mostró que este patrón está generalizado en cinco tareas reales de materials science. Qué hace la herramienta materials-confounding-check es una CLI ( mcc check ) que corre cuatro sub-tests de falsificación sobre tu dataset (descriptores químicos + metadata bibliográfica + propiedad objetivo): Clasificador de metadata — ¿se puede predecir la bibliografía (autor/revista/año) a partir de los descriptores químicos? Si es above-chance , hay una señal bibliográfica presente. Huella bibliográfica — ¿un modelo que usa solo la metadata predicha se acerca al modelo con descriptores? Entonces el dataset no descarta hacer "trampa" por bibliografía. Split por grupo/tiempo — ¿colapsa el rendimiento si separas por autor/año en vez de al azar? Veredicto — un score low / medium / high de riesgo de confounding. El rigor que exige el test (para especialistas) El punto delicado de cualquier "test de significancia" es fijar el umbral a mano. Si ajustas el margen hasta que tu fixture pase, el test no prueba nada: es el anti-patrón Clever-Hans que el propio proyecto d