标签：#engineering

Everyone talks about prompts memory RAG But production issues were actually loops false completion replay retries wrong tool non deterministic execution Here are the top 7. That's what led us to build Failproof AI not because we wanted another framework, but because we kept seeing the same reliability problems across every framework.

i've been that person. standing in front of leadership with an 18-month architecture diagram, explaining why we need six months of infrastructure before a user touches a single feature. and it made sense. for 25 years it made sense. writing boilerplate was expensive. every feature came with a tax — database migrations, routing config, auth wiring. build a shared platform first, pay that tax once. the roadmap justified the investment. then i saw a stat that wouldn't leave me alone. roughly 60% of features on a six-month roadmap are obsolete by launch. not slightly off. obsolete. the customer's problem shifted. the market moved. you spent six months building a precise answer to a question nobody asks anymore. the longer you invest before showing something real, the more expensive it is to admit you were wrong. so you don't. you ship the wrong thing and call it "on schedule." i've done it. i've watched it happen. AI didn't create this problem. but agents are making it impossible to ignore. the 82-point gap mckinsey's 2025 survey: 88% of organizations use AI. only 6% see real bottom-line impact. that 82-point gap isn't about tools. everyone has the same tools. but something shifted in their may 2026 report. they describe agents working overnight — enriching requirements, generating code, packaging outputs for morning review. they call it the "24-hour sprint." leading organizations see 3-5x productivity with 60% smaller teams. a product owner logs in at 9am and finds a feature went from requirements to tested code overnight. nobody worked late. agents did. that's not autocomplete. that's a different delivery model. and here's what most teams miss: it only works when the work is small, bounded, and complete. agents need to know where a task starts and ends. horizontal platform architectures don't give them that. the codebase is the prompt jeremy d. miller built wolverine for .NET. in june 2026 he wrote: "the structure of your codebase is now, effectively, part of the prom

The cohesion paper series is now published in full — five papers that build a chain from the concept of cohesion to the Independent Variation Principle (IVP) . The chain: On the Nature of Cohesion — defines cohesion as a $2k$-tuple: for $k$ partitioning rules, $k$ (purity, completeness) pairs. Proves the knowledge-embodiment theorem: maximal cohesion under a rule coincides with exact knowledge embodiment under that rule. Shows that every published algorithmic cohesion metric measures a structural proxy (method-call overlap, shared-field density), not cohesion as defined by a principle. DOI: 10.5281/zenodo.20785752 Causal Cohesion — instantiates the schema under one concrete rule — change-driver-assignment identity: elements belong together iff $\Gamma(e_1) = \Gamma(e_2)$. Develops the metric $H_\text{causal}(M) = (\text{purity}(M), \text{completeness}(M))$, a two-dimensional score that fills one slot of the $2k$-tuple. DOI: 10.5281/zenodo.20785881 Four Necessary Conditions for Optimal Modularization — from the schema plus the objective of minimizing change propagation, proves four conditions — Admissibility, Element Form, Separation, Unification — are necessary and jointly exhaustive, uniquely pinning the $\Gamma$-equality partition $E / \tilde{\Gamma}$. DOI: 10.5281/zenodo.21362420 Why Minimizing Change Propagation Minimizes Maintenance Cost — decomposes total maintenance cost into access, alignment, cognitive, and domain-fixed components. Proves that minimizing change propagation cost is equivalent to minimizing total maintenance cost under an explicit coefficient condition, justifying the objective paper 5 assumed. DOI: 10.5281/zenodo.21362542 The Independent Variation Principle — synthesizes the chain into a single structural principle and examines the premises (change drivers, functional model, change isolation), preconditions (driver independence, decisional autonomy), and scope boundary. DOI: 10.5281/zenodo.21362618 Two derivations Last month's preprint — Der

Meta recently open-sourced Brain2Qwerty v2, a noninvasive Brain–Computer Interface (BCI) that can decode sentences from thoughts using electroencephalography (EEG) or magnetoencephalography (MEG) signals from the brain. In evaluations, the system achieved a word accuracy rate 61% on average, compared to 8% for other non-invasive methods. By Anthony Alford

Drawing from the enduring adaptability of HTML and HTTP, Seph Gentle proposes embedding self-contained schemas directly into file headers, ensuring data remains readable without external definitions. His experimental format prioritises forward, backwards, and sideways compatibility, enabling data format evolution without central coordination or data loss By Olimpiu Pop

TL;DR Traditional SaaS streaming requires exporting sensitive data to a vendor cloud, creating security risks and egress costs. BYOC reverses this model by running the data plane inside the customer’s cloud while the vendor manages the control plane. This keeps data within the enterprise perimeter while still providing a managed platform. Condense builds on this model with AI-driven automation, unified monitoring, and marketplace deployment, enabling private, compliant, and cost-efficient real-time data streaming. The enterprise data landscape is currently defined by a conflict between real-time AI data streaming utility and the strict requirements of data sovereignty . For years, the standard SaaS model forced a compromise. To access premium analytics, companies had to export sensitive telemetry to a vendor cloud. This created massive cloud egress costs and introduced significant security vulnerabilities. Bring Your Own Cloud (BYOC) for data streaming platforms has emerged as the professional solution to this dilemma. It allows a business to keep data within its own perimeter while benefiting from a fully managed, high-performance ecosystem. The BYOC Architecture: Privacy by Design An experienced analyst views BYOC as a clean separation of concerns. The architecture splits the environment into two distinct layers to ensure raw data never leaves the authorized environment. SaaS Control Plane: This is the management layer hosted by the provider. It handles the brain of the operation. It manages orchestration, user access, and pipeline configuration without ever seeing the actual data packets. Private Data Plane: This is the muscle. The managed Kafka clusters , Kubernetes (K8s) nodes, and storage engines like ClickHouse live inside the customer Virtual Private Cloud (VPC) . By keeping the data plane inside the customer perimeter, telemetry collection remains private. This architecture is the most direct path to satisfying internal security audits and global regulatory

Here is a reliable test: find a prompt that isn't working. Read it carefully. Now ask yourself — at which specific sentence did the model get permission to do what it did wrong? You will almost always find it. A hedged instruction. A missing constraint. An ambiguous scope. The model did not misunderstand you — it followed the most statistically probable interpretation of what you wrote. That interpretation was not the one you intended. These are not beginner mistakes. They are structural patterns that reappear at every experience level, because they look reasonable when you write them and only reveal themselves in the output. TL;DR: Prompts fail because they hand interpretive control to the model on dimensions where you had a specific requirement. Each of the seven mistakes below is a different way of doing that — and each has a specific, testable fix. Mistake 1: Placing Critical Instructions in the Middle of the Prompt Language models process all tokens simultaneously through attention mechanisms , but the effective weight any individual token receives depends heavily on its position. Instructions near the beginning and end of a prompt receive disproportionately more attention weight than those in the middle. This is not a quirk — it is a consequence of how positional embeddings interact with self-attention across long contexts. This effect is well-documented. The "Lost in the Middle" study (Stanford / UC Berkeley, 2023) showed that retrieval accuracy from long-context windows degrades significantly for information placed in the middle — even in capable models. The same mechanism applies to instruction prompts: GPT-4o and Claude 3.5 Sonnet both exhibit measurably lower constraint adherence for instructions buried mid-context compared to those at the leading or trailing position. Open-weight models including DeepSeek-V3 and Llama 3 display the same positional bias — this is not a proprietary model quirk, it is a structural property of the transformer architecture. T

Leave a carton of milk on the counter and it spoils in a day. Put the same carton in a refrigerator and it lasts a week or more. Nothing about the milk has changed — the same bacteria, the same enzymes, the same chemistry. What changed is temperature, and temperature does not nudge reaction rates gently. It controls them with an exponential lever. A swing of just a few degrees can stretch shelf life from hours to days. This article explains the equation behind that lever — the Arrhenius equation — what each term means physically, how to use it to compare rates at two temperatures, and the mistakes that quietly corrupt activation-energy estimates. Why this calculation matters Almost any process that involves chemistry running over time depends on the temperature-rate relationship. Food spoilage, drug degradation, battery aging, polymer curing, corrosion, and the cracking reactions in a refinery all speed up or slow down with temperature in the same exponential way. Engineers who design accelerated life tests rely on it directly: they run a product hot for weeks to predict how it behaves cold for years. The reason a quantitative model is essential is that intuition fails here. A linear guess — "twice as hot, twice as fast" — is badly wrong. Reaction rate climbs far faster than temperature does, and how much faster depends on the activation energy of the specific reaction. Without the Arrhenius equation you cannot convert an oven-shelf test into a real-world prediction, and you cannot tell whether a 5 C process drift matters or not. The core formula Svante Arrhenius proposed the relationship in 1889, building on earlier work by van 't Hoff. It states that the rate constant k of a reaction depends on temperature as: k = A * exp( -Ea / (R * T) ) Here A is the frequency factor (sometimes called the pre-exponential factor), Ea is the activation energy in J/mol, R is the universal gas constant 8.314 J/mol K, and T is the absolute temperature in kelvin. The physical picture

A panel on data ownership challenged the definition of "ownership," arguing it must extend beyond simple account control to include structural independence, interoperability, and community governance. Speakers like Zenna Fiscella, Paul Frazee, Boris Mann, and Robin Berjon emphasised the need for shared standards, unbundled platforms, and better tools to support user sovereignty. By Olimpiu Pop

DoorDash details the architecture behind Ask DoorDash, its AI-powered conversational shopping assistant, combining LLMs, specialized AI agents, MCP-based tooling, and an intelligence layer with persistent consumer memory and live backend data. Early results show up to 24% higher checkout conversion, 17% larger baskets, and improved intent accuracy using memory-backed sessions. By Leela Kumili

If you ask me what part of the system I protect the most, the answer is the database. I've been writing software alone for twenty-four years, and across every platform I've built, the rule has stayed the same: the web servers can take whatever you throw at them, the batches can be rebuilt, but the database has to stay idle on purpose. Not because I love idle databases, but because the day a database actually starts to struggle is a day with very few good options. This article is about what "keep the database idle on purpose" actually means in practice, and about one particular kind of graph that, in my experience, almost nobody is watching. The three layers and what each of them gets I think of a production system as having three tiers, and each tier gets a different rule. The web server tier can be horizontally scaled. If load grows, you add machines. If something is wrong, you take a machine out of the pool, and the others handle it. Failures here are visible immediately, and they're cheap to recover from. The batch server tier can be scaled up or out depending on the work. A batch that's too slow can be split. A batch that crashes can be retried. End users don't see batch servers, so a stuck batch is a problem for me and not for them. Some headroom up here is fine. The database tier is the one I treat completely differently. The database is not where you absorb load. The database is what you protect from load. The reason is simple: the other tiers can be rebuilt or re-scaled. The database is the irreplaceable record. If it slows down, everything slows down. If it falls over, you don't have many minutes before the rest of the stack notices. So my rule for the database is: keep it idle. Not idle in the sense of "doing nothing." Idle in the sense of "running well below its capacity, at all times, so that any extra load it picks up has somewhere to go." For more than a decade I ran a large appliance-grade database where I kept the load average below 1 at all times. N

Davide de Paolis discusses the realities of rolling out cloud infrastructure compliance without fracturing developer relations. Drawing from a real-world platform team reboot at Sevdesk, he explains how to implement "minimum viable governance" on AWS, utilize event-driven Slack alerting to automate policy feedback, and shift from rigid enforcement to high-empathy, data-driven collaboration. By Davide de Paolis

For decades, software was designed around one assumption: A human would be the one using it. That...

Hello Dev Community! 👋 It is officially Day 136 of my software engineering marathon! Today, I engineered the absolute heart of my MERN Stack capstone application, Sprintix : The complete Product Collection Grid & Faceted Filter Sidebar View ( /collection ) ! ⚛️🛍️🗂️ To prepare the application for seamless full-stack state management integration later, I built this layout using dynamic state arrays and object schemas. This ensures that switching from demo arrays to live API streams will happen effortlessly. 🛠️ Deconstructing the Day 136 Catalog Architecture As displayed across my browser rendering workspace in "Screenshot (311).jpg" and "Screenshot (312).jpg" , phase one of the product engine splits into structural layout segments: 1. Faceted Category Filter Sidebar Organized dedicated verification check-boxes mapping out specific consumer collections: Categories: Segmented target groups (Men, Women, Kids). Type Filters: Segmented style formats (Top Wear, Bottom Wear, Winter Wear). Styled within minimal box borders to give users an uncluttered desktop searching experience. 2. Header Control Grid & Sort Registries Installed a top-level workspace header showing "All Collection" alongside an interactive drop-down management node ( Sort by: relevant / low-to-high / high-to-low ). Ready to hold local state flags that rearrange the data arrays instantly before looping. 3. Deep Route Parameter Mapping Preparation Look at the hover elements in "Screenshot (311).jpg" ! Every single rendering card passes localized hex-token structures mapping toward dynamic pathways like: text /product/:id (e.g., /product/6a436b5c921b7aa010d29318)

Resources vs Tools The split: Tools → actions the LLM executes (verbs) LLM decides when to call; calls may have side effects Examples: create_issue, update_status Resources → data the LLM reads (nouns) Host decides when to inject; read-only, no side effects Examples: current Sprint status, project statistics The rule: "reading a state" → Resource. "Executing an operation" → Tool. The same data can have both: get_issue as a Tool (LLM controls when to call it), jira://issue/PROJ-101 as a Resource (Host injects automatically when relevant). Pattern 1: Dynamic Resources A static Resource returns the same data every time (like a project list). A dynamic Resource returns the current state on each read — content changes as the underlying data changes. Sprint status: every read returns live data _sprint_progress_pct = 65 @server.read_resource () async def read_resource ( uri : str ) -> str : if str ( uri ) == " jira://sprint/current " : global _sprint_progress_pct _sprint_progress_pct = min ( 100 , _sprint_progress_pct + random . randint ( 0 , 3 )) return json . dumps ({ " sprint_name " : " Sprint 42 " , " progress_pct " : _sprint_progress_pct , # ← different each time " last_updated " : datetime . now ( timezone . utc ). isoformat (), # ← timestamp changes " days_remaining " : 5 , " p0_open " : count_p0_open (), # ← tracks live state }, indent = 2 ) Test output: Read 1: progress=65% last_updated=...62+00:00 Read 2: progress=67% last_updated=...04+00:00 → ✓ data changed between reads Hardcoding sprint progress in a Prompt means the LLM works from a stale snapshot. A Dynamic Resource gives it the current number on every read. Mark the Resource as dynamic in its description so the LLM knows to re-read when it needs fresh data: Resource ( uri = " jira://sprint/current " , description = ( " Live status of the active sprint: progress, issue counts. " " Read when the user asks about sprint health. " " Re-read if you need up-to-date data — content changes over time. " # ↑ explicit

You spent the afternoon writing that piece. Every claim sourced, every argument tight. You hit publish and watched the numbers. Twenty-four hours later: 41 views. Meanwhile, someone else posted a single sentence — "I quit coffee for 90 days and found something uncomfortable" — and collected 120,000 impressions before lunch. The difference was not effort. It was not even quality. It was a single decision made in the first three words of the title: which emotional circuit to activate. Viral content is not liked into existence. It is clicked into existence. And clicks are not rational — they are reflexive. Understanding the five neural mechanisms that drive that reflex, and knowing how to engineer them deliberately with AI, is the most asymmetric skill advantage available to content creators right now. TL;DR: Every high-CTR title activates one of five hardwired emotional responses. This guide decodes the neuroscience behind each, shows you before/after title rewrites, and demonstrates how a single AI prompt can generate all five variants from any content idea — so you stop guessing which trigger to use and start testing them systematically. Why "Good Writing" and "High CTR" Are Different Problems Before getting into the triggers, it is worth being precise about why these are separate problems — because conflating them is the source of most content creators' frustration. Content quality governs retention : how long someone stays, whether they finish, whether they return. CTR governs distribution : whether the platform's algorithm decides to show your content to more people at all. From a quantitative perspective, these are two entirely separate conditional probabilities that multiply together to determine your content's actual reach: P(Reach) = P(Click)P(Retention|Click) Most creators obsess over P(Retention|Click) — the quality of the experience after the click. But platform distribution algorithms gate on P(Click) first. A piece of content with a retention rate of 0.9

Cargar 112.647 filas en una tabla de hechos debería tardar segundos. A mí me tardaba más de 25 minutos, y acababa cancelando la query. Los datos estaban bien, el SQL estaba bien, las dimensiones se poblaban sin problema. El culpable era otro, y descubrirlo fue la parte más instructiva de todo el proyecto. Todo esto surgió construyendo un Data Warehouse en estrella sobre datos reales de e-commerce: no una tabla bonita para hacer un SELECT * , sino un modelo dimensional completo, reproducible desde cero, capaz de responder preguntas de negocio de verdad. El dataset Trabajé con el Brazilian E-Commerce Public Dataset by Olist : pedidos reales de un marketplace brasileño entre septiembre de 2016 y octubre de 2018. Son 9 CSV relacionados entre sí: 99.441 pedidos y 112.650 líneas de venta 103.886 pagos y 104.719 reseñas 32.951 productos, 3.095 vendedores 1.000.163 registros de geolocalización Y con trampas de datos reales que hay que ver antes de que te muerdan: Un pedido puede tener varios pagos y varias reseñas. Si los unes tal cual a la tabla de hechos, duplicas ventas . Es el error clásico y silencioso: los totales salen inflados y nadie se entera. customer_id no es un cliente. Olist crea uno por cada pedido; la persona real es customer_unique_id . Contar mal aquí te cambia el KPI: hay 99.441 cuentas frente a 96.096 personas. El CSV de productos trae una errata en la cabecera ( product_name_lenght , con "lenght"). Si tu esquema la escribe bien y cargas por interfaz gráfica (que empareja por nombre ), esas columnas se quedan vacías sin que nadie avise. El proceso Monté una arquitectura en capas: CSV → staging → modelo dimensional → vistas → análisis , todo en cuatro scripts ejecutables en orden y idempotentes (el esquema se recrea desde cero, se puede relanzar mil veces). El modelo es un star schema : una tabla de hechos fact_sales al grano de línea de producto dentro de un pedido , y cinco dimensiones (cliente, producto, vendedor, pago y fecha), con claves sustitutas,

We Taught a Snowflake Warehouse to Judge World Cup Conviction and Write the Verdict Back to Solana Soumyadeep Dey Soumyadeep Dey Soumyadeep Dey Follow Jul 12 We Taught a Snowflake Warehouse to Judge World Cup Conviction and Write the Verdict Back to Solana # devchallenge # weekendchallenge # snowflake 5 reactions 1 comment 16 min read

skip에서 partition overwrite로: business_date 재처리를 Iceberg로 다시 표현하기 이전 글에서는 같은 source_hash 가 다시 들어왔을 때 기존 successful run을 재사용하는 idempotency를 다뤘다. 하지만 재처리에는 두 종류가 있다. 1. 같은 입력이 다시 들어온 경우 -> skip이 맞다. 2. 같은 business_date의 정정 입력이 들어온 경우 -> skip하면 안 된다. -> 같은 날짜의 gold 결과를 중복 없이 교체해야 한다. manufacturing-data-platform-mini 의 B5 slice는 두 번째 문제를 아주 작게 다룬다. 전체 Spark pipeline을 만든 것이 아니다. gold_daily_metrics Iceberg table 하나를 local Spark에서 만들고, business_date partition overwrite와 snapshot evidence만 검증했다. Scenario 이미 아래 gold row가 있다. business_date=2026-06-29 plant-a / line-1 / gearbox-a units_produced=120 defect_count=3 나중에 같은 business_date=2026-06-29 에 대한 정정 source가 들어온다. 운영자가 원하는 것은 append가 아니다. 원하지 않는 상태: 2026-06-29 old row 2026-06-29 corrected row -> 같은 날짜 결과가 중복됨 원하는 상태: 2026-06-29 corrected row만 남음 2026-06-30 같은 다른 날짜 partition은 그대로 유지됨 재처리 전후 snapshot evidence가 남음 그래서 이 slice의 질문은 이렇다. 같은 business_date의 정정 source를 처리할 때, gold table에서 해당 날짜 partition만 중복 없이 교체하고, 어떤 run이 어떤 Iceberg snapshot을 만들었는지 남길 수 있는가? Decision Pressure Slice1의 CSV pipeline은 already-successful source를 안전하게 skip할 수 있다. dataset_id + business_date + source_hash 이 key가 같으면 같은 입력이다. 다시 계산해도 같은 결과이므로 기존 run을 재사용한다. 하지만 source_hash 가 달라졌다면 의미가 다르다. same business_date different source_hash 이건 retry가 아니라 correction이다. CSV run-folder 방식에서는 새 run output을 만들 수는 있지만, "현재 gold table에서 해당 날짜를 원자적으로 교체한다"는 table-level 의미가 약하다. Iceberg를 붙이는 이유는 여기 있다. source_hash -> 같은 입력인지 판단하는 idempotency key business_date partition -> 정정 시 교체할 gold table 범위 snapshot_id -> table commit의 evidence 즉 Spark/Iceberg는 도구 이름을 추가하려고 붙인 것이 아니라, 재처리 상태 전이를 더 명확히 표현하기 위해 붙였다. Options Option 장점 문제 판단 same source면 항상 재계산 단순함 retry 때 불필요한 commit이 계속 생김 제외 corrected source를 append 구현 쉬움 같은 날짜 gold row가 중복될 수 있음 제외 whole-table overwrite 단순함 다른 날짜 partition까지 지울 위험 제외 business_date partition overwrite correction 범위가 명확함 Spark/Iceberg 설정과 test가 필요 선택 MERGE/upsert 강력함 이번 skeleton에 과함 backlog 이번 구현은 DataFrameWriterV2.overwritePartitions() 를 사용했다. corrected_d

wide CSV 여러 개를 EAV로 모아 gold mart 만들기 현실의 데이터 소스는 한 가지 모양으로 오지 않는다. 같은 의미의 값도 어떤 파일에서는 생산수량 , 다른 파일에서는 units , 또 다른 파일에서는 made 로 올 수 있다. 온도도 어떤 곳은 섭씨, 어떤 곳은 화씨일 수 있다. 이걸 매번 pipeline code에 if source == ... 로 박기 시작하면 source가 늘 때마다 코드가 지저분해진다. manufacturing-data-platform-mini 의 EAV mini slice는 이 문제를 작게 다룬다. 여러 wide CSV를 mapping config로 표준 attribute에 맞춘 뒤, EAV long format으로 모으고, 다시 gold metric mart로 pivot/aggregate한다. 데이터는 모두 synthetic이고, 회사 코드·고객 데이터·실제 schema는 쓰지 않았다. 1. Scenario 서로 다른 공장/라인/벤더에서 비슷한 제조 지표 파일이 들어온다. 예: plant_a.csv: 설비ID, 생산수량, 불량수, 온도C, 압력kPa plant_b.csv: machine_id, output_units, defects, temp_f, pressure_bar vendor_d.csv: unit_name, made, scrap, deg_c, kpa 비즈니스적으로는 같은 지표를 보고 싶다. units_produced defect_count temperature_c pressure_kpa 문제는 source마다 column name과 unit이 다르다는 점이다. 2. Decision Pressure 단순 구현은 source마다 코드를 늘린다. if source == "plant_a": 생산수량을 units_produced로 읽는다 if source == "plant_b": output_units를 units_produced로 읽는다 temp_f를 섭씨로 변환한다 if source == "vendor_d": made를 units_produced로 읽는다 이 방식은 작게는 빨라 보이지만 source가 늘수록 문제가 된다. 새 파일 형식마다 pipeline code를 고쳐야 한다. column mapping과 transform logic이 섞인다. unit conversion이 흩어진다. quality check가 source별로 갈라진다. gold mart grain을 설명하기 어려워진다. 그래서 mapping은 config로 빼고, pipeline은 표준 attribute를 처리하게 만들었다. 3. Options option result risk source별 hard-coded parser 처음엔 빠름 source가 늘 때 code change 반복 모든 source를 wide table 하나로 합치기 보기 쉬움 sparse/heterogeneous column 폭발 EAV long format 이종 attribute를 표준 형태로 모음 pivot/quality 설계가 필요 full mapping DSL/rules engine 유연함 mini project에는 과함 이 프로젝트의 선택은 단순한 JSON mapping + EAV long + gold pivot이다. 4. Decision 각 source는 JSON config로 자신의 column을 표준 attribute에 매핑한다. source column -> standard attribute output_units -> units_produced temp_f -> temperature_c with f_to_c pressure_bar -> pressure_kpa with bar_to_kpa pipeline 흐름: wide CSVs -> mapping configs -> EAV long rows -> gold entity_daily_metrics -> quality checks -> catalog/lineage EAV row의 핵심 shape: entity_id business_date attribute value v