今日已更新 153 条资讯 | 累计 21372 条内容
关于我们

标签:#AR

找到 3812 篇相关文章

AI 资讯

Real-Time AI Feature Engineering with Spark Structured Streaming and Databricks Feature Store

Building point-in-time correct, production-grade feature pipelines — from raw Kafka events to online feature serving in milliseconds, using Spark Structured Streaming and the Databricks Feature Store. Table of Contents The Feature Engineering Problem Architecture Overview Feature Store Concepts: ERD Environment Setup Streaming Feature Pipeline Point-in-Time Correct Training Dataset Generation Writing Features to the Online Store Serving Features at Inference Time Feature Table Reference References The Feature Engineering Problem Feature engineering is where most ML projects silently fail in production. Not because the model is wrong — but because the features the model sees at training time are different from the features it sees at inference time . This is called training-serving skew , and it's the #1 silent killer of ML systems. Three specific failure modes cause it: Online/offline inconsistency — the batch pipeline that computes training features uses different logic than the real-time service that computes inference features Data leakage — training features accidentally include information from the future (e.g. joining on a label that was created after the event) Feature staleness — a model trained on 30-day rolling averages is served features that are 6 hours stale because the pipeline backfills are slow The Databricks Feature Store — now part of Unity Catalog as Feature Engineering in Unity Catalog — solves all three by: Storing feature computation logic alongside the data (no drift between training and serving) Enforcing point-in-time lookups during training dataset creation Providing a unified API for both batch offline reads and low-latency online reads Architecture Overview Feature Store Concepts: ERD Understanding the data model behind the Feature Store is essential for designing correct pipelines. Here's how the entities relate: The critical relationship: a Model Version is bound to a Training Set , which records exactly which feature tables and which p

2026-06-24 原文 →
AI 资讯

Apache Spark Query Optimization on Databricks: Catalyst, AQE, and Photon Engine

A deep dive into how Spark transforms your SQL into a physical execution plan — and how Databricks layers Adaptive Query Execution and the Photon vectorized engine on top to squeeze out maximum performance. Table of Contents Why Query Optimization Matters The Catalyst Optimizer Pipeline Stage 1: Parsing — From SQL to Unresolved Logical Plan Stage 2: Analysis — Binding to the Catalog Stage 3: Logical Optimization — Rule-Based Rewrites Stage 4: Physical Planning — Strategies and Cost Models Adaptive Query Execution (AQE) The Photon Engine Reading Explain Plans Tuning Reference Table References Why Query Optimization Matters A Spark query written by a human and a Spark query executed by the engine are often very different things. The gap between them — the optimization — is what separates a job that runs in 3 minutes from one that runs in 3 hours on identical hardware. Databricks compounds Spark's native Catalyst optimizer with two additional layers: Adaptive Query Execution (AQE) — re-optimizes the query at runtime using actual statistics collected mid-job Photon — a C++ vectorized execution engine that replaces the JVM-based Spark executor for eligible operators Understanding all three lets you write queries that cooperate with the engine rather than fight it. The Catalyst Optimizer Pipeline Catalyst is Spark's rule-based and cost-based query optimizer. Every query — whether written in SQL, DataFrame API, or Dataset API — passes through the same four-stage pipeline before a single byte of data is read. Stage 1: Parsing — From SQL to Unresolved Logical Plan # ── Catalyst Stage 1: Parsing ───────────────────────────────────────────────── # Spark uses ANTLR4 to parse SQL into an Abstract Syntax Tree (AST). # At this point column names are NOT validated — the plan is "unresolved". from pyspark.sql import SparkSession spark = SparkSession . builder . appName ( " catalyst-demo " ). getOrCreate () # Both of these produce identical internal representations df_api = ( spark .

2026-06-24 原文 →
开发者

Article: Beyond CLEAN and MVP: Architecting an Offline-first Reactive Data Layer in Android

With the Reactive Data Layer Architecture (RDLA), you establish a clear boundary between public data APIs and private, framework-specific data-source implementations. Your presentation layer operates in a purely reactive manner, observing data changes rather than procedurally querying them. RDLA also simplifies testing by encouraging you to program to interfaces and use clean seeding patterns. By Mervyn Anthony

2026-06-24 原文 →
AI 资讯

Exploring Polymarket's 1-Hour Markets: Data Analysis, Mispricing Opportunities, and Automated Trading Strategies

Prediction markets have become increasingly popular among traders looking for alternative ways to speculate on asset movements. While much of the attention has been focused on short-term 5-minute and 15-minute markets, I believe one of the most overlooked opportunities right now is the 1-hour market on Polymarket. In this article, I'll share some of my ongoing research, explain how I'm collecting and analyzing market data, discuss potential arbitrage and mispricing opportunities, and show how automation can help traders capitalize on these inefficiencies. Why I'm Focusing on the 1-Hour Market Many traders are currently concentrated on the 15-minute Bitcoin prediction markets. While these markets can be profitable, competition has increased significantly, and recent fee changes have made certain strategies less attractive. The 1-hour markets, however, present a different opportunity. These markets offer: Longer trading windows More time to manage positions Higher flexibility for order placement Potentially lower competition No trading fees on some hourly markets Because of the longer duration, traders have more time to identify inefficiencies and execute strategies that may be difficult to implement in shorter timeframes. Collecting Market Data Directly from Polymarket One of the projects I've been working on involves collecting market data directly from Polymarket and monitoring token price movements in real time. Rather than relying solely on the displayed market prices, I use blockchain-based data sources that can provide updates faster than the front-end interface. This allows me to analyze: YES token price swings NO token price swings Order book movements Temporary mispricings Combined token costs The goal is to understand how both sides of a market move throughout the trading period and identify situations where the combined cost of YES and NO tokens falls below $1. Understanding YES and NO Token Swings One interesting metric I track is the lowest price reached

2026-06-24 原文 →
AI 资讯

I built an interactive 11-chapter guide to how LLM inference actually works

Production vLLM is 100,000+ lines of C++, CUDA, and Python. It powers most of the industry's LLM serving — but reading it cold is brutal. So I built a study series around nano-vLLM , an open-source reimplementation of vLLM's core ideas in ~1,200 lines of pure Python. Every algorithm is visible. Every design decision is legible. It turned out to be the perfect lens for actually understanding how LLMs generate text. The result is an 11-chapter interactive guide. No ML background required — every piece of jargon is explained from scratch with analogies, diagrams, annotated source code, interactive simulators, and quizzes. What it covers: What Is LLM Inference? — tokens, autoregressive generation, Q/K/V attention, HBM vs SRAM Architecture — how 1,200 lines are organised; CPU control plane vs GPU data plane KV Cache — why storing Keys and Values turns O(N²) recomputation into O(1) lookup PagedAttention — virtual memory for the KV cache; how fragmentation wastes 60–80% of GPU memory The Scheduler — continuous batching; keeping the GPU at 95% utilisation instead of 12% Prefill vs Decode — same model, two completely different bottlenecks (compute-bound vs memory-bound) Prefix Caching — skip prefill for shared tokens; ~700ms → ~90ms TTFT Sampling Strategies — greedy, temperature, top-k, top-p, and what each does to the distribution Tensor Parallelism — splitting a model across GPUs; column/row parallel and all-reduce The Optimization Stack — FlashAttention, kernel fusion, CUDA Graphs, torch.compile Benchmarks — measuring honestly; why nano-vLLM matches vLLM on core throughput Each chapter is fully self-contained and interactive. A few of the simulators I'm most happy with: a PagedAttention block allocator you can fill up and watch fragment, a live scheduler you step through token by token, and a sampling playground where you reshape the probability distribution with sliders and sample from it. 🔗 Read the full series: https://ashwing.github.io/vllm-guide/ It's free and open.

2026-06-24 原文 →
AI 资讯

Bootstrap confidence intervals for your LLM eval metrics

TL;DR: A single eval number hides its own uncertainty. Eval confidence intervals from bootstrap resampling turn a point estimate like 84.2% accuracy into a range, so you stop shipping models on a difference that is noise. Two checkpoints came back from a fine-tuning run at 84.2% and 85.7% on our 500-example agent eval set. The 1.5 point gap read like a win, and someone wanted to promote the second checkpoint to staging. Before that, I wanted eval confidence intervals on both numbers, because a 500-example set carries more sampling error than most teams admit. At 500 examples, the 95% interval on a single accuracy near 85% spans roughly 3 points on each side. The win sat well inside the noise. I lead the fine-tuning and evaluation team at Nexus Labs, and the most common mistake I see is treating an eval score as exact. It isn't. Your eval set is a sample drawn from the input space you care about, and a different 500 examples would return a different number. Confidence intervals make that variance visible. What an eval confidence interval actually tells you An eval confidence interval is a range around a metric, like accuracy or F1, that quantifies how much the score would move if you resampled the eval set. A 95% bootstrap interval of [81.0%, 87.1%] means that across thousands of resamples of your data, 95% of the recomputed scores fell in that band. It measures sampling noise, not model quality. That distinction matters. Two checkpoints scoring 84.2% and 85.7% with overlapping intervals are, as far as your eval set can tell, indistinguishable. Card et al. showed in "With Little Power Comes Great Responsibility" that many NLP experiments are underpowered to detect the effect sizes they report. Computing bootstrap confidence intervals The bootstrap is resampling with replacement. You take your per-example results, draw N of them with replacement many times, recompute the metric each time, and read percentiles off the resulting distribution. There's no assumption that

2026-06-24 原文 →
AI 资讯

I got a merged PR into a YC startup before they ever replied to my job application

I applied to a YC W25 startup the normal way. Filled out the form, wrote a decent cover letter, hit submit. Silence. While waiting, I found their open-source repo on GitHub. Read through the codebase out of genuine curiosity I wanted to understand what they were actually building. Found a bug. Fixed it. Opened a PR. It got merged in 2 days. They still hadn't replied to my application. Here's what that taught me about job hunting in 2025: A cover letter tells someone what you claim you can do. A merged PR shows them. One of those gets read. The other gets filed under "maybe later" -which is just "no" with extra steps. I'm not saying cold applications are dead. I'm saying they're the last resort, not the first move. If a company has a public repo, you have a backdoor that most applicants don't even think to try. Read the code deep and find something small but real. Fix it and Open a PR. Now you're not a stranger in their inbox you're someone who already ships for them. The reply came eventually, by the way. But by then, the maintainers already knew my GitHub handle. That matters more than you think. Have you ever landed something through a contribution instead of an application? Drop it in the comments curious how many people have done this.

2026-06-24 原文 →
AI 资讯

# Unit of Work: Managing Database Transactions Like a Pro with Python

Introduction Every serious backend developer eventually faces the same problem: you need to make multiple changes to a database as part of a single business operation, and you need all of them to succeed or none of them to go through. Partial updates are worse than no updates at all - they leave your data in an inconsistent state that can be nearly impossible to debug in production. This is not a new problem. Enterprise developers have been solving it for decades, and Martin Fowler documented the canonical solution in his 2002 book Patterns of Enterprise Application Architecture : the Unit of Work pattern. In this article we are going to go deep on what Unit of Work is, why it exists, how it works internally, and how to build a clean, production-quality implementation from scratch in Python using only the standard library. By the end you will have a working implementation you can adapt to any project, and a solid understanding of how popular frameworks like SQLAlchemy and Django ORM implement this pattern under the hood. The full source code is available on GitHub: 👉 github.com/diegocastillo12/unit-of-work-python - ## Background: What is the Unit of Work Pattern? The Unit of Work pattern is part of Martin Fowler's catalog of Patterns of Enterprise Application Architecture (PoEAA), a collection of battle-tested solutions for common problems in enterprise software design. Fowler defines it as follows: > "A Unit of Work maintains a list of objects affected by a business transaction and coordinates the writing out of changes and the resolution of concurrency problems." Let's unpack that definition carefully. "Maintains a list of objects affected by a business transaction" - this means the Unit of Work acts as a tracker. When your business logic creates a new object, modifies an existing one, or marks one for deletion, it does not immediately write to the database. Instead, it registers the change with the Unit of Work, which keeps an in-memory list of everything that ne

2026-06-24 原文 →