标签：#Python

A deep dive into how Spark transforms your SQL into a physical execution plan — and how Databricks layers Adaptive Query Execution and the Photon vectorized engine on top to squeeze out maximum performance. Table of Contents Why Query Optimization Matters The Catalyst Optimizer Pipeline Stage 1: Parsing — From SQL to Unresolved Logical Plan Stage 2: Analysis — Binding to the Catalog Stage 3: Logical Optimization — Rule-Based Rewrites Stage 4: Physical Planning — Strategies and Cost Models Adaptive Query Execution (AQE) The Photon Engine Reading Explain Plans Tuning Reference Table References Why Query Optimization Matters A Spark query written by a human and a Spark query executed by the engine are often very different things. The gap between them — the optimization — is what separates a job that runs in 3 minutes from one that runs in 3 hours on identical hardware. Databricks compounds Spark's native Catalyst optimizer with two additional layers: Adaptive Query Execution (AQE) — re-optimizes the query at runtime using actual statistics collected mid-job Photon — a C++ vectorized execution engine that replaces the JVM-based Spark executor for eligible operators Understanding all three lets you write queries that cooperate with the engine rather than fight it. The Catalyst Optimizer Pipeline Catalyst is Spark's rule-based and cost-based query optimizer. Every query — whether written in SQL, DataFrame API, or Dataset API — passes through the same four-stage pipeline before a single byte of data is read. Stage 1: Parsing — From SQL to Unresolved Logical Plan # ── Catalyst Stage 1: Parsing ───────────────────────────────────────────────── # Spark uses ANTLR4 to parse SQL into an Abstract Syntax Tree (AST). # At this point column names are NOT validated — the plan is "unresolved". from pyspark.sql import SparkSession spark = SparkSession . builder . appName ( " catalyst-demo " ). getOrCreate () # Both of these produce identical internal representations df_api = ( spark .

An AI agent that's flawless in the demo can still fall apart the first time a tool fails in production: a timeout, a network error, a response that comes back corrupted. The fix isn't a smarter model. It's testing the agent against those failures on purpose, before your users do, and hardening it one failure type at a time.

Production vLLM is 100,000+ lines of C++, CUDA, and Python. It powers most of the industry's LLM serving — but reading it cold is brutal. So I built a study series around nano-vLLM , an open-source reimplementation of vLLM's core ideas in ~1,200 lines of pure Python. Every algorithm is visible. Every design decision is legible. It turned out to be the perfect lens for actually understanding how LLMs generate text. The result is an 11-chapter interactive guide. No ML background required — every piece of jargon is explained from scratch with analogies, diagrams, annotated source code, interactive simulators, and quizzes. What it covers: What Is LLM Inference? — tokens, autoregressive generation, Q/K/V attention, HBM vs SRAM Architecture — how 1,200 lines are organised; CPU control plane vs GPU data plane KV Cache — why storing Keys and Values turns O(N²) recomputation into O(1) lookup PagedAttention — virtual memory for the KV cache; how fragmentation wastes 60–80% of GPU memory The Scheduler — continuous batching; keeping the GPU at 95% utilisation instead of 12% Prefill vs Decode — same model, two completely different bottlenecks (compute-bound vs memory-bound) Prefix Caching — skip prefill for shared tokens; ~700ms → ~90ms TTFT Sampling Strategies — greedy, temperature, top-k, top-p, and what each does to the distribution Tensor Parallelism — splitting a model across GPUs; column/row parallel and all-reduce The Optimization Stack — FlashAttention, kernel fusion, CUDA Graphs, torch.compile Benchmarks — measuring honestly; why nano-vLLM matches vLLM on core throughput Each chapter is fully self-contained and interactive. A few of the simulators I'm most happy with: a PagedAttention block allocator you can fill up and watch fragment, a live scheduler you step through token by token, and a sampling playground where you reshape the probability distribution with sliders and sample from it. 🔗 Read the full series: https://ashwing.github.io/vllm-guide/ It's free and open.

Introduction Every serious backend developer eventually faces the same problem: you need to make multiple changes to a database as part of a single business operation, and you need all of them to succeed or none of them to go through. Partial updates are worse than no updates at all - they leave your data in an inconsistent state that can be nearly impossible to debug in production. This is not a new problem. Enterprise developers have been solving it for decades, and Martin Fowler documented the canonical solution in his 2002 book Patterns of Enterprise Application Architecture : the Unit of Work pattern. In this article we are going to go deep on what Unit of Work is, why it exists, how it works internally, and how to build a clean, production-quality implementation from scratch in Python using only the standard library. By the end you will have a working implementation you can adapt to any project, and a solid understanding of how popular frameworks like SQLAlchemy and Django ORM implement this pattern under the hood. The full source code is available on GitHub: 👉 github.com/diegocastillo12/unit-of-work-python - ## Background: What is the Unit of Work Pattern? The Unit of Work pattern is part of Martin Fowler's catalog of Patterns of Enterprise Application Architecture (PoEAA), a collection of battle-tested solutions for common problems in enterprise software design. Fowler defines it as follows: > "A Unit of Work maintains a list of objects affected by a business transaction and coordinates the writing out of changes and the resolution of concurrency problems." Let's unpack that definition carefully. "Maintains a list of objects affected by a business transaction" - this means the Unit of Work acts as a tracker. When your business logic creates a new object, modifies an existing one, or marks one for deletion, it does not immediately write to the database. Instead, it registers the change with the Unit of Work, which keeps an in-memory list of everything that ne

Line AI Chatbot In Production: A CTO's Honest Breakdown Three months ago I was staring at our infrastructure bill wondering where the hell our runway went. We'd been running a customer-facing chatbot powered by a popular "enterprise" AI provider, and the cost curve looked like a hockey stick in the wrong direction. Every new sign-up bled money. I knew we had to make a change before our next board meeting, but I also couldn't afford a six-week migration that would tank our product velocity. What I found surprised me. After running the numbers, testing 184 models through Global API, and stress-testing everything at scale, I cut our inference costs by more than half without touching quality. This isn't a theoretical comparison from a vendor whitepaper. These are the real numbers from my production stack, with my actual users, in my actual platform. If you're a CTO weighing your options for 2026, here's everything I wish someone had told me before I started. Why The Line AI Chatbot Approach Matters Now Most chatbot guides treat AI integration like a toy problem. Send a prompt, get a response, ship the demo. That's fine for a hackathon, but it's not how you run a production system. The questions I care about are different: What's my cost per active user? How do I avoid vendor lock-in? Where's the single point of failure? How fast can I iterate on model choice when something better drops next Tuesday? The Line AI Chatbot framework flips the typical approach. Instead of treating the model as a black box you can't replace, you build a thin abstraction layer over a model-agnostic API. That single architectural decision is what unlocked every other win I describe below. If you're not thinking about model portability on day one, you're going to pay for it later. I learned this the hard way. In 2026, the market has matured to a point where you genuinely have 184 models to choose from, with input prices ranging from $0.01 to $3.50 per million tokens. That's not a marketing line.

I'll be honest — I didn't set out to build a developer tool. I'm an engineer by trade. I build structural and forensic engineering software. C++, WinUI 3, heavy desktop apps. But a big chunk of my prototyping and internal tooling happens in Python — and every time I sat down to spin up a quick Python desktop app, I hit the same wall. Every launcher, every hot-reload tool, every dev cockpit I found wanted something from me. Install this. License that. Set up a virtual environment. Add five dependencies just to watch a file change. I just wanted to run my app, see it update when I changed something, and get back to work. So I built ILX Launcher. The rule I gave myself was simple: pure Python stdlib and tkinter. Nothing else. If it couldn't be done with what Python already ships with, I didn't need it. What came out of that constraint surprised me. No pip install. No virtual environment required. No licensing headaches. You clone it, you run it, it works. That's it. It's a developer cockpit for Python desktop apps — run, hot-reload, test, profile, and ship, all from one place. The kind of tool I wished existed six months ago. It's early. It's rough around the edges. But it works, and it's already saving me time every single day. If you've ever felt like your dev tooling was getting in the way of actually building — I'd love for you to try it and tell me what you think. 👉 github.com/ilxstudio/ILX-Launcher And if it saves you even five minutes — drop a ⭐ on the repo. It genuinely helps others find it.

About A simple calculator built in PyCharm at the age of 14 ! I created this learning solely through YouTube tutorials which allowed me to become an intermediate Python programmer . It helped me learn and understand core programming concepts . This project is part of my journey toward becoming a stronger Python programmer GitHub & Code - https://github.com/muhammadkharwa/Calc.v2 I’m always open to feedback, so feel free to let me know if there’s anything I could improve or any mistakes I may have made.

There's a moment in every ambitious AI engineering project where you convince yourself that more agents means more power. I hit that moment early in building my Python orchestration framework — and I spent several painful weeks learning exactly why that intuition is wrong. The seductive pitch: decompose complex tasks into specialized sub-agents, run them in parallel, let them coordinate. What actually happened was a reliability nightmare that taught me more about agentic architecture than any framework documentation ever could. The Problem I Actually Built My Python orchestration system was designed to automate complex, multi-step workflows — the kind that require planning, research, code generation, and validation to happen in a coherent sequence. Early on, I structured it as a web of parallel agents: a planner, several workers, a validator, and a synthesizer, all exchanging structured messages. On paper it was elegant. In practice, it had three failure modes I couldn't engineer away: Context drift. Each agent only saw the slice of information it was handed. The worker writing one module couldn't see what the worker writing another module had decided. By the time the synthesizer tried to combine outputs, I had conflicting assumptions baked into the results — variable names that clashed, patterns that contradicted each other, interfaces that didn't align. Cascading partial failures. When one agent produced ambiguous output, every downstream agent amplified the ambiguity. A planner that returned a slightly underspecified task description produced workers that each interpreted it differently. Nothing failed loudly. Everything just drifted, quietly, until the final output was incoherent. Debugging opacity. When something went wrong in a parallel multi-agent system, tracing the failure was miserable. Was it the planner? One of the workers? The message-passing layer? I'd rebuilt the worst parts of distributed systems debugging inside a single Python process. The Architec

GitHub热门项目 | Official, AWS-supported MCP servers, skills, and plugins to help AI agents build on AWS | Stars: 958 | 13 stars today | 语言: Python

GitHub热门项目 | A community-supported supercharged document management system: scan, index and archive all your documents | Stars: 42,384 | 46 stars today | 语言: Python

How I Stopped Burning Cash on Token Limits — A CTO's Field Notes Three months ago, I was staring at our monthly AI bill wondering where it all went wrong. We'd built what I thought was a pretty elegant LLM pipeline. Production-ready, observability wired up, the whole nine yards. Then the invoices started arriving, and I realized I had built a money furnace. Our token consumption was spiking 3x week over week, the 429s were everywhere, and our latency had become a meme inside the company. This is the post I wish I'd had six months ago. If you're a technical founder or a CTO running LLM workloads at scale, bookmark this. I'm going to walk you through the exact architecture decisions, the exact numbers, and the exact code that took us from "this bill is going to kill us" to "oh, this is actually manageable." The Real Problem Nobody Talks About Here's the dirty secret about running LLM-powered products: token limit errors aren't really about token limits. They're a symptom of a much deeper architectural problem. When your app throws "context length exceeded" at 2am, what it's really telling you is that you didn't think hard enough about prompt design, document chunking, model selection, and cost routing on day one. I learned this the hard way. My team was defaulting to GPT-4o for everything because, honestly, it works and the API is reliable. We were paying $2.50 per million input tokens and $10.00 per million output tokens. For a startup processing millions of documents a month, that math is brutal. We were essentially funding OpenAI's next training run with our Series A. The wake-up call came when I ran the actual numbers. Our average request was burning through maybe 8K input tokens and producing 2K output tokens. At our volume, we were spending more on inference than on two senior engineers. That is not a sustainable burn rate for a 12-person company. The Architecture Decision That Changed Everything The first question I asked myself wasn't "which model is cheapest?

Many Python beginners can write basic programs but get stuck when they try to run a real project on their own laptop. The issue is not always coding. Sometimes the real problem is setup . You may know loops, functions, and lists, but still face problems like: ModuleNotFoundError Python is not recognized Package installed but not working in VS Code Wrong interpreter selected These are common beginner setup issues. Why online compilers are not enough Online compilers are good for quick practice. But real Python projects need: project folders multiple files external packages virtual environments dependency files terminal commands debugging tools Git basics So, when the goal is to build real projects, it is better to move to a local Python setup early. Basic Python project setup flow A simple Python project setup flow looks like this: Install Python Install VS Code Create project folder Create virtual environment Activate virtual environment Install packages Save requirements.txt This setup may look basic, but it prevents many beginner-level errors later. Example folder structure A beginner-friendly Python project folder can look like this: python-project/ │ ├── main.py ├── requirements.txt ├── README.md └── venv/ Here is what each file or folder means: main.py is the main Python file. requirements.txt stores project dependencies. README.md explains the project. venv/ contains the virtual environment. Create a virtual environment Create a virtual environment using: python -m venv venv Activate it on Windows: venv \S cripts \a ctivate Activate it on Mac/Linux: source venv/bin/activate A virtual environment keeps each project’s packages separate. This helps avoid package conflicts when working on multiple Python projects. Install a package After activating the virtual environment, install packages using pip . Example: pip install requests Now create a Python file: import requests response = requests . get ( " https://api.github.com " ) print ( response . status_code ) If

The easiest way to misunderstand LangGraph is to see it as “LangChain, but with more steps.” That misses the point. LangGraph becomes useful when an agent is no longer a single prompt or a simple chain. It becomes useful when the workflow has state, branches, tool calls, human approval, checkpointing, and recovery behavior that must be inspected before the agent is trusted inside a real AI host. I used the Doramagic LangGraph manual as the source-backed reading layer for this note: https://doramagic.ai/en/projects/langgraph/manual/ This is an independent project guide, not an official LangGraph document. I use it as a pre-adoption checklist: what should be understood before wiring a project into Claude, ChatGPT, Cursor, Codex, or another AI host. The point is not to create another prompt library. The useful artifact is a capability resource pack: a manual, source map, boundary notes, pitfall log, smoke check, lightweight eval criteria, feedback notes, and host-ready context that help a developer decide what to verify before adoption. 1. The real boundary is State, not the prompt For a one-shot model call, the prompt is often the main boundary. For LangGraph, the first boundary is the State schema: which fields move between nodes; which fields a node may update; how concurrent branches merge values; which values enter a checkpoint; which values should never be persisted. This is why reducers matter. A message list is usually not just overwritten. It needs an append or merge rule such as add_messages or the TypeScript equivalent. That small implementation detail decides whether parallel work preserves context or silently drops it. My preferred first run is not a “universal agent.” It is a tiny graph with one State schema, one node, one partial update, and one explicit reducer. If that is not clear, adding tools will only hide the problem. 2. compile() is the boundary between description and runtime Before compile() , a LangGraph graph is a description: nodes, edges, c

Part 6 of a beginner-friendly series on learning Python from scratch. In Part 5 , we learned to organize data with lists, dictionaries, and other collections. Now it's time to organize our code itself. A function is a reusable block of code that performs a specific task. Instead of writing the same code multiple times, you write it once in a function, then call that function whenever you need it. This is the foundation of writing clean, maintainable programs. Defining and Calling Functions The basics def greet (): print ( " Hello, World! " ) greet () # Call the function Use def to define a function. The function name is followed by parentheses and a colon. The indented block below is the function's body. When you call the function (by typing its name with parentheses), Python runs the code inside it. Functions with parameters Most functions need information to work with. That's what parameters are for: def greet ( name ): print ( f " Hello, { name } ! " ) greet ( " Ramesh " ) # Hello, Ramesh! greet ( " Priya " ) # Hello, Priya! name is a parameter (placeholder). When you call greet("Ramesh") , name becomes "Ramesh" inside the function. Multiple parameters: def add ( x , y ): print ( x + y ) add ( 5 , 3 ) # 8 add ( 10 , 20 ) # 30 Return values A function can calculate something and give the result back to you with return : def add ( x , y ): return x + y result = add ( 5 , 3 ) print ( result ) # 8 The return statement stops the function and sends a value back. The caller can then use that value. def greet ( name ): message = f " Hello, { name } ! " return message greeting = greet ( " Ramesh " ) print ( greeting ) # Hello, Ramesh! A function can return multiple values as a tuple: def get_user_info (): return " Ramesh " , 25 , " Chennai " name , age , city = get_user_info () print ( name , age , city ) # Ramesh 25 Chennai Arguments: Positional vs Keyword There are two ways to pass values to a function: Positional arguments Arguments are matched by position: def describ

I spent the last few months trying to understand why reconciliation errors keep appearing in high-throughput pipelines. Here is what I found. In the race to process millions of transactions daily, modern fintech ecosystems have achieved a genuine miracle of scale. But beneath the surface of that velocity lies a structural problem most engineering teams aren't measuring: causality violations in async event pipelines. Most teams assume that if a transaction shows "Success" in the database, the job is done. At high concurrency levels, that assumption breaks quietly. When "Eventual Consistency" Becomes "Eventual Loss" In distributed systems, Kafka partitions and database shards experience micro-millisecond timing gaps. When a network retry delays a validation webhook, the downstream ledger can commit a wallet update before the validation that should have preceded it completes. To the user, the app glitches. To the engineering team, it's a reconciliation ticket. To the CFO, it's untracked operational cost. The Reconciliation Tax I built a simulation modelling this exact failure mode across 5,000 concurrent transactions. With an 8% network retry probability, conservative for high-traffic payment rails, the causality violation rate was 8.3%. At one million daily transactions, that's over 80,000 unvalidated commits every day requiring manual review. The operational cost compounds across three dimensions: engineering hours spent patching database state, fraud model accuracy degrading on out-of-order training data, and audit trails that cannot demonstrate strict causal sequence to regulators. The Fix The solution is enforcing strict event ordering at the ingestion layer before state commits happen, not better monitoring after the fact. When safeguards including partition-aware routing, exponential backoff, and idempotency controls were added to the same simulation, the violation rate dropped to 0%. Full simulation code and methodology: github.com/yakuburoseline1-gif/cif-simul

Here's the scene I've watched play out on a dozen teams. Accuracy plateaus. Someone rips out the logistic regression, drops in XGBoost, and waits for the jump. It doesn't come — or it comes with two points you can't explain to anyone. So the week disappears into hyperparameter tuning, and you end up with a slower, heavier, less interpretable model that's barely better than where you started. The model was almost never the bottleneck. The features were. This post is the long, practical version of that argument. We'll define the two camps in plain language, run real code, look at when boosting genuinely wins, and then walk through the failure mode nobody warns you about — the one where the fancy model is "winning" because it's quietly cheating. A note before we start: keep your examples generic. We'll predict a numeric target — think demand, a quantity, a score on a tabular dataset. The principles are the same everywhere, and you should validate them on your own data. The two camps, in plain terms Linear / logistic regression fits a straight-line relationship: each feature gets a weight (a coefficient), and the prediction is a weighted sum. Logistic regression is the same idea bent for classification — it outputs a probability. from sklearn.linear_model import LogisticRegression model = LogisticRegression ( max_iter = 1000 ) model . fit ( X_train , y_train ) # the whole model, readable in one line per feature: for name , weight in zip ( feature_names , model . coef_ [ 0 ]): print ( f " { name : < 20 } { weight : + . 3 f } " ) That loop is the entire model. A positive weight means "more of this pushes the prediction up," and you can hand that table to a stakeholder and defend every number. The cost: it assumes the relationship is roughly linear and that features act independently. Real data often isn't that polite. Gradient boosting (XGBoost, LightGBM, sklearn's GradientBoostingClassifier ) builds hundreds of small decision trees, each one correcting the mistakes of th

I spent a week building Wingman , an open source MCP server that renders a persistent task panel inline in Claude conversations using MCP Apps (SEP-1865). The spec is solid. The SDK is solid. But I hit two bugs that cost me most of a weekend each, and neither is documented anywhere I could find. Writing them up here in case they save someone else the time. Bug 1: resourceUri has two valid-looking locations, and only one works MCP Apps needs a way to tell the host "render this resource as a UI for this tool call." That pointer lives in _meta.ui.resourceUri . The question is: meta on what? I started with a parameterized resource template, ui://wingman/panel/{plan_name} , registered per plan. That was my first mistake. Parameterized templates get listed under resources/templates/list , not resources/list , and hosts do not prefetch or render anything from the templates list. The fix was straightforward once I found it: register one static resource, ui://wingman/panel , and pass the actual plan data through structuredContent on the tool result instead of baking it into the URI. That fix surfaced the real bug. My show_plan tool was returning a plain Python dict: return { " plan " : plan_data , " _meta " : { " ui " : { " resourceUri " : " ui://wingman/panel " }} } This looks correct. It is not. FastMCP's result conversion takes a returned dict and serializes the whole thing into structuredContent , verbatim, including any _meta key the dict happens to carry. So the actual wire result looked like this: result . structuredContent [ " _meta " ][ " ui " ][ " resourceUri " ] # == "ui://wingman/panel", but wrong place result . meta # None — this is what the host actually reads MCP Apps hosts read resourceUri off the top-level _meta on the CallToolResult , not off whatever ended up inside structuredContent . With that pointer effectively missing, the host had nowhere to bind the iframe. The visible symptom was strange: actions in the UI would update on screen but nothing persist

No fancy infrastructure. Just a 50-line Python function that picks the right model for the right query. Last month my LLM API bill hit $340. This month: $67. Same traffic. Same product. The only change was adding a simple router that stops sending every request to Claude Sonnet when GPT-4o mini can handle it just as well. Here's exactly how it works. The Problem When you prototype, you pick one model and hardcode it everywhere. Usually something capable like GPT-4o or Claude Sonnet, because you want good results fast. Then you ship, traffic grows, and you get a bill that makes you question your life choices. The thing is — not all queries need a flagship model. In a typical RAG app: "What is the return policy?" → GPT-4o mini handles this fine "Summarize these 5 conflicting documents and identify the key disagreement" → needs Sonnet You're paying Sonnet prices for return policy questions. That's the bug. The Fix: A Complexity Router import anthropic from openai import OpenAI openai_client = OpenAI() anthropic_client = anthropic.Anthropic() def classify_complexity(query: str) -> str: """Returns 'simple' or 'complex'.""" simple_indicators = [ len(query.split()) < 15, query.endswith("?") and query.count("?") == 1, not any(w in query.lower() for w in [ "compare", "analyze", "summarize", "explain why", "difference between", "pros and cons", "evaluate" ]) ] return "simple" if sum(simple_indicators) >= 2 else "complex" def route(query: str, context: str = "") -> str: complexity = classify_complexity(query) if complexity == "simple": # $0.15/M input — GPT-4o mini response = openai_client.chat.completions.create( model="gpt-4o-mini", messages=[ {"role": "system", "content": context}, {"role": "user", "content": query} ] ) return response.choices[0].message.content else: # $3.00/M input — Claude Sonnet (only when needed) response = anthropic_client.messages.create( model="claude-sonnet-4-6", max_tokens=1024, system=context, messages=[{"role": "user", "content": query}] ) retur

The Problem: Feature Velocity Was Creating Structural Debt The system originally started as a simple feature delivery backend: A Django API powering agricultural insights Celery workers handling asynchronous processing Independent endpoints for each new capability A growing set of Earth Observation computations (NDVI, NDWI, etc.) At first, it worked. But as more features were added, a pattern emerged: Each feature introduced its own pipeline logic Observability was inconsistent across services API contracts drifted between frontend and backend Debugging required tracing multiple disconnected systems We weren’t scaling functionality. We were scaling fragmentation. The Turning Point: Features vs Platforms The key realization was simple: Features solve user problems. Platforms solve system problems. We were repeatedly rebuilding: Authentication flows Data ingestion logic Processing pipelines API validation layers Monitoring hooks Each feature was solving its own version of these concerns. That is where platform engineering became necessary. The Shift: Introducing a Platform Layer We introduced a platform layer between feature delivery and infrastructure. Instead of building isolated pipelines, we standardized: 1. Unified API Surface All Earth Observation workflows (NDVI, NDWI, and future indices) were normalized into a consistent API contract. Shared request/response structure Versioned endpoints Schema validation through serializers Central routing logic This eliminated endpoint fragmentation. 2. Standardized Processing Pipeline Celery tasks were refactored into a reusable pipeline pattern: Ingestion Validation Computation Storage Publishing Instead of feature-specific workers, we moved toward composable tasks. This allowed new indices or processing logic to plug into the same execution flow. 3. Observability as a First-Class Layer One of the biggest failures in the original system was visibility. We introduced: Structured logging across all services Traceable job IDs

GitHub热门项目 | 🔰 Home Assistant Operating System | Stars: 7,212 | 37 stars this week | 语言: Python