标签：#spark

EVs account for under 10% of total new-vehicle sales in the US, and the numbers are declining. From a climate perspective, that’s pretty dismal, especially because the transportation sector is the single biggest source of greenhouse-gas emissions in the country. One thing that could help turn that around? Slate Auto’s new truck—a vehicle that seems…

There’s a hunt for new sources of hydrogen, and the gas (or at least the right conditions to make it) could be hiding beneath our feet. Hydrogen can be used as a fuel in everything from large trucks to planes to steelmaking. It’s often hailed as a climate solution because when burned, it produces water…

This summer has been a scorcher for much of the Northern Hemisphere. June and July marked the hottest two-month stretch in Europe since record-keeping began. The contiguous US endured its hottest month on record in July. South Korea saw its highest-ever recorded temperature. The heat isn’t over yet, but some scientists are already looking ahead…

I thought this would be a simple task. I already had a Python Kafka producer running. Kafka was up in Docker. The topic existed, and I could send a message into it successfully. The next step sounded straightforward: Python Producer ↓ Kafka ↓ Spark Structured Streaming All I wanted Spark to do was read a JSON message from a Kafka topic. Instead, I ran into one error after another. At first, it looked like one problem: Spark cannot read Kafka. It was not one problem. It turned into a chain of failures across several different layers: Python / PySpark ↓ Spark runtime ↓ Kafka connector ↓ Hadoop / Windows ↓ Docker ↓ Kafka networking ↓ Ivy dependency resolution The useful part of this experience was not any single fix. It was learning how to separate the layers and stop treating every error as a problem in my Python code. This is the full debugging path. What I Was Building This was part of an financial data engineering project. The batch side of the project already looked roughly like this: Financial Data Source ↓ Python ingestion ↓ AWS S3 ↓ Snowflake ↓ dbt ↓ Financial anomaly models I wanted to add a streaming extension for newly arriving financial events. For the first version, I kept it intentionally simple: Python Kafka Producer ↓ Kafka topic: financial_events ↓ Spark Structured Streaming The producer sent a simulated financial event: { "company_id" : "COMPANY_001" , "company_name" : "Sample Company" , "report_type" : "quarterly_report" , "reporting_date" : "2026-08-08" , "event_id" : "FIN-20260808-001" , "source" : "simulated_financial_event" } Kafka accepted the message successfully. I could even read it with Kafka's console consumer. So Kafka itself was working. Then Spark entered the picture. Failure #1: PySpark Worked, but spark-submit Didn't I installed PySpark: pip install pyspark Then I installed Java 17 and verified it: java -version After reopening my terminal, Java was available. I tested Spark directly through Python: python -c "from pyspark.sql import S

I've spent more than a decade building data pipelines, and the part nobody warns you about isn't the pipeline logic. It's the tuning. Executor memory, shuffle partitions, cluster size, thread counts. You pick numbers, ship it, and a few weeks later something breaks in a way that's obviously tuning-related but not obviously what to change . The pattern repeats enough times that you start recognizing it before you've even opened the logs. Job's slow, thousands of tiny shuffle tasks, someone way overestimated the partition count. Job dies on OOM, memory's set for last quarter's data volume, nobody updated it since. Cloud bill jumps, a cluster's been sized for peak load and just sits there mostly idle the other 20 hours a day. Every senior data engineer has this pattern-matching running in their head. It's tribal knowledge, and it lives in one or two people's heads on most teams, which means it doesn't scale and it definitely doesn't survive someone leaving. So I built a small tool to make that pattern-matching explicit instead of tribal: it reads your pipeline's config alongside its actual run metrics and tells you what's likely wrong, with the reasoning shown, not just a suggested number. Why rules instead of a model The obvious move in 2026 is to reach for an ML model. I didn't, and it wasn't because I don't think ML has a place here eventually. It's that for this specific problem, a handful of threshold rules already gets you most of the value, and they're something you can actually audit. If a rule fires, I can point at the exact condition and the exact number: average heap usage 28%, peak 47%, five runs, no OOM errors, therefore memory's over-provisioned, shrink it by roughly a fifth. That's checkable. You can look at your own metrics and see whether the reasoning holds. A model's confidence score doesn't give you that, and for something that's about to change a production config, I want the person approving it to be able to say "yes, I see why" rather than "the m

Intro Day 20! I lined up 10 AIs that turn a single photo into a few seconds of video. Half ran locally on my DGX Spark, half in the cloud 🐱 What I used: DGX Spark (LTX-2.3 / Wan 2.2) / 8 cloud models via fal.ai / ComfyUI / ffmpeg The setup Item Value Input One identical photo (my cat on a desk) Length 6 seconds Settings Identical The only variable The prompt Easy prompt The cat looks at the camera and meows once. It opens its mouth, meows, then closes it. Its tail flicks and its ears twitch. Hard prompt The cat stands upright on its hind legs in a kitchen, wearing a small apron, holding a knife in its front paws and chopping vegetables on a cutting board. Steam rises from a pot behind it. Please, just watch it Some of the cats came out with very long legs. Anyway. First half is the easy prompt, second half the hard one. On the easy prompt, local and cloud were a fair match . On the hard one... cloud, I think...! Three rankings below. Ranking 1: Time Time per 6-second clip on the hard prompt. Rank Model Where Time 🥇 LTX-2.3 Cloud 41s 🥈 Wan 2.7 Cloud 92s 🥉 Happy Horse 1.1 Cloud 97s 4 Veo 3.1 Cloud 128s 5 Kling 3 Pro Cloud 205s 6 Seedance 2.0 Cloud 210s 7 LTX-2.3 Local 315s 8 Wan 2.2 Local 651s 9 daVinci-MagiHuman Cloud 710s 10 HunyuanVideo 1.5 Cloud 796s A 19x spread. Look at 1st and 7th. Same model, LTX-2.3 , nearly the same resolution. The only difference is where it ran — 7.6x . Local setup DGX Spark (GB10, 128GB unified memory, ~273GB/s). ComfyUI headless, workflows over its API. LTX-2.3 is distilled fp8 at 8 steps. At 1088×1920 peak memory hit 77.8GB, about 60% of 128GB. That was the ceiling. Dropping to 512×768 finishes in 70s, but with one-fifth the pixels. Wan 2.2 is I2V-A14B fp8, 20 steps, 480×640. Higher resolution does not finish in reasonable time. Ranking 2: Cost Rank Model Per 6 seconds 🥇 Local Electricity only 🥈 LTX-2.3 (cloud) $0.36 🥉 Wan 2.7 $0.90 4 Kling 3 Pro $1.01 5 Happy Horse 1.1 $1.08 6 Veo 3.1 $2.40 7 Seedance 2.0 $4.09 — HunyuanVideo / MagiHum

In June 2024, a small company called Zanskar purchased a geothermal power plant in New Mexico that was failing fast. The water coming from the underground reservoir was getting colder by the day, making the plant uneconomical to run. Now, two years later, that plant is running at full capacity again, thanks to a new…

The problem with "just add more workers" Most Spark performance issues on Databricks aren't solved by scaling the cluster — they're caused by shuffle and skew , and no amount of extra nodes fixes a badly partitioned join. This post builds a realistic pipeline (order events joined against a small dimension table, aggregated, and written to Delta Lake) from the ground up, and uses it to work through: How Spark's shuffle actually behaves during a wide transformation Diagnosing and fixing data skew with salting and adaptive query execution (AQE) Laying out the resulting Delta table with Z-Ordering so downstream queries skip irrelevant files Governing access to the whole pipeline with Unity Catalog Architecture overview Pipeline shape — a batch job reading raw events, joining against a dimension table, aggregating, and writing to a governed Delta table: What happens inside a shuffle stage — this is the part most tutorials skip, and it's the key to understanding why skew hurts: Step 1 — Set up governed tables in Unity Catalog Everything downstream depends on tables being registered under Unity Catalog, which gives you centralized access control and lineage instead of per-workspace table grants. -- setup.sql, run in a Databricks SQL or notebook cell CREATE CATALOG IF NOT EXISTS retail_analytics ; CREATE SCHEMA IF NOT EXISTS retail_analytics . events ; CREATE TABLE IF NOT EXISTS retail_analytics . events . raw_orders ( order_id STRING , customer_id STRING , product_id STRING , quantity INT , event_ts TIMESTAMP ) USING DELTA LOCATION 'abfss://data@<storage-account>.dfs.core.windows.net/raw_orders' ; CREATE TABLE IF NOT EXISTS retail_analytics . events . dim_products ( product_id STRING , category STRING , unit_cost DOUBLE ) USING DELTA LOCATION 'abfss://data@<storage-account>.dfs.core.windows.net/dim_products' ; GRANT SELECT ON TABLE retail_analytics . events . raw_orders TO `analysts` ; Step 2 — Read and force a broadcast join for the small dimension table dim_products is s

On July 3, as a heat wave swept the region, New York State’s grid imported 52 gigawatt-hours of electricity from Canada—enough to meet about 9% of its total electricity demand that day. Some of that power shuttled in on a 339-mile power line stretching from Quebec to Queens called the Champlain Hudson Power Express (CHPE).…

It feels as if it should be illegal to even think about heating appliances during the height of summer—seriously, these heat waves in New York have been brutal—but we need to talk about heat pumps. The appliances use electricity for heating, they’re incredibly efficient, and they’re on the rise. (For what it’s worth, many heat…

Meta's pitch to users is Spark's ability to handle large agentic workloads, fix bugs, and help with large code migrations — the kind of automation that enterprises are increasingly turning to AI companies to provide.

I was really looking forward to July 4, and not just because I love a poolside barbecue. This year the American holiday also marked a big symbolic deadline for US nuclear power. Last year the Trump administration set a goal to see three new microreactors achieve criticality, a technical milestone establishing that a reactor can…

Google's 24/7 agentic assistant, Gemini Spark, comes to Mac alongside other improvements, like real-time tracking and support for more apps.

It’s been hard to look away from headlines about the European heat wave this week. Temperatures are breaking records across the continent, and the weather is threatening lives, shutting down schools, and in one particularly ironic case, forcing the cancellation of a London Climate Action Week event about extreme heat. As the summer ramps up…

Building point-in-time correct, production-grade feature pipelines — from raw Kafka events to online feature serving in milliseconds, using Spark Structured Streaming and the Databricks Feature Store. Table of Contents The Feature Engineering Problem Architecture Overview Feature Store Concepts: ERD Environment Setup Streaming Feature Pipeline Point-in-Time Correct Training Dataset Generation Writing Features to the Online Store Serving Features at Inference Time Feature Table Reference References The Feature Engineering Problem Feature engineering is where most ML projects silently fail in production. Not because the model is wrong — but because the features the model sees at training time are different from the features it sees at inference time . This is called training-serving skew , and it's the #1 silent killer of ML systems. Three specific failure modes cause it: Online/offline inconsistency — the batch pipeline that computes training features uses different logic than the real-time service that computes inference features Data leakage — training features accidentally include information from the future (e.g. joining on a label that was created after the event) Feature staleness — a model trained on 30-day rolling averages is served features that are 6 hours stale because the pipeline backfills are slow The Databricks Feature Store — now part of Unity Catalog as Feature Engineering in Unity Catalog — solves all three by: Storing feature computation logic alongside the data (no drift between training and serving) Enforcing point-in-time lookups during training dataset creation Providing a unified API for both batch offline reads and low-latency online reads Architecture Overview Feature Store Concepts: ERD Understanding the data model behind the Feature Store is essential for designing correct pipelines. Here's how the entities relate: The critical relationship: a Model Version is bound to a Training Set , which records exactly which feature tables and which p

A deep dive into how Spark transforms your SQL into a physical execution plan — and how Databricks layers Adaptive Query Execution and the Photon vectorized engine on top to squeeze out maximum performance. Table of Contents Why Query Optimization Matters The Catalyst Optimizer Pipeline Stage 1: Parsing — From SQL to Unresolved Logical Plan Stage 2: Analysis — Binding to the Catalog Stage 3: Logical Optimization — Rule-Based Rewrites Stage 4: Physical Planning — Strategies and Cost Models Adaptive Query Execution (AQE) The Photon Engine Reading Explain Plans Tuning Reference Table References Why Query Optimization Matters A Spark query written by a human and a Spark query executed by the engine are often very different things. The gap between them — the optimization — is what separates a job that runs in 3 minutes from one that runs in 3 hours on identical hardware. Databricks compounds Spark's native Catalyst optimizer with two additional layers: Adaptive Query Execution (AQE) — re-optimizes the query at runtime using actual statistics collected mid-job Photon — a C++ vectorized execution engine that replaces the JVM-based Spark executor for eligible operators Understanding all three lets you write queries that cooperate with the engine rather than fight it. The Catalyst Optimizer Pipeline Catalyst is Spark's rule-based and cost-based query optimizer. Every query — whether written in SQL, DataFrame API, or Dataset API — passes through the same four-stage pipeline before a single byte of data is read. Stage 1: Parsing — From SQL to Unresolved Logical Plan # ── Catalyst Stage 1: Parsing ───────────────────────────────────────────────── # Spark uses ANTLR4 to parse SQL into an Abstract Syntax Tree (AST). # At this point column names are NOT validated — the plan is "unresolved". from pyspark.sql import SparkSession spark = SparkSession . builder . appName ( " catalyst-demo " ). getOrCreate () # Both of these produce identical internal representations df_api = ( spark .

Solar geoengineering is often portrayed as a sort of emergency brake. Something along the lines of Pull in case of climate emergency to scatter light-reflecting particles to bounce sunlight out of the atmosphere and cool the planet. But it might be less like a simple brake and more like a complicated, entirely unsolved puzzle. Some…

It’s a tale of two nuclear industries. In China, large reactors are coming together at a stunning pace. The country has nearly doubled its nuclear fleet since 2016, reaching nearly 60 gigawatts of total power capacity. The new facilities are nearly all gigawatt-scale pressurized-water reactors. Meanwhile, the US has built just two reactors in that…

Would you take a payment to ramp down your electricity use? Would it change anything if you were doing so to help power a local data center? Google just signed a new deal to help pay for a virtual power plant (VPP) in the largest power grid in the US. The agreement is with Voltus,…

After migrating Spark pipelines to Azure Kubernetes Service, two infrastructure settings interacted destructively: spark.kubernetes.local.dirs.tmpfs=true backed shuffle spill with RAM instead of disk, and a hard podAffinity rule forced all executors onto one node. Together, they caused repeated OOM kills invisible to standard diagnostics. By Pranav Bhasker