找到 11 篇相关文章
At TechCrunch Disrupt 2026, the AI Stage is back to dig into the single hottest topic in the community for the past few years, presented by Google for Startups.
AI is expensive, Ali Ghodsi tells TechCrunch. With so many investors wanting into his latest round, he said yes to more than planned.
Most Snowflake-to-Databricks migrations get sold on cost and delivered on something else. The credit line item is what gets the project funded, but the teams that finish happy are usually the ones that moved for a different reason: they wanted ML, streaming and GenAI workloads living next to the analytics data instead of shuttling between two platforms. If your only justification is the bill, read the breakeven section below before you commit — the honest number is longer than the deck says. We're a Databricks shop , and we've written elsewhere about how to choose between the two platforms if you haven't committed yet. This post assumes you have. What actually changes underneath The two platforms look similar from a SQL console and are structurally different behind it. The mapping worth internalising before planning anything: Layer Snowflake Databricks Storage Proprietary micro-partitions inside Snowflake Delta Lake files in your own S3/ADLS/GCS bucket Compute Virtual warehouses, T-shirt sized Job clusters, all-purpose clusters, SQL Warehouses, Photon Governance Role hierarchy, row access policies, masking policies Unity Catalog across tables, models, notebooks, dashboards Sharing Secure Data Sharing Delta Sharing (open protocol) Billing unit Credits DBUs, priced differently per compute type The storage row is the one with the most downstream consequences. On Snowflake, storage and compute are separate line items on the same bill; on Databricks, storage is your cloud provider's problem and your cloud provider's invoice. That's a genuine benefit — the data stays readable by other engines — but it also means your "Databricks cost" and your "data platform cost" stop being the same number, and finance needs to know that before the first invoice arrives. Pick a strategy before you pick a tool Three patterns, and the choice determines everything after it: Lift-and-shift. Replicate schemas one-to-one, translate the SQL, cut over. Fastest, and it faithfully preserves every
Databricks is expanding its Lakebridge toolkit by introducing an agentic code conversion feature designed to help organizations migrate from legacy data warehouses. This new capability uses Genie Code to rewrite complex SQL scripts, allowing customers to transition their workloads to the Databricks lakehouse environment with higher efficiency and less manual intervention. Advanced Automation for Complex Code Translation The core of this update is the agentic code converter, a system that utilizes AI subagents to manage the heavy lifting of migration projects. These agents perform a variety of tasks including deep analysis of source code and the parallel conversion of multiple files. They also validate translated SQL and can autonomously retry sections that fail during the initial pass. This iterative approach is a significant step forward from traditional methods that often require human developers to step in when software hits a wall. By allowing developers to set specific migration rules for unique enterprise SQL structures, the tool provides a level of customization that previous automated systems lacked. The Lakebridge suite already offers several transpilation engines, such as the pattern-based BladeBridge technology and the compiler-based Morpheus engine. However, the addition of agentic AI introduces a reasoning layer that these older technologies do not possess. This reasoning is vital for moving beyond simple syntax mapping and into the realm of complex logic. Traditional transpilers like Morpheus are excellent at handling standard syntax mapping. They easily manage date functions and basic join commands. Problems arise when these tools encounter control-flow reasoning, cursors, or dynamic SQL that is generated at runtime. These complex elements often differ significantly across platforms like Oracle or Teradata. Industry experts note that these difficult sections usually represent about 15 percent of a codebase but consume the vast majority of manual labor
At TechCrunch Disrupt 2026, the AI Stage is back to dig into the single hottest topic in the community for the past few years, presented by Google for Startups.
Every data team eventually asks the same question: what runs our pipelines, on what schedule, with what retry logic, and who gets paged when it fails. The answer used to default to Airflow because there wasn't a real alternative. Now there are three reasonable defaults, and they optimize for different things. Picking wrong doesn't break anything on day one — it shows up eighteen months later as either an operations team drowning in scheduler maintenance or an engineering team fighting a platform that won't do what they need it to. Here's the actual tradeoff, not the vendor pitch version. Databricks Workflows: the path of least resistance, if you're all-in on Databricks Databricks Workflows is the orchestrator built into the platform. Jobs, clusters, Unity Catalog permissions, and Workflows all share the same control plane, which means you're not maintaining a separate scheduler, not managing a second set of credentials, and not debugging why an external system can't see a table that Unity Catalog says it can. Task dependencies, retries, cluster reuse across tasks, and job-level alerting all come for free. The cost is exactly what you'd expect from a platform-native tool: it orchestrates Databricks well and everything else poorly. There's no first-class way to trigger a task in your orchestration DAG that waits on a Salesforce export, calls an internal API, or coordinates a dbt run against a warehouse that isn't Databricks SQL. You can bolt these in with webhooks and external scripts, but you're fighting the tool rather than using it. Workflows also doesn't give you the asset-lineage or testing story that Dagster does — it schedules tasks, not data assets. If your data platform genuinely is Databricks end to end — ingestion, transformation, ML, serving — Workflows removes an entire category of operational overhead you'd otherwise be paying for nothing. Teams in this position who reach for Airflow anyway usually do it out of habit, not need, and end up running two sch
The problem with "just add more workers" Most Spark performance issues on Databricks aren't solved by scaling the cluster — they're caused by shuffle and skew , and no amount of extra nodes fixes a badly partitioned join. This post builds a realistic pipeline (order events joined against a small dimension table, aggregated, and written to Delta Lake) from the ground up, and uses it to work through: How Spark's shuffle actually behaves during a wide transformation Diagnosing and fixing data skew with salting and adaptive query execution (AQE) Laying out the resulting Delta table with Z-Ordering so downstream queries skip irrelevant files Governing access to the whole pipeline with Unity Catalog Architecture overview Pipeline shape — a batch job reading raw events, joining against a dimension table, aggregating, and writing to a governed Delta table: What happens inside a shuffle stage — this is the part most tutorials skip, and it's the key to understanding why skew hurts: Step 1 — Set up governed tables in Unity Catalog Everything downstream depends on tables being registered under Unity Catalog, which gives you centralized access control and lineage instead of per-workspace table grants. -- setup.sql, run in a Databricks SQL or notebook cell CREATE CATALOG IF NOT EXISTS retail_analytics ; CREATE SCHEMA IF NOT EXISTS retail_analytics . events ; CREATE TABLE IF NOT EXISTS retail_analytics . events . raw_orders ( order_id STRING , customer_id STRING , product_id STRING , quantity INT , event_ts TIMESTAMP ) USING DELTA LOCATION 'abfss://data@<storage-account>.dfs.core.windows.net/raw_orders' ; CREATE TABLE IF NOT EXISTS retail_analytics . events . dim_products ( product_id STRING , category STRING , unit_cost DOUBLE ) USING DELTA LOCATION 'abfss://data@<storage-account>.dfs.core.windows.net/dim_products' ; GRANT SELECT ON TABLE retail_analytics . events . raw_orders TO `analysts` ; Step 2 — Read and force a broadcast join for the small dimension table dim_products is s
📚 Series: Databricks on AWS (Part 0, prologue) The Target Architecture ← you are here Building a Databricks AI Platform on AWS RBAC with Function-Role Groups Compute Governance: Pools, Policies, Clusters The BOOTSTRAP_TIMEOUT Mystery Fixing It with AWS PrivateLink How We Structure the Terraform Before the build story, here's the destination. This is the target-state data architecture we designed the whole platform toward — the three principles that shaped every later decision, and the Unity Catalog governance model that keeps production data safe from human hands. The rest of this series is a build log: workspaces, RBAC, compute, the networking rabbit hole, the Terraform layout. But every one of those decisions was made in service of a target picture we drew first . This post is that picture — the "to-be" architecture, not the scaffolding we happened to have up on any given week. It's built on three things Databricks basically hands you if you lean into them: the Lakehouse (one store, ACID tables, no separate warehouse to sync), the Medallion architecture (raw → cleaned → integrated → business, each layer a promotion), and Unity Catalog as the single governance plane across all of it. The interesting part isn't reciting those three buzzwords — it's the specific way we wire them so that prod, dev, and analyst sandboxes never step on each other. Three principles, and everything follows Almost every concrete rule later in this series is a consequence of one of these three. 1. Nobody touches production by hand. Create, update, delete in prod data happens only through an automated, code-reviewed pipeline running as a service principal. Human accounts don't get write on prod — not analysts, not engineers, not admins. The blast radius of a bad afternoon is capped at whatever a person can do with read-only. This one principle is why the whole "promote" flow later exists. 2. Never copy production to look at it. If an analyst wants to explore the gold layer, they read it in p
Building point-in-time correct, production-grade feature pipelines — from raw Kafka events to online feature serving in milliseconds, using Spark Structured Streaming and the Databricks Feature Store. Table of Contents The Feature Engineering Problem Architecture Overview Feature Store Concepts: ERD Environment Setup Streaming Feature Pipeline Point-in-Time Correct Training Dataset Generation Writing Features to the Online Store Serving Features at Inference Time Feature Table Reference References The Feature Engineering Problem Feature engineering is where most ML projects silently fail in production. Not because the model is wrong — but because the features the model sees at training time are different from the features it sees at inference time . This is called training-serving skew , and it's the #1 silent killer of ML systems. Three specific failure modes cause it: Online/offline inconsistency — the batch pipeline that computes training features uses different logic than the real-time service that computes inference features Data leakage — training features accidentally include information from the future (e.g. joining on a label that was created after the event) Feature staleness — a model trained on 30-day rolling averages is served features that are 6 hours stale because the pipeline backfills are slow The Databricks Feature Store — now part of Unity Catalog as Feature Engineering in Unity Catalog — solves all three by: Storing feature computation logic alongside the data (no drift between training and serving) Enforcing point-in-time lookups during training dataset creation Providing a unified API for both batch offline reads and low-latency online reads Architecture Overview Feature Store Concepts: ERD Understanding the data model behind the Feature Store is essential for designing correct pipelines. Here's how the entities relate: The critical relationship: a Model Version is bound to a Training Set , which records exactly which feature tables and which p
A deep dive into how Spark transforms your SQL into a physical execution plan — and how Databricks layers Adaptive Query Execution and the Photon vectorized engine on top to squeeze out maximum performance. Table of Contents Why Query Optimization Matters The Catalyst Optimizer Pipeline Stage 1: Parsing — From SQL to Unresolved Logical Plan Stage 2: Analysis — Binding to the Catalog Stage 3: Logical Optimization — Rule-Based Rewrites Stage 4: Physical Planning — Strategies and Cost Models Adaptive Query Execution (AQE) The Photon Engine Reading Explain Plans Tuning Reference Table References Why Query Optimization Matters A Spark query written by a human and a Spark query executed by the engine are often very different things. The gap between them — the optimization — is what separates a job that runs in 3 minutes from one that runs in 3 hours on identical hardware. Databricks compounds Spark's native Catalyst optimizer with two additional layers: Adaptive Query Execution (AQE) — re-optimizes the query at runtime using actual statistics collected mid-job Photon — a C++ vectorized execution engine that replaces the JVM-based Spark executor for eligible operators Understanding all three lets you write queries that cooperate with the engine rather than fight it. The Catalyst Optimizer Pipeline Catalyst is Spark's rule-based and cost-based query optimizer. Every query — whether written in SQL, DataFrame API, or Dataset API — passes through the same four-stage pipeline before a single byte of data is read. Stage 1: Parsing — From SQL to Unresolved Logical Plan # ── Catalyst Stage 1: Parsing ───────────────────────────────────────────────── # Spark uses ANTLR4 to parse SQL into an Abstract Syntax Tree (AST). # At this point column names are NOT validated — the plan is "unresolved". from pyspark.sql import SparkSession spark = SparkSession . builder . appName ( " catalyst-demo " ). getOrCreate () # Both of these produce identical internal representations df_api = ( spark .
Enterprise AI is entering a different phase now, one where enterprises are no longer evaluating whether AI is exciting. They are evaluating whether it is safe to deploy broadly.