今日已更新 405 条资讯 | 累计 33040 条内容
关于我们

How to Replicate MySQL to BigQuery with Sling

Fritz Larco 2026年08月18日 20:59 1 次阅读 来源:Dev.to

How to Replicate MySQL to BigQuery with Sling Last updated: July 2026 Getting MySQL data into BigQuery usually means picking a tradeoff. Hand-rolled scripts are cheap to start and expensive to keep alive once schemas drift. Managed connectors are quick to set up but bill per row and put your pipeline behind someone else's control plane. Sling sits in between: a single binary, a few lines of YAML, and a load path that uses BigQuery's own bulk ingest underneath. This guide walks through a real replication, end to end. Everything below — the row counts, the timings, the type mapping — comes from an actual run against a MySQL 8.4 source and a live BigQuery dataset. You can reproduce it. Installation Sling is a single binary with no runtime dependencies. Install it however suits your setup: # macOS / Linux curl -fsSL https://slingdata.io/install.sh | bash # Windows irm https://slingdata.io/install.ps1 | iex # Python pip install sling Confirm it's on your path: sling --version Connection setup Sling needs two connections: the MySQL source and the BigQuery target. Both can be set with sling conns set , which writes them to ~/.sling/env.yaml . MySQL source sling conns set mysql_source type = mysql host = 127.0.0.1 port = 3306 \ user = root password = mypass database = demo Or with a connection string: sling conns set mysql_source url = "mysql://root:mypass@127.0.0.1:3306/demo" BigQuery target BigQuery authenticates with a service-account key. The account needs BigQuery Data Editor and BigQuery Job User on the target project. sling conns set bigquery_target type = bigquery \ project = my-project dataset = demo \ key_file = /path/to/service-account.json If you have a Google Cloud Storage bucket handy, add gc_bucket=my-bucket . Sling will stage batches there and trigger a BigQuery load job from GCS, which is the fastest bulk path. Without a bucket, Sling stages locally and still loads in bulk — that's the setup used for every number in this guide. Test both connections sling c

本文内容来源于互联网,版权归原作者所有
查看原文