标签：#Query

How to Replicate MySQL to BigQuery with Sling Last updated: July 2026 Getting MySQL data into BigQuery usually means picking a tradeoff. Hand-rolled scripts are cheap to start and expensive to keep alive once schemas drift. Managed connectors are quick to set up but bill per row and put your pipeline behind someone else's control plane. Sling sits in between: a single binary, a few lines of YAML, and a load path that uses BigQuery's own bulk ingest underneath. This guide walks through a real replication, end to end. Everything below — the row counts, the timings, the type mapping — comes from an actual run against a MySQL 8.4 source and a live BigQuery dataset. You can reproduce it. Installation Sling is a single binary with no runtime dependencies. Install it however suits your setup: # macOS / Linux curl -fsSL https://slingdata.io/install.sh | bash # Windows irm https://slingdata.io/install.ps1 | iex # Python pip install sling Confirm it's on your path: sling --version Connection setup Sling needs two connections: the MySQL source and the BigQuery target. Both can be set with sling conns set , which writes them to ~/.sling/env.yaml . MySQL source sling conns set mysql_source type = mysql host = 127.0.0.1 port = 3306 \ user = root password = mypass database = demo Or with a connection string: sling conns set mysql_source url = "mysql://root:mypass@127.0.0.1:3306/demo" BigQuery target BigQuery authenticates with a service-account key. The account needs BigQuery Data Editor and BigQuery Job User on the target project. sling conns set bigquery_target type = bigquery \ project = my-project dataset = demo \ key_file = /path/to/service-account.json If you have a Google Cloud Storage bucket handy, add gc_bucket=my-bucket . Sling will stage batches there and trigger a BigQuery load job from GCS, which is the fastest bulk path. Without a bucket, Sling stages locally and still loads in bulk — that's the setup used for every number in this guide. Test both connections sling c

This isn't a syntax error or a bug in the query — it's Power Query's Formula Firewall refusing to combine data from more than one source until it knows whether that's actually safe. Combining a private/organizational source with a public one (an internal database and a public web API, for example) can leak data from one into the other; the firewall blocks it by default rather than guessing. Why This Exists Every data source in Power Query has a privacy level — Public, Organizational, or Private — set the first time it's connected to. When a query's steps end up needing to send data from one source into a call against a different source, Power Query checks whether the privacy levels involved allow that combination. Originally published on PBIDocs — Power BI documentation covering DAX, Power Query, data modeling, and Microsoft Fabric.

Querydsl is one of those libraries that everyone used for years and then quietly stopped updating. The 5.0 release has been "coming soon" since 2019. The GitHub shows commits but no milestone. The issue tracker has a thread titled "Is Querydsl dead?" with hundreds of comments. It's not dead. But if you're starting a new project in 2026 and you're picking between Querydsl and something that's actively maintained, has Spring Boot 4 support, works with MongoDB and in-memory collections, generates OpenAPI docs automatically, and has companion frontend libraries... well, you see where I'm going. This isn't a "Querydsl bad, Spring Filter good" article. Querydsl pioneered type-safe querying for Java and it deserves credit. But migrations happen, and if you're considering one, here's what the conversion looks like. Side-by-side: basic filtering Querydsl: QCar car = QCar . car ; BooleanExpression filter = car . year . gt ( 2020 ) . and ( car . km . lt ( 50000 )) . and ( car . color . eq ( Color . RED )); List < Car > results = new JPAQuery <>( entityManager ) . select ( car ) . from ( car ) . where ( filter ) . fetch (); Spring Filter (query string): @Filter Specification < Car > spec // URL: ?filter=year > 2020 and km < 50000 and color : 'red' List < Car > results = carRepo . findAll ( spec ); Spring Filter (programmatic builder): FilterNode filter = fb . field ( "year" ). greaterThan ( fb . input ( 2020 )) . and ( fb . field ( "km" ). lessThan ( fb . input ( 50000 ))) . and ( fb . field ( "color" ). equal ( fb . input ( Color . RED ))) . get (); Specification < Car > spec = converter . convert ( filter ); List < Car > results = carRepo . findAll ( spec ); Spring Filter (type-safe builder): FilterNode f = CarFilter . where ( fb ) . year (). greaterThan ( 2020 ) . and () . km (). lessThan ( 50000 ) . and () . color (). equal ( Color . RED ) . build (); Specification < Car > spec = converter . convert ( f ); List < Car > results = carRepo . findAll ( spec ); The type-safe bui

El 12 de junio de 2026, Google Cloud publicó el Open Knowledge Format (OKF) , una especificación abierta que intenta resolver un problema que suena aburrido pero es carísimo: cómo darle a un agente de IA el contexto que necesita para no inventar. La propuesta es tan simple que da un poco de desconfianza —una carpeta de archivos Markdown con un encabezado YAML— y esa simpleza es, al mismo tiempo, su mayor virtud y el blanco de todas las críticas. Vale la pena entender qué anuncian, porque detrás del formato aparentemente trivial hay una apuesta bastante ambiciosa sobre cómo van a compartir conocimiento las empresas en la era de los agentes. El problema: el conocimiento vive en silos En casi cualquier organización, lo que un modelo necesita saber está desparramado y encerrado en formatos incompatibles: catálogos de metadatos con APIs propietarias, wikis internas, comentarios de código, docstrings, celdas de notebooks y —el clásico— la cabeza de dos o tres ingenieros senior. Cuando un agente tiene que responder algo tan concreto como "¿cómo calculo los usuarios activos semanales a partir del stream de eventos?" , tiene que ensamblar la respuesta juntando pedacitos de superficies que no se hablan entre sí. El resultado: cada equipo que arma un agente resuelve el mismo rompecabezas desde cero, y el conocimiento queda preso del sistema que lo generó. No hay portabilidad. La propuesta: un formato, no una plataforma La respuesta de Google no es "otro servicio de conocimiento en la nube" —y ese es el punto que más recalcan—. Es un formato . OKF v0.1 representa el conocimiento como: Solo Markdown : legible en cualquier editor, renderizable en GitHub, indexable por cualquier buscador. Solo archivos : se transporta como un tarball, se hospeda en cualquier repo git, se monta en cualquier filesystem. Solo frontmatter YAML : campos consultables como type , title , description , resource , tags y timestamp . Cada "concepto" (una tabla, un dataset, una métrica, un runbook) es un arc

We opened Chapter 1 with a single line, SELECT * FROM users WHERE id = 1 . For that line to leave the client and come back as a result row, the PostgreSQL backend went through five stages. First it decided which processing path the message should take; then the parser and analyzer turned the text into a tree and gave it meaning from the catalog. The rewriter expanded views and injected policies to transform the tree, the planner weighed the possible execution paths by cost and picked the cheapest one, and the executor followed that plan, pulling up one tuple at a time and sending them back to the client. Chapter 1 was a story about how a query is processed . What tree a given SQL becomes, what plan it turns into, in what order it runs. From start to finish, a chain of logical transformations. But what every one of those stages ultimately deals with is data. The executor pulls up tuples, yet where on disk those tuples lie and in what shape, how they come up into memory, Chapter 1 never asked. When the planner judged an index scan cheaper than a sequential scan, it never opened up what that index physically is. Chapter 1 followed only the logical journey of a query, leaving untouched the substance of the data that journey stands on. Chapter 2, Storage & Access Methods, opens up that substance. In what unit data sits on disk (page), where disk and memory meet (buffer manager), where and how a row survives (heap), and how that row is found quickly (B-tree and the specialized indexes). The very tuple the planner weighed by cost and the executor pulled up in Chapter 1, where it actually came from and how it came to be there, is what Chapter 2 reveals. If Chapter 1 was the logical life of a query, Chapter 2 is the physical dwelling of data. We now look at how the data a query reaches for actually lives on disk.

Pinterest introduced MIQPS, a URL normalization system that identifies which query parameters affect page identity using rendered content fingerprints. It reduces duplicate processing across millions of domains by replacing rule-based approaches with offline analysis, anomaly detection, and runtime parameter maps, improving ingestion efficiency and scalability in large-scale content pipelines. By Leela Kumili

It's important to monitor Dataform - jobs executed by Dataform can be the primary source of BigQuery costs in a modern data platform. Forgetting to incrementalise a table, using a table instead of a view in the wrong place or performing complex window functions on a large table can all incur large costs and long run times. Using the WorkflowInvocationAction for each job we can extract its BigQuery Job ID, then extract key metadata for each BigQuery job by querying INFORMATION_SCHEMA.JOBS_BY_PROJECT , before writing the output back to BigQuery so that it can be analysed (maybe even by transforming it in Dataform). from google.cloud import dataform_v1 from google.cloud import bigquery from datetime import datetime # ------------------------------------------------------------ # CONFIG # ------------------------------------------------------------ PROJECT_ID = " my-project " REGION = " europe-west2 " REPOSITORY_ID = " analytics " WORKFLOW_INVOCATION_ID = " 123456789 " BQ_REGION = " region-europe-west2 " OUTPUT_TABLE = " my-project.raw_dataform_monitoring.raw_dataform_bigquery_metrics " # ------------------------------------------------------------ # CLIENTS # ------------------------------------------------------------ dataform = dataform_v1 . DataformClient () bq = bigquery . Client ( project = PROJECT_ID ) repository = dataform . repository_path ( PROJECT_ID , REGION , REPOSITORY_ID ) invocation_name = f " { repository } /workflowInvocations/ { WORKFLOW_INVOCATION_ID } " # ------------------------------------------------------------ # 1. GET WORKFLOW INVOCATION ACTIONS → EXTRACT JOB IDS # ------------------------------------------------------------ job_ids = set () actions = dataform . list_workflow_invocation_actions ( parent = invocation_name ) for action in actions : # only BigQuery actions contain job metadata if hasattr ( action , " bigquery_action " ) and action . bigquery_action : if action . bigquery_action . job_id : job_ids . add ( action . bigquery_action