标签：#bigquery

Turn Google Sheets into a Fully Interactive, Differential ACID Mutation Console for Apache Iceberg without Reverse ETL SaaS or Cloud Servers. Hero Infographic: Interactive Bidirectional Lakehouse Writeback via Google Sheets & Apache Iceberg. Enables business operators to query filtered records from an open Apache Iceberg table on Google Cloud Storage, visually edit values, add new rows, or purge obsolete records directly within a Google Sheets grid with an embedded dark-themed console, and commit atomic, microsecond-tolerant ACID mutations back to Parquet storage via BigQuery without Reverse ETL SaaS or persistent servers. Structural Analysis of the Hero Infographic: The hero infographic illustrates the complete, self-contained operational loop connecting frontline spreadsheet agility with immutable open lakehouse storage across three interconnected stages: 1. Predicate Query (Apache Iceberg Open Lakehouse on GCS) : The left section shows the enterprise analytical foundation hosted on Google Cloud Storage, where Apache Iceberg manages immutable Parquet data files, hierarchical Avro metadata, and commit snapshots. When a user requests high-value records, BigQuery acts as an on-demand distributed compute accelerator, executing SQL queries with predicate pushdown (e.g., SELECT * WHERE price > 1000 ORDER BY id ASC ) to fetch precise subsets in sub-seconds. 2. Frontline Editing in Google Sheets (Intuitive Operational Experience) : The central section features a modern, user-friendly Google Sheets grid docked with the sleek dark-themed Iceberg Lakehouse Console sidebar. A business user effortlessly modifies data on the grid with immediate visual feedback: modifying existing values (e.g., updating price from 1500 to 123 ), appending new rows with unique primary keys ( + ADD (New Row id:121) ), and deleting obsolete rows ( 🗑️ DELETE (Removed id:104) ). Native cell validation guarantees data cleanliness, while a Privacy Mode toggle ( [🔒 Privacy: ON] ) automatically masks sen

How to Replicate MySQL to BigQuery with Sling Last updated: July 2026 Getting MySQL data into BigQuery usually means picking a tradeoff. Hand-rolled scripts are cheap to start and expensive to keep alive once schemas drift. Managed connectors are quick to set up but bill per row and put your pipeline behind someone else's control plane. Sling sits in between: a single binary, a few lines of YAML, and a load path that uses BigQuery's own bulk ingest underneath. This guide walks through a real replication, end to end. Everything below — the row counts, the timings, the type mapping — comes from an actual run against a MySQL 8.4 source and a live BigQuery dataset. You can reproduce it. Installation Sling is a single binary with no runtime dependencies. Install it however suits your setup: # macOS / Linux curl -fsSL https://slingdata.io/install.sh | bash # Windows irm https://slingdata.io/install.ps1 | iex # Python pip install sling Confirm it's on your path: sling --version Connection setup Sling needs two connections: the MySQL source and the BigQuery target. Both can be set with sling conns set , which writes them to ~/.sling/env.yaml . MySQL source sling conns set mysql_source type = mysql host = 127.0.0.1 port = 3306 \ user = root password = mypass database = demo Or with a connection string: sling conns set mysql_source url = "mysql://root:mypass@127.0.0.1:3306/demo" BigQuery target BigQuery authenticates with a service-account key. The account needs BigQuery Data Editor and BigQuery Job User on the target project. sling conns set bigquery_target type = bigquery \ project = my-project dataset = demo \ key_file = /path/to/service-account.json If you have a Google Cloud Storage bucket handy, add gc_bucket=my-bucket . Sling will stage batches there and trigger a BigQuery load job from GCS, which is the fastest bulk path. Without a bucket, Sling stages locally and still loads in bulk — that's the setup used for every number in this guide. Test both connections sling c

El 12 de junio de 2026, Google Cloud publicó el Open Knowledge Format (OKF) , una especificación abierta que intenta resolver un problema que suena aburrido pero es carísimo: cómo darle a un agente de IA el contexto que necesita para no inventar. La propuesta es tan simple que da un poco de desconfianza —una carpeta de archivos Markdown con un encabezado YAML— y esa simpleza es, al mismo tiempo, su mayor virtud y el blanco de todas las críticas. Vale la pena entender qué anuncian, porque detrás del formato aparentemente trivial hay una apuesta bastante ambiciosa sobre cómo van a compartir conocimiento las empresas en la era de los agentes. El problema: el conocimiento vive en silos En casi cualquier organización, lo que un modelo necesita saber está desparramado y encerrado en formatos incompatibles: catálogos de metadatos con APIs propietarias, wikis internas, comentarios de código, docstrings, celdas de notebooks y —el clásico— la cabeza de dos o tres ingenieros senior. Cuando un agente tiene que responder algo tan concreto como "¿cómo calculo los usuarios activos semanales a partir del stream de eventos?" , tiene que ensamblar la respuesta juntando pedacitos de superficies que no se hablan entre sí. El resultado: cada equipo que arma un agente resuelve el mismo rompecabezas desde cero, y el conocimiento queda preso del sistema que lo generó. No hay portabilidad. La propuesta: un formato, no una plataforma La respuesta de Google no es "otro servicio de conocimiento en la nube" —y ese es el punto que más recalcan—. Es un formato . OKF v0.1 representa el conocimiento como: Solo Markdown : legible en cualquier editor, renderizable en GitHub, indexable por cualquier buscador. Solo archivos : se transporta como un tarball, se hospeda en cualquier repo git, se monta en cualquier filesystem. Solo frontmatter YAML : campos consultables como type , title , description , resource , tags y timestamp . Cada "concepto" (una tabla, un dataset, una métrica, un runbook) es un arc

It's important to monitor Dataform - jobs executed by Dataform can be the primary source of BigQuery costs in a modern data platform. Forgetting to incrementalise a table, using a table instead of a view in the wrong place or performing complex window functions on a large table can all incur large costs and long run times. Using the WorkflowInvocationAction for each job we can extract its BigQuery Job ID, then extract key metadata for each BigQuery job by querying INFORMATION_SCHEMA.JOBS_BY_PROJECT , before writing the output back to BigQuery so that it can be analysed (maybe even by transforming it in Dataform). from google.cloud import dataform_v1 from google.cloud import bigquery from datetime import datetime # ------------------------------------------------------------ # CONFIG # ------------------------------------------------------------ PROJECT_ID = " my-project " REGION = " europe-west2 " REPOSITORY_ID = " analytics " WORKFLOW_INVOCATION_ID = " 123456789 " BQ_REGION = " region-europe-west2 " OUTPUT_TABLE = " my-project.raw_dataform_monitoring.raw_dataform_bigquery_metrics " # ------------------------------------------------------------ # CLIENTS # ------------------------------------------------------------ dataform = dataform_v1 . DataformClient () bq = bigquery . Client ( project = PROJECT_ID ) repository = dataform . repository_path ( PROJECT_ID , REGION , REPOSITORY_ID ) invocation_name = f " { repository } /workflowInvocations/ { WORKFLOW_INVOCATION_ID } " # ------------------------------------------------------------ # 1. GET WORKFLOW INVOCATION ACTIONS → EXTRACT JOB IDS # ------------------------------------------------------------ job_ids = set () actions = dataform . list_workflow_invocation_actions ( parent = invocation_name ) for action in actions : # only BigQuery actions contain job metadata if hasattr ( action , " bigquery_action " ) and action . bigquery_action : if action . bigquery_action . job_id : job_ids . add ( action . bigquery_action