找到 795 篇相关文章
The AI buildout shows no signs of slowing. And with hundreds of billions of dollars a year going into data centers and GPUs, compute has become the single biggest cost for anyone building AI products. But for all that spending, there still isn’t a straightforward way to put a price on compute — or for firms to hedge their exposure when the price changes. Silicon Data […]
En el post anterior exploramos qué es un Data Lake y por qué son tan importantes en el ecosistema de datos actual. Ahora es momento de ensuciarnos las manos y ver exactamente qué servicios de AWS necesitamos para construir un Data Lake completamente serverless y cómo orquestarlos. Los Servicios Fundamentales Un Data Lake serverless en AWS se construye sobre cinco pilares fundamentales que trabajan en conjunto para crear una solución escalable y costo-eficiente: Storage Procesamiento Catalogo Seguridad Explotación Amazon S3 - El Corazón del Storage S3 no es solo nuestro sistema de archivos, es la piedra angular del Data Lake. Aquí almacenamos tanto los datos crudos como los procesados, y su organización es crucial para el rendimiento y los costos. Estructura de carpetas de un data lake estandar: data-lake-bucket/ ├── raw/ # Datos sin procesar │ ├── year=2024/ │ ├── month=12/ │ └── day=15/ ├── processed/ # Datos transformados │ ├── bronze/ # Limpieza básica │ ├── year=2024/ │ ├── month=12/ │ └── day=15/ │ ├── silver/ # Transformaciones de negocio │ ├── year=2024/ │ ├── month=12/ │ └── day=15/ │ └── gold/ # Datos listos para consumo │ ├── year=2024/ │ ├── month=12/ │ └── day=15/ └── athena-results/ # Resultados de queries Notarás que todo el data lake se encuentra en un mismo bucket, esto es lo más recomendable ya que S3 tiene un límite de 100 bucket que podemos crear por cuenta (no importa la región, ya que S3 es un servicio global) Configuraciones clave en S3: Versionado habilitado para auditoría y rollback Lifecycle policies para optimizar costos (Standard → IA → Glacier) Server-side encryption con KMS para seguridad si es necesario. Cross-region replication para disaster recovery AWS Glue - El Motor de Transformación Glue es suite de servicios de data serverless que maneja tanto el descubrimiento de esquemas como las transformaciones de datos. Componentes principales: Glue Jobs : Herramienta predilecta para ejecutar ETLs, nos permite procesar y transformar los dato
TerraPower's nuclear power plant possesses a strategic advantage over competitors, especially when chasing after data center deals.
WhatsApp is testing Scam Alert in limited beta, using on device machine learning to detect potential scam messages from non contacts. Meta's architecture keeps message content on the device while using confidential computing, Oblivious HTTP, differential privacy, and model transparency to measure performance and protect model delivery. By Leela Kumili
The cyberattack at CareCloud resulted in one of the largest reported data breaches in the U.S. healthcare industry this year.
ICE collected nearly a million DNA samples last year.
The people-search tool ClarityCheck says its reverse image search service is “private and secure”—but it left a database containing more than 9 million image files exposed.
Relativity Networks deals in hollow-core fiber, a rarely deployed technology that allows data to be transmitted 30% faster than conventional fiber.
In my last post I wrote about an error that cost me a full evening: my pom.xml had the MongoDB starter, but my code was still full of JPA annotations. The compiler kept saying cannot find symbol: class Entity . That post was about the error. This post is about the fix — every single line I had to change to move my Task Manager project from MySQL to MongoDB. If you are planning the same switch, this is the checklist I wish I had. 1. The dependency Before (MySQL + JPA): <dependency> <groupId> org.springframework.boot </groupId> <artifactId> spring-boot-starter-data-jpa </artifactId> </dependency> <dependency> <groupId> com.mysql </groupId> <artifactId> mysql-connector-j </artifactId> <scope> runtime </scope> </dependency> After (MongoDB): <dependency> <groupId> org.springframework.boot </groupId> <artifactId> spring-boot-starter-data-mongodb </artifactId> </dependency> One starter replaces two dependencies. And this is exactly where my problem started — I added the new one but never removed the old one, so half my code still compiled and half did not. Remove the JPA starter completely. If you leave it in, the jakarta.persistence annotations still resolve, and you will not notice you are mixing two worlds until something breaks at runtime. 2. application.properties Before: spring.datasource.url = jdbc:mysql://localhost:3306/taskmanager spring.datasource.username = root spring.datasource.password = yourpassword spring.jpa.hibernate.ddl-auto = update spring.jpa.show-sql = true After: spring.data.mongodb.uri = mongodb://localhost:27017/taskmanager Five lines became one. No ddl-auto because MongoDB has no schema to create. No dialect because there is no SQL being generated. The database and the collection are created automatically the first time you insert a document. 3. The model class This is where most of the work was. Here is my actual Task class after the migration: package com.taskmanager.task_manager ; import com.fasterxml.jackson.annotation.JsonIgnore ; import org.
Why plain k-fold silently overfits your trading model — and the 4-line fix that stops it. The Problem With k-Fold in Time Series Financial data is sequential. k-fold shuffles rows, so a training row from 2 PM Tuesday sits next to a test row from 10 AM Monday. Worse: triple-barrier labels overlap . A label at bar t looks 6 bars into the future; a training row at t+2 "knows" part of that future. The model leaks. V1's history is full of "HIGH overfit" verdicts — train AUC high, test AUC flat. Plain TimeSeriesSplit is only marginally better; it still lets adjacent windows bleed into each other. Purged + Embargoed CV For each test window [t0, t1] : Purge any train row whose label window overlaps the test window. Embargo max_training_horizon bars after the test window — drop those too. Overlapping labels are not i.i.d. Purging + embargoing makes the split honest. def purged_embargo_split ( n , n_splits = 5 , embargo_frac = 0.02 ): idx = np . arange ( n ) fold = np . array_split ( idx , n_splits ) splits = [] for i in range ( n_splits ): test = fold [ i ] emb = int ( len ( test ) * embargo_frac ) lo , hi = max ( 0 , test [ 0 ] - emb ), min ( n , test [ - 1 ] + emb + 1 ) train_mask = np . ones ( n , bool ); train_mask [ lo : hi ] = False splits . append (( idx [ train_mask ], test )) return splits Tune Only When You Have Enough Optuna once "won" a validation set with only 4 decisive rows — statistically meaningless. Rule: never tune when the decisive (non-abstained) validation rows are below ~30–50. Widen the date range or symbol basket first; don't trust the trial. Three-Way Split, Always train (fit) → validation (early stop + HP select) → disjoint calibration set (sigmoid/ isotonic) → test (untouched, final score only). V1 sometimes conflated validation and calibration. Keep them separate. The Promotion Gate Log every trial's train/val/test gap, not just the winner's test score. Promote only if replay AND shadow (≥1 live session) both beat baseline on buyer metrics : 1.5x
If you have ever copy-pasted a table out of a PDF, you already know what happens. Rows collapse into one long line of text. Columns interleave. Numbers land in the wrong cell, or no cell at all. The table on the page looks perfectly structured, but a PDF has no real concept of "table." It only knows where individual characters sit on a page. Every extraction tool, ours included, has to reconstruct the table from scratch, using nothing but the position of each word. That gap between "looks like a table" and "is structured data" is where almost every free PDF tool falls apart. Here is how we handle it, what actually works, and where it still doesn't. Two different jobs, two different tools PDFHaul splits this into two separate tools because they solve different problems. PDF to Excel rebuilds the whole document as a single spreadsheet, in the order it appears on the page: form labels, key-value pairs, section titles, and tables all together. It is for documents where you want the full content, not just the numbers, things like invoices, time sheets, and reports. Extract Tables does the opposite. It ignores everything that isn't a table and hands back one clean sheet per table, nothing else. It is for people who want structured data out, ready to sum, sort, and filter, not a copy of the document. Both tools share the same underlying geometry engine. The difference is what each one keeps and what it throws away. How Extract Tables actually decides what's a table The core problem with table extraction is that "looks tabular" and "is tabular" are not the same thing. A vector chart's axis box, a form's outlined signature field, and a two-column list of allergen names all produce something that a naive extractor will happily read as a grid. None of them are tables. Our pipeline handles this in four phases, all before anything is written to a spreadsheet: Phase 1: classify the page. Every page is scored as bordered (has ruled lines or filled-rectangle grid lines), stream (no
Polars hits the sweet spot between Pandas’ ease and Spark’s scale. If you’ve ever waited on a groupby or cursed a memory error, this cheat sheet is for you. I’ve pulled the patterns that save time in real pipelines, not just toy examples. Bookmark this before your next ETL run. Setup and Basics First, get Polars and a dataset. The lazy API is the default now, so you’ll rarely need to call .lazy() explicitly. Start with a CSV or Parquet file, or create a DataFrame from scratch. pip install polars pyarrow import polars as pl df = pl.read_csv('data.csv') # or pl.read_parquet() df = pl.DataFrame({'a': [1, 2], 'b': ['x', 'y']}) Selecting and Filtering Polars uses expressions, not strings. This feels odd at first but pays off when you chain operations. The syntax is consistent: every column is an expression you can transform, filter, or aggregate. df.select(['a', 'b']) # columns by name df.select(pl.col('a').alias('renamed')) df.filter(pl.col('a') > 10) df.filter(pl.col('b').is_in(['x', 'z'])) df.filter(pl.col('a').is_null()) Transforming Data Polars expressions are composable. You can nest them, reuse them, and even store them in variables. This is where the library shines over Pandas. df.with_columns(pl.col('a').cast(pl.Float64)) df.with_columns(pl.col('a').fill_null(0)) df.with_columns((pl.col('a') * 2).alias('a_doubled')) df.with_columns(pl.col('b').str.to_uppercase()) df.with_columns(pl.col('a').is_between(10, 20)) Grouping and Aggregations Groupbys in Polars are lazy by default. This means you can stack multiple aggregations without materializing intermediate results. The syntax is clean, but watch out for the order of operations. df.group_by('b').agg(pl.col('a').sum()) df.group_by('b').agg([pl.col('a').mean(), pl.col('a').max()]) df.group_by('b').agg(pl.col('a').quantile(0.9)) df.group_by_dynamic('timestamp', every='1d').agg(pl.col('a').sum()) Joins and Concatenation Joins in Polars are explicit. You’ll specify the join type and the columns to join on. Concatenatio
If you build with Prisma, you already know Prisma Studio. Run one command and you get a clean, visual way to browse and edit rows in your database. It's genuinely useful, and I reach for it every day while developing. But somewhere between "I need to look at my data" and "I need to let a support agent safely edit a customer's record in production," Prisma Studio quietly stops being the right tool. It was never trying to be that tool. It's a database viewer. An admin panel is something else, and the gap between the two is exactly the part that matters once real people and real permissions are involved. I ended up building a small package to fill that gap for my own Express + Prisma apps. Writing it forced me to be precise about what an admin panel actually adds on top of a database browser. Here's the distinction as I now understand it. A database browser shows rows. An admin panel governs them. Prisma Studio connects to your database and shows you everything. That's the point of it, and it's also why you'd never hand it to a non-engineer or expose it in production. It has no concept of who is looking, what they're allowed to do, or which rows they're allowed to touch. An admin panel's whole job is those three questions. The package I built mounts a React UI at /admin and a guarded JSON API under /admin/api/* on your existing Express app. Every single request through that API runs the same pipeline, in the same order: authentication → permission check → tenant scope → validation → Prisma mutation/query → optional audit event That ordering is the entire difference. A database browser skips straight to the mutation. An admin panel refuses to run the mutation until it knows the request is authenticated, permitted, scoped to the right tenant, and valid. Permissions and scope are two different questions This was the design decision I care most about, because collapsing these two into one is how data leaks happen. Permissions decide which actions a role may take. Can an ed
En el día a día del desarrollo de software y el análisis de datos, la preparación y limpieza de archivos financieros suele ser una de las tareas más repetitivas y propensas a errores. Tratar con layouts rígidos, filas desfasadas y nombres de columnas que cambian sin previo aviso genera una fricción operativa constante. Para resolver este problema de raíz—y manteniendo un enfoque estricto en la seguridad de la información—desarrollé D-MO (Data Micro-Optimizer) , una potente herramienta web de procesamiento ETL (Extract, Transform, Load) que corre completamente del lado del cliente. El Origen: Privacidad por Diseño Cuando manejamos reportes bancarios o información financiera sensible, subirlos a plataformas externas de conversión representa un riesgo crítico de seguridad. D-MO nació bajo la premisa de la privacidad absoluta: todo el procesamiento ocurre en la memoria local del navegador a través del cliente. Los datos estructurados jamás se envían a un servidor externo, eliminando latencias de red y garantizando un entorno de zero server overhead . Arquitectura del Pipeline (Flujo de Datos) El sistema procesa la información de manera secuencial a través de un flujo desacoplado, lo que permite transformar archivos complejos en datasets listos para producción en un solo clic: [ Archivo Local ] (.csv / .xlsx / .xlsb) │ ▼ ┌──────────────┐ │ DropZone │ ◄── Validación de Extensión y Tamaño └──────┬───────┘ │ (Buffer / Texto plano) ▼ ┌──────────────┐ │ File Parser │ ◄── Detección de delimitadores y headers dinámicos └──────┬───────┘ │ (JSON Normalizado) ▼ ┌──────────────┐ │ ETL Engine │ ◄── Reglas de Negocio, Mapeo de Alias y Filtros CUSTOM └──────┬───────┘ │ (Dataset Limpio) ▼ ┌──────────────┐ │ Export File │ ◄── Generación de reportes limpios listos └──────────────┘ Core Técnico y Capas del Sistema La aplicación está construida sobre Next.js 14 (App Router) y TypeScript , dividiendo su lógica interna en tres componentes principales: 1. Interfaz y Coordinación ( page.tsx )
I got an AWS question and implemented it to make sure that the option is correct. A critical financial application runs on RDS for PostgreSQL. The requirements are tight: 1-second RPO, 60-second RTO, and database credentials rotated every 30 days without taking the application offline. Two independent problems. Two independent solutions. Prerequisites Check these before running terraform apply : RDS Proxy availability RDS Proxy is not available on all instance types. It requires instances with at least 2 vCPUs. db.t3.micro is not supported. db.t3.medium and above work. Terraform executor permissions The IAM principal running Terraform needs, at minimum: rds:CreateDBInstance rds:CreateDBProxy rds:CreateDBProxyTargetGroup rds:RegisterDBProxyTargets rds:ModifyDBInstance iam:CreateRole iam:AttachRolePolicy iam:PutRolePolicy iam:PassRole secretsmanager:CreateSecret secretsmanager:PutSecretValue secretsmanager:RotateSecret lambda:CreateFunction lambda:AddPermission ec2:CreateSecurityGroup ec2:AuthorizeSecurityGroupIngress ec2:CreateDBSubnetGroup AdministratorAccess on the account covers all of these. Lock it down after the initial setup. VPC requirements RDS Proxy runs inside your VPC. You need at least two private subnets in different Availability Zones. The rotation Lambda also runs inside the VPC so it can reach the RDS instance directly during the credential update step. The problem Database failure recovery RPO of 1 second means almost no data loss is acceptable. RTO of 60 seconds means the application must resume within a minute of a failure. A standard single-instance RDS setup fails both requirements: there is no automatic failover, and restoring from a backup takes far longer than 60 seconds. Credential rotation Rotating credentials on a schedule sounds simple until you factor in application downtime. If you update a password and the application still holds connections authenticated with the old one, those connections fail. The rotation mechanism needs to handle
Cloudflare is introducing WriteGuard, now in private beta, to provide fine-grained security controls for MCP (Model Context Protocol) servers. It aims to make AI agents safer by controlling their access to tools that can modify data or perform actions, rather than simply read information. By Sergio De Simone
Netflix open-sourced an agentic workflow for Observational Causal Inference (OCI) that reduces toil in causal analysis. Given observational data and the human user's analysis plan, the agent uses an actor-critic loop to estimate causality, write a report, and suggest next steps. By Anthony Alford
How to Replicate MySQL to BigQuery with Sling Last updated: July 2026 Getting MySQL data into BigQuery usually means picking a tradeoff. Hand-rolled scripts are cheap to start and expensive to keep alive once schemas drift. Managed connectors are quick to set up but bill per row and put your pipeline behind someone else's control plane. Sling sits in between: a single binary, a few lines of YAML, and a load path that uses BigQuery's own bulk ingest underneath. This guide walks through a real replication, end to end. Everything below — the row counts, the timings, the type mapping — comes from an actual run against a MySQL 8.4 source and a live BigQuery dataset. You can reproduce it. Installation Sling is a single binary with no runtime dependencies. Install it however suits your setup: # macOS / Linux curl -fsSL https://slingdata.io/install.sh | bash # Windows irm https://slingdata.io/install.ps1 | iex # Python pip install sling Confirm it's on your path: sling --version Connection setup Sling needs two connections: the MySQL source and the BigQuery target. Both can be set with sling conns set , which writes them to ~/.sling/env.yaml . MySQL source sling conns set mysql_source type = mysql host = 127.0.0.1 port = 3306 \ user = root password = mypass database = demo Or with a connection string: sling conns set mysql_source url = "mysql://root:mypass@127.0.0.1:3306/demo" BigQuery target BigQuery authenticates with a service-account key. The account needs BigQuery Data Editor and BigQuery Job User on the target project. sling conns set bigquery_target type = bigquery \ project = my-project dataset = demo \ key_file = /path/to/service-account.json If you have a Google Cloud Storage bucket handy, add gc_bucket=my-bucket . Sling will stage batches there and trigger a BigQuery load job from GCS, which is the fastest bulk path. Without a bucket, Sling stages locally and still loads in bulk — that's the setup used for every number in this guide. Test both connections sling c
In this article, we'll explore the mechanisms to solve the coordination problem. 8. Introducing Leases To address the problem of permanent ownership, distributed systems typically replace it with temporary ownership. This concept is known as a lease . Instead of granting indefinite control over a resource, the coordination service assigns ownership for a limited period of time. Rather than stating, “You own this resource until you explicitly release it,” the system instead says, “You own this resource for the next 30 seconds.” This changes the interaction model significantly. Acquire Lease | v Execute Work | v Renew Lease | v Continue Processing As long as the application remains healthy, it periodically renews the lease to maintain ownership. If the application crashes or becomes unresponsive, it can no longer renew the lease. Once the lease duration expires, ownership is automatically revoked. At that point, another application becomes eligible to acquire the lease and continue the work. Leases solve a critical problem in distributed systems: they prevent abandoned locks from blocking progress indefinitely . The system can recover automatically without manual intervention. However, while leases improve availability, they also introduce a new class of subtle and more complex problems. Leases Depend on Time To understand the next challenge, assume the lease duration is thirty seconds. Application A successfully acquires the lease. Lease Granted Duration = 30 seconds After twenty seconds, the JVM begins a long Full Garbage Collection cycle. This pause lasts forty seconds, significantly longer than the lease duration. The timeline now becomes problematic. Lease Granted | | Processing | | GC Pause (40 sec) | | Lease Expires While Application A is paused, the lease expires. During this time, another application requests access to the same resource. The coordination service observes that the previous lease has expired and therefore grants ownership to Application B. Appl
Getting started with machine learning WEKA for Beginners: A Practical Introduction to Machine Learning Without Code Getting started with machine learning often means learning Python, libraries, datasets, and a lot of new terminology at the same time. WEKA offers a different approach. WEKA (Waikato Environment for Knowledge Analysis) is a machine-learning and data-mining workbench that lets you explore datasets and experiment with algorithms through a graphical interface. It is particularly useful for students and beginners who want to understand the machine-learning workflow before writing everything from scratch in code. What Can You Do With WEKA? WEKA provides tools for several common machine-learning tasks: Data preprocessing Classification Regression Clustering Association-rule mining Attribute selection Model evaluation Data visualization The Explorer interface is usually the best place for beginners to start. A typical workflow looks like: Dataset ↓ Preprocessing ↓ Feature Selection ↓ Algorithm ↓ Model Evaluation ↓ Interpretation Step 1: Load Your Dataset WEKA commonly works with ARFF (Attribute-Relation File Format) files, although it can also work with formats such as CSV. A simple ARFF dataset might look like: @relation students @attribute study_hours numeric @attribute attendance numeric @attribute passed {yes,no} @data 5,90,yes 2,60,no 8,95,yes 3,70,no The header describes the attributes, while the data section contains the individual instances. Understanding the structure of your dataset is important before applying any algorithm. Step 2: Preprocess the Data After loading the dataset, use WEKA's Preprocess section to inspect and prepare the data. You can examine: Attributes Number of instances Missing values Class distribution Attribute types WEKA also provides filters for operations such as removing attributes, handling missing values, normalization, and other transformations. Good preprocessing can have a significant impact on model performance. Step 3