标签：#data

Most LlamaIndex setups end up with two separate backends once you go beyond plain vector search: a vector store for VectorStoreIndex , and a separate graph database for PropertyGraphIndex when you need relationship-aware retrieval (GraphRAG). Two services, two connection strings, two things to keep in sync. This is a walkthrough of backing both index types with SynapCores instead — one engine, one connection, both index types. Setup docker run -d --name synapcores -p 8080:8080 \ -e AIDB_ACCEPT_LICENSE = 1 \ -v synapcores-data:/var/lib/synapcores \ ghcr.io/synapcores/community:latest pip install llama-index llama-index-vector-stores-synapcores llama-index-graph-stores-synapcores Both integration packages are independently published on PyPI: llama-index-vector-stores-synapcores llama-index-graph-stores-synapcores Vector store — standard RAG from llama_index.core import VectorStoreIndex , StorageContext , Document from llama_index.vector_stores.synapcores import SynapCoresVectorStore vector_store = SynapCoresVectorStore ( uri = " http://localhost:8080 " , embedding_dim = 1536 ) storage_context = StorageContext . from_defaults ( vector_store = vector_store ) docs = [ Document ( text = " SynapCores runs vector search, graph traversal, and SQL in one engine. " )] index = VectorStoreIndex . from_documents ( docs , storage_context = storage_context ) query_engine = index . as_query_engine () response = query_engine . query ( " What does SynapCores combine into one engine? " ) print ( response ) The vector store implements the full BasePydanticVectorStore ABC — add , delete , query , delete_nodes , clear , plus the async surface. Metadata filtering supports the full MetadataFilters grammar: all 12 operators ( EQ , NE , GT / GTE / LT / LTE , IN , NIN , TEXT_MATCH , TEXT_MATCH_INSENSITIVE , CONTAINS , IS_EMPTY ) with AND / OR / NOT and nested groups — so you're not giving up filtering power by moving off a dedicated vector DB. If you already have data in SynapCores from a prev

The UK’s energy regulator is using a variety of tricks to keep speculative data center projects from plugging into the power grid. The country’s AI ambitions hang in the balance.

Short answer: use a scheduled cleanup endpoint when one indexed, bounded pass can finish predictably; use a queue when cleanup must be divided into independently retriable batches. For a fintech SaaS that fans out shipment updates to many subscribers, latency and cost should be judged at the system boundary: a cheap cleanup run is not a good bargain if it contends with delivery or leaves retention evidence incomplete. The first design decision is to keep shipment fan-out separate from retention work. A shipment update has a latency-sensitive path. Expired subscriptions, old delivery attempts, and temporary fan-out records usually have a policy-driven path. They may share a database, but they should not share an unbounded transaction or an execution budget. This distinction matters more than the spelling of a cron expression. It also gives the team a useful test: can the cleanup be repeated safely while the shipment update path continues to make progress? How should a Node.js SaaS choose a cron or queue for scheduled cleanup? Measure the worst case first. Count eligible records by tenant, check the relevant index, estimate lock pressure, and measure a bounded pass while the database is serving normal shipment traffic. The median duration is not the decision variable; the tail is. A scheduled data cleanup is a good fit for one HTTP-triggered run when its cutoff, tenant scope, batch size, and completion state can be recorded and the run has room to finish before its execution limit. The cutoff should be computed by the application and persisted with the run. A schedule has jitter, and a paused schedule may not replay every missed invocation. “Delete records older than the cutoff captured at run start” is therefore more auditable than silently recalculating the boundary for every page. The query should also exclude legal holds, active disputes, and any retention exception required by the business policy. Keep it bounded. The boundary is operational. When a tenant can mo

Today's two tasks are both about a new base. A feature branch that needs to sit on top of a master that has moved. A database instance that needs to come back from a snapshot taken when things were fine. In each case, the interesting question is the same: what carries over, and what do you have to say out loud? One Git task, one AWS task. Rebase a feature branch onto master without creating a merge commit, then snapshot an RDS instance and restore it into a new one. The tasks come from the KodeKloud Engineer platform. Rebase: not moving commits, replaying them The requirement was specific, and the specificity is the lesson. A developer's feature branch was behind master. Bring it up to date without losing any feature work, and without a merge commit. That second clause rules out git merge master . Merge joins two histories and records the join, which is the merge commit. Rebase does something else entirely. cd /usr/src/kodekloudrepos/media git branch git log --oneline --graph --all --decorate git checkout feature git rebase master git log --oneline --graph --decorate Git's own documentation describes what happens under git rebase master : it lists the commits on your branch that are not on master, checks out master, and then replays each of your commits on top of it, one at a time, in a way it compares to running git cherry-pick for each one. Replays. Not moves. Every commit that comes out the other side has a new hash, because a commit's identity includes its parent, and the parent is different now. Your work is preserved, the commits carrying it are not the same objects they were. That is exactly why there is no merge commit. Rebase does not join two histories, it rewrites yours so it looks like it was always based on master's current tip. You get a straight line, at the cost of a history that is no longer a record of what actually happened. Two things I had to be deliberate about. Direction. Rebase applies to the branch you are standing on and takes the branch yo

More than 15 politicians from across the country have signed on to the AI Pact, vowing to regulate data centers and AI. “We’ve got to get this right,” says Senate candidate Dan Osborn of Nebraska.

O NutriApp é um projeto de estudos: plataforma de saúde conectando pacientes, nutricionistas, médicos e personal trainers, cada perfil enxergando só o que sua permissão libera. Stack: React 19 + TypeScript, TanStack Start (SSR, rotas file-based e server functions), Tailwind v4 + shadcn/ui, react-hook-form + Zod para formulários tipados, TanStack Query para cache, e Lovable Cloud (Supabase) com Postgres e Row Level Security. O maior desafio foi o controle de acesso por papéis. Três tabelas centrais — profiles, user_roles e pacientes — todas com RLS ativado. Paciente lê só seus próprios registros; profissionais e administradores enxergam todos os pacientes. Pra evitar recursão de política (problema clássico de RLS), criei funções SECURITY DEFINER como has_role e is_profissional, quebrando o ciclo de verificação. Autenticação e segurança: Login por email/senha, com rota administrativa separada (/admin/login) Server functions protegidas com requireSupabaseAuth, checando papel antes de qualquer ação administrativa Validação client-side com Zod: senha entre 6-72 caracteres, email até 255, telefone opcional Usuários criados por admin já nascem confirmados e ativos, reduzindo fricção operacional Automação como diferencial: o perfil de saúde calcula IMC em tempo real e gera um plano inicial baseado no objetivo selecionado (emagrecimento, ganho de massa ou controle de patologias) — reduzindo trabalho manual do profissional. Aprendizados principais: RLS bem modelado desde o início evita gambiarra depois — pensar em papéis antes da primeira quere economiza retrabalho. Verificação de papel precisa estar no backend, nunca só na UI. Separar login de paciente/profissional do login admin simplifica segurança e UX ao mesmo tempo.

The site, called Stud or Dud, helps daters dig up dirt on potential paramours. It’s fueled by the same public data as PeopleFinders.com—and comes with many of the same concerns.

In this article, author Srikanth Mamidala discusses the data lake architecture used for analytics, reporting, and machine learning and shows how to manage the consumer lag metrics when using Kafka and Apache Hudi. By Srikanth Mamidala

Diagrid Catalyst 2.0 applies Dapr-based recovery, signed workflow history and execution attestation across several agent frameworks. Architects should compare it with framework-native durability and established workflow engines, while evaluating benchmark evidence and operational trade-offs. By Mark Silvester

The schema catalog for an AI assistant is the artefact that answers the question "what does this database look like right now". Whether the database is Postgres, MySQL, SQL Server or Redshift, the shape of the problem is the same: the catalog carries table names, column names, types, keys, and enough relationships to let the assistant write a query that resolves. It lives somewhere between the database and the assistant, has to stay in sync with a database that changes underneath it, and is almost always built the same weekend the team decides they want an AI assistant reading their data. It runs fine for the first three tables. The problems start around the fourth week, and none of them look like the same problem twice. The distinction worth naming early is between the connection layer (how the assistant reaches the database) and the knowledge layer (what the assistant knows about the database's shape). The connection layer receives most of the attention, because credentials, network isolation and query cost are visible failure modes and easy to argue about. The knowledge layer is where most of the actual quality of the assistant lives, and it decays quietly. The AI database context page covers why this second layer matters at all when the first one exists. Why not just point the assistant at the database Connecting the AI directly to production is the shortest path and the one most teams reject after five minutes of thinking about it. The assistant would get read access on tables it should not see, its queries can be arbitrarily expensive, its credentials would live somewhere they should not, and the audit trail becomes hard to reason about. What most teams end up building is a layer in between: a representation of the database that the assistant can read cheaply and safely without ever touching production. That layer is what this article is about. It is not the connection. It is the catalog. The five recipes teams build Ask fifteen senior developers how to build

The PostgreSQL manual is unusually direct about this: When an application receives this error message, it should abort the current transaction and retry the whole transaction from the beginning. "The whole transaction" is doing a lot of work in that sentence, and it is the part that gets dropped. TypeORM issue #9806 — "Auto Retry options on error in transactions (e.g. Deadlock)" — has been open since February 2023. Thirty 👍, six comments, no implementation. Meanwhile typeorm-transactional , at 188,000 downloads a week, ships @Transactional() with isolation levels and seven propagation modes and no retry at all. So the ecosystem's actual answer to "how do I use SERIALIZABLE in Node" is: don't. Use READ COMMITTED , don't think about write skew, and hope. I spent a while building the thing that issue asks for. The short version of what I found: the feature as literally requested cannot be built correctly , and the reason is more interesting than the feature. The implementation everyone reaches for first Wrap the query. It's the obvious move — the error came from a query, so retry the query: async function withRetry < T > ( fn : () => Promise < T > , attempts = 3 ): Promise < T > { for ( let i = 1 ; ; i ++ ) { try { return await fn (); } catch ( e ) { if ( i >= attempts || ! isSerializationFailure ( e )) throw e ; await sleep ( 50 * i ); } } } await dataSource . transaction ( ' SERIALIZABLE ' , async ( em ) => { const from = await em . findOneOrFail ( Account , { where : { id : fromId } }); const to = await em . findOneOrFail ( Account , { where : { id : toId } }); await withRetry (() => em . decrement ( Account , { id : fromId }, ' balance ' , amt )); // ← here await withRetry (() => em . increment ( Account , { id : toId }, ' balance ' , amt )); // ← and here }); This does nothing. Worse than nothing — it turns one clear error into a confusing one. When PostgreSQL raises 40001 , it does not fail that statement . It aborts the entire transaction . The connection is now

This article was written by Lin Borland Aggregation pipelines are one of the most powerful tools in MongoDB. They let you filter, reshape, compute, and group documents in a single query. In practice, the aggregation framework feels almost like a language of its own. With its combination of stages, expressions, and operators, you can describe everything from straightforward filtering to sophisticated transformation logic. This expressive power is what makes aggregation pipelines so useful, and is also why they have a learning curve associated with them. If you’ve worked with MongoDB in Go, you may know that the existing syntax for writing pipelines in Go can be cumbersome to work with. This is especially true when a pipeline includes several stages, repeated computed logic, or deeply nested expressions. In these cases, both readability and writability may begin to suffer. There’s a need for a more Go-native way to build aggregation pipelines. This is why we’re introducing a new approach: an experimental aggregation builder in Go. In this article, we’ll compare the traditional and new approaches, then go through an example. The traditional BSON-based approach Today, if you want to build an aggregation pipeline with the Go driver, you typically do it with bson.D, bson.A, and mongo.Pipeline. While this approach is flexible, it can be hard to spot small mistakes. Let’s use a simple example from the sample_mflix.movies collection. Suppose we want to find movies released after the year 2000. Here’s a pipeline that demonstrates how easy it can be to get the shape wrong: mongo . Pipeline { bson . D {{ Key : "$match" , Value : bson . E { Key : "$gte" , Value : bson . E { Key : "$year" , Value : 2000 }}}}} At a glance, the mistake might not be obvious. The document is valid BSON, but the pipeline uses “bson.E” instead of “bson.D” for some values, resulting in a pipeline that returns zero results. If we try to fix the nesting, we can still end up with a pipeline that is structu

Data has become one of the most valuable assets for modern enterprises, powering everything from business intelligence dashboards to machine learning models and generative AI applications. However, the biggest challenge organizations face today is not collecting data — it is trusting it. Enterprise data often contains duplicate records, missing values, inconsistent schemas, outdated information, and inaccurate entries that silently reduce the quality of analytics and AI predictions. These hidden data quality issues can lead to poor business decisions, increased operational costs, compliance risks, and unreliable AI outcomes. While most organizations implement basic validation rules, traditional data quality frameworks are largely rule-based, difficult to maintain, and unable to detect complex anomalies that continuously evolve across modern cloud data platforms. This article introduces the Data Trust Score Engine, an AI-powered cloud-native solution designed to automatically measure and improve enterprise data reliability. Instead of relying solely on manual validation or predefined rules, the platform combines metadata intelligence, large-scale analytics, and machine learning to calculate a dynamic Trust Score (0–100) for every dataset. The score is generated by evaluating multiple quality dimensions, including data completeness, consistency, uniqueness, freshness, schema compliance, null-value distribution, statistical anomalies, and AI-detected outliers. As a result, organizations can quickly identify fake, duplicate, corrupted, or low-quality datasets before they impact reporting, business intelligence, or downstream AI models. Learn about Medium’s values The solution is built entirely on Google Cloud Platform (GCP) using BigQuery as the scalable analytical data warehouse, Data Catalog for centralized metadata management and governance, and Vertex AI for intelligent anomaly detection and predictive quality analysis. BigQuery processes billions of records efficie

AI coding agent Cursor has launched Origin, a git based code hosting platform embedded inside its AI-powered editor, positioning it as an alternative to GitHub for teams that already work in Cursor. Origin is rolling out in early beta on Pro, Teams and Enterprise plans, and lives inside a new Codebase tab within the Cursor application. By Matt Saunders

DuckDB Labs has previewed DuckDB v2.0, codenamed "Cyanoptera." This release includes over 10000 commits and introduces a client/server mode, enabling network connections. Improvements also encompass extension portability, advanced data types, and a new parser. Performance enhancements include asynchronous I/O and storage optimisations. General availability is expected in fall 2026. By Olimpiu Pop

There’s no clearer sign of the data center boom than rampant gas projects that have been proposed or that are already under construction.

Classic Machine Learning Through the Eyes of an SRE — Part 8 The most dangerous output in my whole Week-1 study set wasn't a bad prediction. It was a beautiful tree. Hierarchical clustering produces a dendrogram, that elegant diagram where every account, ticket, or incident nests inside ever-larger families. It looks like discovered truth. Stakeholders lean in. Someone screenshots it for the QBR deck. Nothing else in the set looks as convincing while being as capable of being completely wrong. A bad K-Means gives you blobs that feel arbitrary, and people push back. A dendrogram built with the wrong linkage on flat data still looks like a family tree of your business. Nobody pushes back on a tree. The bet and the build Hierarchical clustering completes the answer-finding taxonomy I've been using through this series. That's my own shorthand, not standard terminology: K-Means SEARCHES, DBSCAN DEFINES, PCA SOLVES, and hierarchical clustering BUILDS. Start with every point as its own cluster. Repeatedly merge the closest two clusters. Never undo. Greedy and irreversible, a little like growing a decision tree. Same skeleton, different family. There is also a top-down version, called divisive clustering, which starts with everything together and splits it. In practice, when people say hierarchical clustering, they're usually talking about the bottom-up, agglomerative version. Two things were genuinely new to me. You choose the cut after seeing the structure. Fitting doesn't require you to decide K upfront. The dendrogram gives you the hierarchy, and you choose where to cut it to get the number of clusters you want. That makes the output unusually flexible. For a delivery organization it also feels natural, because account family → sub-segment → individual account is already how a lot of governance gets organized. Linkage is a selectable worldview. "Closest clusters" needs a definition, and every definition makes a different assumption. Ward pushes toward compact, variance-

Intro While building a simple A‑share market monitor for my quant lab work, I initially only cared about extracting obvious metrics: last price, total trading volume, and so on. My naive assumption was that pulling raw JSON from an A‑share real‑time market API and rendering it would finish the job. Once I started running short‑term trading simulation workflows, I realized most actionable insight lives inside structured order‑book data. Level‑2 data is far more than a basic price snapshot. It carries granular bid‑ask tiers plus real‑time order change events. Bad parsing logic will desync your local order book from the real exchange state and mislead your trading simulation decisions. Pain points: Regular market data vs Level‑2 data Standard market APIs return lightweight records built for simple UI display. You mostly get last traded price, total volume, and price change. Level‑2 is designed to reconstruct the full order book. It exposes five‑tier bid/ask prices & volumes, trade direction flags, and order‑update events. You can clearly observe shifts between buying pressure and selling pressure. One common gotcha: A‑share real‑time market APIs don’t follow uniform field naming. Some wrap order tiers inside arrays, others split bids and asks into separate top‑level fields. Without standardized parsing logic, order‑book ratio calculations and strength comparisons will produce wrong results. A typical five‑tier order‑book object includes ticker symbol, bid array, ask array, and timestamp. In my workflow I keep bid‑side and ask‑side processing separate: Bid side : extract best‑bid price and volume, aggregate total buy‑side depth Ask side : extract best‑ask price and volume, assess selling pressure Keeping them isolated makes multi‑side calculations cleaner and speeds up debugging. Efficiency note: Don’t compute directly on raw API payloads I never feed unprocessed Level‑2 raw responses straight into indicator calculations. A normalization step is mandatory. Raw unnormali

Originally published at ai.bedvibe.studio . There is an assumption almost every practitioner carries without examining it: if your dataset has bad samples in it, the loss will tell you. Corrupted rows spike. Broken files stick out. Sort by per-sample loss, look at the top of the list, and there is your garbage. I believed it too. Two separate failures in my own work say it is wrong, and they fail in the same direction — quietly. The reproducible one: a dataset that cannot be learned While validating trainproof I ran a controlled fault-injection study: one base setup, a Qwen2.5-3B QLoRA, run six ways, three seeds each, eighteen runs total. Every log ships in the repo so the verdicts can be checked rather than believed. One configuration shuffled the dataset's labels into pure noise. The labels no longer corresponded to the inputs at all. This is not a hard dataset or a noisy dataset. It is a dataset that cannot be learned , because there is no relationship left in it to learn. That run reduced its loss by 62%. On its own curve it was textbook-healthy — a clean downward slope, no spike, no plateau, nothing a human or a rule would flag. It was learning nothing useful. It was memorising the statistics of noise, which any sufficiently large network will happily do. From a single run's loss curve it is indistinguishable from a real one. That is where the assumption broke for me. Not "loss is a weak signal for this." Loss is not a signal for this at all, in isolation. The production one, and what I can and cannot prove about it The second failure came from real work rather than an experiment, and it is the one I think about more. Building a text-to-speech corpus of roughly 110,000 recordings, a small number of the files were pure loud white noise. Not corrupted in the file-format sense — they opened fine, played fine, had valid headers and valid duration. They simply contained no speech. Just noise, at volume. They did not surface as high-loss outliers. Being precise about

Solid-state transformers could also benefit EV charging and someday households.