标签：#embeddings

Every system that does "semantic" anything — RAG pipelines, recommendation engines, image search, dedup — boils down to one operation: given this vector, find the closest ones out of millions. The vectors are embeddings, a few hundred to a couple thousand numbers each, and "closest" means closest in meaning. You'd assume the database either scans all of them (slow but correct) or uses some clever tree to jump straight to the answer. It does neither. Instead it deliberately settles for the approximately closest vectors — and that compromise is the entire reason vector search is fast enough to exist. Two algorithms do almost all the heavy lifting in practice, in pgvector, Qdrant, FAISS, and the rest: IVF and HNSW . Here's what they're actually doing under the hood, and how to choose between them. Why "exact" is off the table The natural objection is: why approximate? Just find the real nearest neighbor. In two or three dimensions you could — a k-d tree or similar structure prunes away big regions of space and finds the true closest point quickly. The trouble is that embeddings live in hundreds of dimensions, and high-dimensional space is deeply weird. It's called the curse of dimensionality . As dimensions grow, the distance to your nearest point and the distance to your farthest point drift toward being almost the same. Formally, the contrast (d_max − d_min) / d_min shrinks toward zero. When everything is roughly equidistant from everything else, a tree can't confidently say "skip this whole branch, it's too far" — the bounding regions all overlap, every branch looks plausible, and the search degrades into checking nearly everything. Exact indexes quietly collapse back into brute force. So we change the question. Instead of "prove you found the nearest," we ask "quickly find something very probably among the nearest." That's approximate nearest neighbor (ANN) search, and it swaps a guarantee for speed. The quality knob becomes recall : of the true top-k neighbors, wh

[Paper Notes] JEPA: Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture 🔗 TL;DR: JEPA learns a a generalized semantic representation with less data pairs by predicting missing information in the embedding space , which helps it disregard unnecessary noisy from input(pixel)-level details and learns at a higher abstraction level with good semantic generalization. 1. Innovation & Significance The Bottleneck: Image-text data pair labels are hard to find Pixel level pre-training paired & data augmentation are strongly biased towards trained data distribution, hard to determine proper generalization and level of abstraction. JEA's (Joint Embedding Architecture) collapse probelm: encoder & decoder attempts to cheat by always landing on trivial constant when predicting itself (reconstruction) and gets away with an easy Error=0. The Solution: > Chain-of-thought ⭕ Mask pre-training to reduce data & generalize↓❌ Bad/lower semantic representation without semantic target, could be learning noisy local pixel correlation↓⭕ Learn at the embedding level to omit pixel input and generalize⭕ Adds context encoder & positional encoding to inject context and force model to pick up image inherent structure from reconstructing multiple masked patches with one target.↓❌ JEAs wants to cheat: if I always map all pixels to a constant for both the predictor and end target encoder then the reconstruction error is always collapsed to zero! Hehe~ ↓ ⭕ EMA (Exponential moving avg.): Update target encoder parameters from the EMA of context encoders. This 'delays' the target encoder to prevent collapsing (a trick from the BYOL paper[2020], proven essential to training JEAs with ViT). 2. Model & High-Level Intuitions 2.1 Model Architecture Input: randomly samples block masks from original image within certain aspect ratio changes, and apply mask for context image 2.1.2 Context Context Encoder: ViT encodes context image to embedding SxS_x S x ​ Mask Token : an [1,D] random