Paper Reading Notes: [JEPA]
[Paper Notes] JEPA: Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture 🔗 TL;DR: JEPA learns a a generalized semantic representation with less data pairs by predicting missing information in the embedding space , which helps it disregard unnecessary noisy from input(pixel)-level details and learns at a higher abstraction level with good semantic generalization. 1. Innovation & Significance The Bottleneck: Image-text data pair labels are hard to find Pixel level pre-training paired & data augmentation are strongly biased towards trained data distribution, hard to determine proper generalization and level of abstraction. JEA's (Joint Embedding Architecture) collapse probelm: encoder & decoder attempts to cheat by always landing on trivial constant when predicting itself (reconstruction) and gets away with an easy Error=0. The Solution: > Chain-of-thought ⭕ Mask pre-training to reduce data & generalize↓❌ Bad/lower semantic representation without semantic target, could be learning noisy local pixel correlation↓⭕ Learn at the embedding level to omit pixel input and generalize⭕ Adds context encoder & positional encoding to inject context and force model to pick up image inherent structure from reconstructing multiple masked patches with one target.↓❌ JEAs wants to cheat: if I always map all pixels to a constant for both the predictor and end target encoder then the reconstruction error is always collapsed to zero! Hehe~ ↓ ⭕ EMA (Exponential moving avg.): Update target encoder parameters from the EMA of context encoders. This 'delays' the target encoder to prevent collapsing (a trick from the BYOL paper[2020], proven essential to training JEAs with ViT). 2. Model & High-Level Intuitions 2.1 Model Architecture Input: randomly samples block masks from original image within certain aspect ratio changes, and apply mask for context image 2.1.2 Context Context Encoder: ViT encodes context image to embedding SxS_x S x Mask Token : an [1,D] random