标签：#review

[Paper Notes] JEPA: Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture 🔗 TL;DR: JEPA learns a a generalized semantic representation with less data pairs by predicting missing information in the embedding space , which helps it disregard unnecessary noisy from input(pixel)-level details and learns at a higher abstraction level with good semantic generalization. 1. Innovation & Significance The Bottleneck: Image-text data pair labels are hard to find Pixel level pre-training paired & data augmentation are strongly biased towards trained data distribution, hard to determine proper generalization and level of abstraction. JEA's (Joint Embedding Architecture) collapse probelm: encoder & decoder attempts to cheat by always landing on trivial constant when predicting itself (reconstruction) and gets away with an easy Error=0. The Solution: > Chain-of-thought ⭕ Mask pre-training to reduce data & generalize↓❌ Bad/lower semantic representation without semantic target, could be learning noisy local pixel correlation↓⭕ Learn at the embedding level to omit pixel input and generalize⭕ Adds context encoder & positional encoding to inject context and force model to pick up image inherent structure from reconstructing multiple masked patches with one target.↓❌ JEAs wants to cheat: if I always map all pixels to a constant for both the predictor and end target encoder then the reconstruction error is always collapsed to zero! Hehe~ ↓ ⭕ EMA (Exponential moving avg.): Update target encoder parameters from the EMA of context encoders. This 'delays' the target encoder to prevent collapsing (a trick from the BYOL paper[2020], proven essential to training JEAs with ViT). 2. Model & High-Level Intuitions 2.1 Model Architecture Input: randomly samples block masks from original image within certain aspect ratio changes, and apply mask for context image 2.1.2 Context Context Encoder: ViT encodes context image to embedding SxS_x S x ​ Mask Token : an [1,D] random

From the opening moments of bitknot, it's obvious that Feeble Little Horse has found an entirely new gear. Where on Girl with Fish the blown-out textures were more '90s indie rock and shoegaze, on their latest LP, there's a more modern edge to the distortion and the riffs cut cleaner. Similarly, where the digital glitchiness […]

The M-Track Duo HD Producer Pack removes the last remaining excuse for terrible audio and potentially terrible opinions.

Despite headwinds from the current administration, automakers continue to release well-equipped EVs with bigger battery packs and increasingly faster charging speeds. For those who want to travel further between plugging in, the future is still bright, just slightly tinted. But there haven't been many sedans starting around or below $50,000, as crossover SUVs have largely […]

As handheld consoles continue to grow and push the limits of what you can actually hold in your hands, the Arduboy FX-C comes in a refreshingly pocketable package. It manages to squeeze the best features of past models and some welcome upgrades into a handheld that’s still no larger or thicker than a few credit […]

I live in a part of Los Angeles where I feel safer bringing pepper spray on walks. The problem is, I don't always remember to bring it with me, and it's not legal to carry it everywhere I go. Pebblebee's $59.99 Halo Bluetooth tracker surprised me by being a suitable replacement because it doubles as […]

A roundup of recent reviews published by Engadget.

Slots & Daggers, a low-key, fantasy-themed slot machine roguelike, was one of my favorite games last year. That may sound like a complicated description, but the game mixes ideas from deckbuilding roguelikes with slot machines to create an engrossing loop, and there's steady meta-progression that helps you push further with just about every run. Perhaps […]

A new book looks into the long history of people who have opposed vaccines.

Keychron's K2 HE Concrete Edition sounds like a cute gimmick, but as I discovered, there's a really solid keyboard beyond the absurd choice of materials.

With inflation and gas prices rising, I’m trying to save money wherever I can. I tested Tello’s budget cell phone plan, and for me, it turns out prepaid can be just as good.

Made for those who hate having a TV in the home, Samsung’s Movingstyle monitor-on-wheels brings the entertainment when you need it and hides away in a closet when you don’t.

Of course the Hitman devs made a great James Bond game.

To get to $600, Apple and HP made different compromises. If you demand more power and performance, the HP Omnibook 3 delivers.

Audi has developed an entirely new electric torque-vectoring rear differential.

Wherever I go, whatever I do, people point at my finger and ask, "Is that an Oura Ring?" Lots of people find they like the design, and they tell me why they're thinking about switching to a smart ring from a smartwatch. But the people who scrunch up their noses? They usually say something along […]

Created from fabric scraps that would otherwise end up in a landfill, this thoughtfully constructed bag looks great and can carry everything you need for a weeklong trip.

With sharper resolution and lightning-fast performance, Canon’s latest full-frame mirrorless punches well above its class.

I’m a chronic overpacker, but this backpack is my perfect match, with its huge capacity, numerous pockets, and impressive durability.

Pretty little phones with pretty big price tags.