Comparing prices across retailers is a unit-normalization problem, not a scraping problem

Disclosure: I'm the founder of Popgot , which I use as the example below. The problem and the approach apply regardless of what you build on. Every price comparison project I've seen starts the same way: scrape a bunch of retailers, store the prices, sort ascending. And then it produces garbage rankings, because price is not a comparable field. Here's the classic failure. Three listings for AA batteries: Listing Price Count Brand A $5.99 16 Brand B $6.99 20 Brand C $11.94 40 Sort by price and Brand A "wins" at $5.99. Sort by cost per battery and the order flips completely: Brand C is ~29.9c per cell, Brand A is ~37.4c. The cheapest listing is the worst deal on the page. Why this is hard The naive fix is "just divide price by quantity." The problem is that quantity almost never exists as a clean number. It's buried in the title, and the title is written by whoever uploaded the listing: AA Batteries 24 Pack AA Alkaline Batteries, 1.5 Volts, 24 Count 48-Pack (2 x 24) Double A So you end up writing a title parser. Then you discover the same product needs a different unit depending on the category: per fluid ounce for detergent, per serving for protein powder, per 100g for coffee, per sheet for paper towels. Then you discover that some categories need a spec filter before unit price is even meaningful. A fish oil at 20c per serving isn't cheaper than one at 34c per serving if the first one has half the EPA+DHA. You're comparing two different products. That last part is the piece people underestimate. Normalization is only valid within a set of products that actually satisfy the same requirement, which means something has to read the label, not just the title. What a normalized record looks like This is the problem I ended up building Popgot around, so rather than describe it abstractly, here's the shape of the data. The developer API returns listings with the unit math already done: GET /api/developer-api/products?query=aa+batteries&limit=10 { "products" : [ { "display_t