今日已更新 344 条资讯 | 累计 37249 条内容
关于我们

The head of your CSV is lying: how 9,291 invoice numbers almost vanished

Ahmed Abdeltawab 2026年08月15日 14:19 3 次阅读 来源:Dev.to

Real transaction data is never clean — and the worst part is that it looks clean. This is a short story from a real dataset (UCI Online Retail: 541,909 e-commerce transactions) about the quietest way to destroy data: silent type coercion. All numbers below come verbatim from an executed notebook. The head looks perfect Peek at the first rows of the file and InvoiceNo parses as clean integers — 100% parse rate, full confidence. Any type-inference step, mine included, would call it int64 and move on. Measure the whole file instead of the head, and the number drops to ~98%. The other 2%: invoice numbers starting with "C" — which in this dataset marks a cancellation . Coerce the column to numeric and every one of them becomes NaN : Invoice numbers destroyed by numeric coercion: 9,291 DextraLoaderWarning: load: ambiguous decision(s): column 'InvoiceNo': ambiguous - float64 at parse_rate=0.98 An entire class of business events — silently gone. No exception, no crash. That's what makes coercion the quietest bug in data work: the pipeline succeeds . Why those 9,291 rows matter They are not noise. They are the returns side of the business : cancelled orders worth 8.4% of everything sold. Lose them and every revenue number downstream is quietly wrong. One example of what they catch: the dataset's apparent #1 bestseller, "PAPER CRAFT, LITTLE BIRDIE" (168,470 GBP), is a phantom — a single 80,995-unit order entered at 09:15 and fully cancelled at 09:27 the same morning. Only the preserved cancellation rows expose it. The genuine bestseller is a cake stand. The fix: identifiers are labels, not quantities No library can know that "InvoiceNo" is an ID — that's domain knowledge. What a tool can do is disclose its guess and hand you a replayable plan you can correct: naive , plan = dx . load ( CSV_PATH , return_params = True ) # warns: ambiguous at 0.98 plan [ " columns " ][ " InvoiceNo " ][ " dtype " ] = " object " # invoices are labels plan [ " columns " ][ " StockCode " ][ " dtype

本文内容来源于互联网,版权归原作者所有
查看原文