标签：#pandas

By Michael Nocito , data analyst · Published August 8, 2026 By the end of this page you can load a CSV into pandas, find out in twenty seconds what type every column became, stop the identifier columns losing their leading zeros, get dates read the way they were written, and turn a money column that arrived as text into numbers. It is about twenty-five minutes, and every output below was produced by running the code. Here is what to do today, the moment after you first load a file. Run df.dtypes . Not df.head() , which shows you what the values look like, but dtypes , which shows you what they are. A column of identifiers that says int64 has already lost its leading zeros, and a money column that says object or str is text that will refuse to add up. The short version: read_csv reads characters and guesses a type per column. The guess is usually right, it is silent when it is wrong, and four arguments replace guessing with instruction. The same characters becoming two different values is the idea, so it gets the picture. The original carries a diagram here. In words: On the left, a strip of five small square boxes holds one character each, reading zero, eight, zero, five, three, as the characters appear in the file. Two arrows branch out from that strip. The upper arrow leads to a strip of five boxes in which the first box is empty, crossed through and outlined in amber, while the remaining four hold eight, zero, five and three; the leading character has been discarded. The lower arrow leads to a strip of five boxes holding zero, eight, zero, five and three, identical to the original, outlined in blue. Both destinations came from the same source strip, and only one of them still contains everything the file did. Every output on this page is real. Run on pandas 3.0.2 against a small CSV built to contain the four problems every real export has: an identifier with leading zeros, ambiguous dates, a text marker for missing values, and money with a thousands separator. If

By Michael Nocito , data analyst · Published August 8, 2026 By the end of this page you can turn transactions into a monthly series, add period-on-period change and a cumulative total, get a share-of-total column, smooth a noisy line, and run all of it separately for every group. It is about twenty-five minutes, and every number below came out of running the code. Here is what to do today, on the series you already have. Count its rows against the number of periods in your date range. If your data covers January to May and the series has four rows, a period produced nothing, it never became a row, and every change figure after the gap is comparing the wrong pair. The short version: pct_change() divides each value by the one in the row above; cumsum() adds everything up to and including the current row. Both trust the rows you gave them to be the periods you meant. What happens when the previous period is zero is the idea, so it gets the picture. The original carries a diagram here. In words: Three bar positions stand on a baseline, labelled Mar, Apr and May. The March position holds a tall bar and the May position holds a slightly shorter tall bar. The April position holds no bar at all; there is only a short flat mark sitting on the baseline where a bar would start, drawn in amber to show a value of zero. An arc runs from the top of the March bar down to the April mark, and the figure minus one hundred percent is printed on it, which is a perfectly ordinary answer. A second arc runs from the April mark up to the top of the May bar, and the symbol printed on that one is not a percentage at all but the sideways figure eight that means infinity. The picture shows that a fall to nothing has an answer and a rise from nothing does not. Every number on this page is real. The sixteen-row orders table used across this whole set of guides, run in pandas 3.0.2. It runs from 5 January to 25 May 2026 and contains no April orders at all, which is not staged for this page; it is

By Michael Nocito , data analyst · Published August 8, 2026 By the end of this page you can attach columns from one DataFrame to another on a shared key, choose the right how for the question, see at a glance which rows failed to match, and catch the failure that quietly inflates every total in the frame. It is about twenty-five minutes, and every output below was produced by running the code. Here is what to do today, on every merge you write. Print the row count immediately before and immediately after it. A left merge must not change the row count, and if it did, the right-hand table has the key more than once and your totals have just gone up. The short version: merge pairs rows from two frames wherever their keys match, and the number of rows that come out depends on how many times each key appears on each side. One key twice on the right is the idea, so it gets the picture. The original carries a diagram here. In words: On the left a single row is drawn as a wide box, holding the key Desk and the value 880. To its right stands a small lookup table with two rows, and both of those rows carry the same key, Desk. Two lines run from the single left-hand row, one to each of the two matching lookup rows, so the one row is paired twice. On the far right the result is drawn as two separate output rows, and both of them contain Desk and 880; the value 880 is ringed in amber in each of them to show that it is the same original figure appearing twice. One row went in and two came out, without anything being added to the left-hand table. Every output on this page is real. Sixteen orders totalling 9,890 and a three-row product table, the same tables used across this whole set of guides, merged in pandas 3.0.2 with the results copied back. If you know SQL joins , this is the same operation with different words, and the two failure modes are identical. 1. merge in one line Two frames, one shared column, one call. orders.merge(products, on="product", how="left") order_id prod

Real transaction data is never clean — and the worst part is that it looks clean. This is a short story from a real dataset (UCI Online Retail: 541,909 e-commerce transactions) about the quietest way to destroy data: silent type coercion. All numbers below come verbatim from an executed notebook. The head looks perfect Peek at the first rows of the file and InvoiceNo parses as clean integers — 100% parse rate, full confidence. Any type-inference step, mine included, would call it int64 and move on. Measure the whole file instead of the head, and the number drops to ~98%. The other 2%: invoice numbers starting with "C" — which in this dataset marks a cancellation . Coerce the column to numeric and every one of them becomes NaN : Invoice numbers destroyed by numeric coercion: 9,291 DextraLoaderWarning: load: ambiguous decision(s): column 'InvoiceNo': ambiguous - float64 at parse_rate=0.98 An entire class of business events — silently gone. No exception, no crash. That's what makes coercion the quietest bug in data work: the pipeline succeeds . Why those 9,291 rows matter They are not noise. They are the returns side of the business : cancelled orders worth 8.4% of everything sold. Lose them and every revenue number downstream is quietly wrong. One example of what they catch: the dataset's apparent #1 bestseller, "PAPER CRAFT, LITTLE BIRDIE" (168,470 GBP), is a phantom — a single 80,995-unit order entered at 09:15 and fully cancelled at 09:27 the same morning. Only the preserved cancellation rows expose it. The genuine bestseller is a cake stand. The fix: identifiers are labels, not quantities No library can know that "InvoiceNo" is an ID — that's domain knowledge. What a tool can do is disclose its guess and hand you a replayable plan you can correct: naive , plan = dx . load ( CSV_PATH , return_params = True ) # warns: ambiguous at 0.98 plan [ " columns " ][ " InvoiceNo " ][ " dtype " ] = " object " # invoices are labels plan [ " columns " ][ " StockCode " ][ " dtype

Pandas is an open-source library for data analysis and manipulation in Python. It provides fast, flexible and expressive data structures for working with relational and labelled data. Originally developed by Wes McKinney in 2008, it has become a foundational tool in modern data science and serves as a highly programmable analogue to spreadsheet software. Key characteristics NumPy foundation: Built on top of NumPy, it inherits highly optimised, array-based computational performance. Label-driven alignment: Data are automatically aligned according to explicit row and column labels, thereby improving the reliability of calculations involving partially mismatched datasets. Heterogeneous typing: Unlike strict numerical arrays, Pandas can accommodate mixed data types, including integers, strings, floats and booleans, within a single tabular structure. Missing-data resilience: It provides native support for detecting, representing and handling missing values, such as NaN. Core data structures Series: A one-dimensional labelled array capable of holding any data type. In practical terms, it resembles a single column in a spreadsheet. DataFrame: A two-dimensional tabular data structure with labelled rows and columns. It may be regarded as a collection of Series sharing a common index, analogous to a table in SQL or a worksheet in Excel. Core features and capabilities Robust input/output parsing: Pandas supports efficient reading and writing across multiple formats, including CSV, Excel, SQL databases, JSON and Parquet. Advanced data cleaning: Built-in methods enable users to identify, filter and remove duplicates, and to impute missing values. Flexible wrangling and reshaping: The library facilitates pivoting, melting, slicing and subsetting operations based on conditional logic. High-performance merging: Relational operations such as inner, outer, left and right joins, as well as concatenation, can be executed in concise code. Split-apply-combine (GroupBy): Data may be group

Introduction: The Question of Relevance Imagine you’re a college student, fresh off mastering pandas , and you’re eyeing the freelancing market for data cleaning and automation gigs. You’ve heard of openpyxl , but as you dig deeper, you hit a wall: every resource seems to peg it as a relic for handling 2010 Excel sheets . That’s it. No modern use cases, no integration with cutting-edge tools, just a dusty library stuck in the past. So, you pause. Is openpyxl still relevant in 2023, or is it a dead end for someone trying to build a competitive freelancing portfolio? This dilemma isn’t just about openpyxl—it’s about the mechanism of perception in tech. When a tool is associated with outdated formats, its capabilities are often misinterpreted or overlooked . Openpyxl’s documentation and community discourse rarely highlight its modern applications, leaving newcomers like you to assume it’s obsolete. But here’s the catch: openpyxl isn’t just a 2010 Excel handler. It’s a low-level Excel manipulator that, when paired with libraries like pandas and numpy, can handle complex tasks that these libraries alone can’t. The problem isn’t openpyxl’s functionality—it’s the information gap between its perceived and actual utility. The stakes are clear: if you dismiss openpyxl as outdated, you risk missing out on a tool that could complement your pandas and numpy skills , making your freelancing services more efficient and versatile. But if you invest time in it without understanding its modern applications, you might waste effort on a tool that doesn’t align with current demands. The question isn’t whether openpyxl is relevant—it’s whether you’re looking at it through the right lens. In this investigation, we’ll dissect openpyxl’s role in 2023 freelancing, addressing its perceived limitations and uncovering its hidden strengths. By the end, you’ll have a clear rule for deciding whether to include it in your toolkit: If your freelancing gigs involve Excel-specific tasks that pandas ca