标签：#polars

Polars hits the sweet spot between Pandas’ ease and Spark’s scale. If you’ve ever waited on a groupby or cursed a memory error, this cheat sheet is for you. I’ve pulled the patterns that save time in real pipelines, not just toy examples. Bookmark this before your next ETL run. Setup and Basics First, get Polars and a dataset. The lazy API is the default now, so you’ll rarely need to call .lazy() explicitly. Start with a CSV or Parquet file, or create a DataFrame from scratch. pip install polars pyarrow import polars as pl df = pl.read_csv('data.csv') # or pl.read_parquet() df = pl.DataFrame({'a': [1, 2], 'b': ['x', 'y']}) Selecting and Filtering Polars uses expressions, not strings. This feels odd at first but pays off when you chain operations. The syntax is consistent: every column is an expression you can transform, filter, or aggregate. df.select(['a', 'b']) # columns by name df.select(pl.col('a').alias('renamed')) df.filter(pl.col('a') > 10) df.filter(pl.col('b').is_in(['x', 'z'])) df.filter(pl.col('a').is_null()) Transforming Data Polars expressions are composable. You can nest them, reuse them, and even store them in variables. This is where the library shines over Pandas. df.with_columns(pl.col('a').cast(pl.Float64)) df.with_columns(pl.col('a').fill_null(0)) df.with_columns((pl.col('a') * 2).alias('a_doubled')) df.with_columns(pl.col('b').str.to_uppercase()) df.with_columns(pl.col('a').is_between(10, 20)) Grouping and Aggregations Groupbys in Polars are lazy by default. This means you can stack multiple aggregations without materializing intermediate results. The syntax is clean, but watch out for the order of operations. df.group_by('b').agg(pl.col('a').sum()) df.group_by('b').agg([pl.col('a').mean(), pl.col('a').max()]) df.group_by('b').agg(pl.col('a').quantile(0.9)) df.group_by_dynamic('timestamp', every='1d').agg(pl.col('a').sum()) Joins and Concatenation Joins in Polars are explicit. You’ll specify the join type and the columns to join on. Concatenatio

You need to hand a dataset of Chilean RUTs to an outside analytics team. They will join it against other tables by identifier, run the cohort analysis, and hand back a model. They do not need to know, and should never learn, who any of these people are. Asterisk the RUT column and the join dies on contact: **********-K matches every other asterisked RUT in the file. Not almost every one. Every one. You need the same input to reappear as the same output, shaped like a real, check-digit-valid identifier the rest of your schema still recognizes, and eight weeks later, when a fraud investigator needs the original RUT back for one row, you need to be able to give it to them. Irreversible masking cannot do any of this. Hashing gets you consistency but not the format, and never the value back. What you need is format-preserving encryption: run a digit string through a cipher and get out another digit string, same length, same shape, that decrypts to the original under the key you hold. Nothing else. What FPE actually does MaskOps exposes this as mask_pii_fpe . It masks digit-based PII, cards, phones, RUT, CPF, Argentine DNI, in place, and gives back something the same length and shape: import maskops import secrets key = secrets . token_bytes ( 32 ) # AES-256, client holds this tweak = secrets . token_bytes ( 7 ) # per-column/per-dataset context df . with_columns ( maskops . mask_pii_fpe ( " rut_column " , key , tweak )) 76.354.771-K becomes some other RUT-shaped, check-digit-valid string of the same length, under this key and tweak. Run it back through with the same key and tweak and it decrypts. Non-digit PII, IBAN, VAT, email, IP, EU national IDs, gets none of this. It always asterisks. There is no clean digit domain to encrypt into, so MaskOps does not pretend there is. The key never touches MaskOps' output. The client generates it, holds it, and passes it in at call time, and because MaskOps makes no network call and keeps no storage layer, there is nowhere for that k

In the global streaming economy, Spotify, Apple Music, and other DSPs process billions of plays daily. Behind this massive transaction layer lies a fragmented, dual-copyright structure: The Recording Copyright (Master Right): Identifies the audio file, registered using the ISRC (International Standard Recording Code). The Composition Copyright (Publishing Right): Identifies the melody, lyrics, and arrangement, registered using the ISWC (International Standard Musical Work Code). Because these registries are managed by separate global entities (IFPI for ISRCs and CISAC for ISWCs), there is no central mapping registry between them. This gap causes millions of dollars in mechanical royalties to sit unclaimed in collective management organization (CMO) "Black Boxes" before being liquidated to major publishers. In this article, we'll design and implement a high-performance Semantic Entity Resolution Protocol (SERP) to bridge this metadata gap programmatically. The SERP Resolution Pipeline Reconciling these records requires a multi-layered classification pipeline. Since manual matching is logistically impossible, we implement a three-tiered algorithmic approach: ┌────────────────────────┐ │ Raw Recording & Work │ │ Data Ingestion │ └───────────┬────────────┘ │ ▼ ┌────────────────────────┐ │ 1. Normalized Title │ ──[Similarity < 0.85]──> [Unmatched Queue] │ Distance Filter │ └───────────┬────────────┘ │ [Similarity >= 0.85] ▼ ┌────────────────────────┐ │ 2. Creator Overlap │ ──[No Overlap]──────────> [Unmatched Queue] │ Intersection Matrix │ └───────────┬────────────┘ │ [Intersection >= 1] ▼ ┌────────────────────────┐ │ 3. Duration Tolerance │ ──[Delta > 4s]──────────> [Manual Verification] │ Guard Check │ └───────────┬────────────┘ │ [Delta <= 4s] ▼ ┌────────────────────────┐ │ Verified Link & │ │ CMO Dispute Ready │ └────────────────────────┘ Step 1: Normalization & String Similarity Filter Title comparisons often fail due to punctuation mismatches, subtitle variations,