标签：#opendata

We open-sourced 449 real equipment financing quotes so nobody has to trust our math Commercial equipment financing sites are almost always a black box: you land on a page, see a monthly payment, and have no way to check how that number was actually derived. The APR is picked out of thin air, the "starting at" price is aspirational, and the amortization math is never shown. We built Equipment Capital Index to do the opposite — every page shows the real per-machine price, the actual amortization schedule, and now we've published the whole underlying dataset so anyone can verify or build on it. What's actually in the dataset equipment-financing-rate-data is a CC BY 4.0 dataset of aggregate financing benchmarks computed from 449 individually priced, real machines — construction equipment, ag machinery, trucking fleet, power equipment, and material handling gear. No survey estimates, no fabricated averages. Current live snapshot: Category Machines tracked Avg APR Avg est. monthly payment Heavy Construction 222 8.25% $3,272 Agriculture 84 7.75% $4,411 Trucking Fleet 63 8.00% $2,501 Power Equipment 44 8.50% $957 Material Handling 36 8.50% $825 Site-wide average: 8.17% APR , $2,954/mo across all 449 machines. Why this exists A couple of principles drove the design: Every number traces back to a real machine. Each of the 449 rows has a sourced price (dealer listing, MSRP, or a documented class-typical estimate — and it's disclosed which one) and a real amortization calculation, not a rounded guess. The math is reproducible, not just displayed. The same aggregation logic that powers the /press page on the site also generates this dataset — one source of truth computed twice, so the numbers can't silently drift apart. It shouldn't require scraping a webpage. The data has three independent, permanent homes: A live JSON API: /api/rate-report.json ( OpenAPI spec ) A self-updating GitHub repo (regenerates from live data every 3 days via GitHub Actions) A permanent, versioned DOI o

Five states, one query language, and an "active licenses only" checkbox that only actually filters two of them. That's the trap in scraping US childcare-license open-data registries: Socrata SODA makes every state's API look identical, but "active" is defined — or not defined at all — differently in every dataset. Quick answer New York, Connecticut, Colorado, Delaware, and Texas all publish their childcare-facility registries through Socrata, and all five accept the same $where query syntax. But only NY and CT ship a server-side status filter this Actor can apply. Colorado and Delaware have no status column in the dataset at all — there's nothing to filter on. Texas does have a status column ( operation_status ), it's just not wired into the active-only filter, so toggling activeOnly doesn't touch Texas rows either way. Treating "active only" as a global switch that behaves the same everywhere will silently hand you closed and revoked facilities in three of the five states while you believe you filtered them out. STATE_CONFIGS : dict [ str , StateConfig ] = { " NY " : StateConfig (..., col_status = " facility_status " , active_where = " facility_status= ' Active '" ), " CT " : StateConfig (..., col_status = " status " , active_where = " status= ' ACTIVE '" ), " CO " : StateConfig (..., col_status = None ), # no status column to filter on " DE " : StateConfig (..., col_status = None ), # no status column to filter on " TX " : StateConfig (..., col_status = " operation_status " ), # status exists, filter isn't wired } Why does "active only" do nothing in three states? Because the filter is applied per-state, not globally, and only two states have both a status column and a configured $where fragment for it: async def _fetch_page (...): params = { " $limit " : str ( page_limit ), " $offset " : str ( offset ), " $order " : config . order_key } if active_only and config . active_where : params [ " $where " ] = config . active_where return await _get_with_retry ( session

"Just query the Socrata API" is true and also useless advice. Socrata SODA is a real open standard — New York, Connecticut, Colorado, and Texas all expose their professional-license registries through the same $limit / $offset / $where query language. The standard ends there. What each state puts inside that standard is four unrelated data models wearing the same protocol. Quick answer Every state's cosmetology/barber/salon registry is one giant multi-profession table with its own column names, its own beauty-credential filter, and its own idea of what "active" means — and one state (Texas) doesn't expose a status column at all, so an activeOnly toggle is a silent no-op there. A generic Socrata client that assumes one schema will either miss most of the data or crash on the first state whose columns don't match. The fix is a per-state config object that maps each state's real column names to one canonical output row, with the active-license filter applied only where the underlying data supports it. @dataclass ( frozen = True ) class StateConfig : state : str endpoint : str order_key : str col_business_name : str | None col_licensee_name : str | None col_status : str | None base_where : str | None = None active_where : str | None = None Why does the same query return different professions per state? Cosmetology licenses don't get their own dataset — they're rows buried inside each state's entire professional-licensing table, next to electricians, dentists, and notaries. Filtering has to happen server-side, in SoQL, before pagination even starts, or you're downloading (and paying to store) irrelevant rows. Texas needs a starts_with() match across three license-type prefixes plus an Establishment wildcard; Connecticut needs an exact in() list of six credential names; Colorado needs a four-code in() list: TX_BEAUTY_WHERE = ( " starts_with(license_type, ' Cosmetology ' ) " " OR starts_with(license_type, ' Class A Barber ' ) " " OR starts_with(license_type, ' Barber ' ) "

The openFDA enforcement API is free, keyless, and well documented on the surface. It is also full of failure modes that return HTTP 200 with quietly wrong data. Every number and error string below was measured against the live API on 2026-07-20; anything I could not reproduce has been cut. Pick the right endpoint first There are four recall-shaped endpoints and they are not interchangeable. Choosing wrong gives you a different universe of records with no warning. Endpoint Records What it is drug/enforcement.json 17,793 Recall Enterprise System (RES) drug recalls device/enforcement.json 39,519 RES device recalls food/enforcement.json 29,224 RES food recalls device/recall.json 58,756 CDRH device recall database, a different schema entirely The three enforcement endpoints share their schema. device/recall.json does not: its fields include cfres_id , product_res_number , k_numbers , root_cause_description , event_date_posted , event_date_terminated and recall_status , and it has no classification field at all ( count=classification.exact returns HTTP 404 "Nothing to count" ). If you are filtering for Class I, you want an enforcement endpoint. The two families also refresh on different clocks. On 2026-07-20 the three enforcement endpoints reported meta.last_updated of 2026-07-08, while device/recall.json reported 2026-07-17. The OR bug that silently returns the wrong answer This is the single most expensive trap, and it is undocumented. An unparenthesized OR discards every clause except the last one. All figures below are from food/enforcement.json . search=classification:"Class I" OR state:"CA" returns 4,003 - exactly the count for state:"CA" alone. Reverse the operands and you get 12,809 - exactly classification:"Class I" alone. Wrap it: search=(classification:"Class+I"+OR+state:"CA") returns 14,822 in both orders. That is the real union (12,809 + 4,003 - 1,990 overlap, and the AND of the two clauses does return 1,990). No error is raised in any case. The nastier varia

On July 8, 2026 I looked up a barcode that does not exist. Eight zeros. I sent them to Open Food Facts, the largest open nutrition database on the web, and it answered HTTP 200. Green light. Then I read the body: "status":0 , "status_verbose":"no code or invalid code" . A success code wrapped around a total miss. Ten seconds of trusting the status line and I would have written that empty result into a calorie tracker as if it were food. That is the whole post. The list of APIs is the easy part. The hard part is that a keyless food API hands you a clean 200 and a wrong answer, and it does it a slightly different way on almost every endpoint. A free food API here means a public nutrition, ingredient, or recipe endpoint that returns JSON with no API key, no signup, and no card. Not a CSV dump, not a partner form, not a portal from 2012. A real REST call you can paste into a terminal right now. I found eight that clear that bar, plus three worth knowing that quietly lean on a shared key. I re-verified every one with a live curl on July 8, 2026 (real HTTP code, real body, trimmed but never paraphrased). If you build calorie trackers, meal planners, grocery tools, or an AI agent that answers "how much sugar is in this," these are the lookups you reach for. Every one of them can lie to you with a 200. Here is the uncomfortable finding before the list. Keyless nutrition data in 2026 is mostly one project. Open Food Facts and its sibling databases (Pet Food, Products, Beauty, Prices) are six of the eight entries below: five distinct databases on one shared engine, with Open Food Facts itself showing up twice because it fails two different ways. Only two entries, Fruityvice and Wger, are independent, and Wger re-imports its data from Open Food Facts anyway. That concentration is not a weakness of the roundup. It is the point. Because it is one engine, the data-quality traps below are systemic, not one-offs. Learn them once and they repeat across the whole family. Let me be st

Here is a question I could not answer from the headlines: which European countries are actually losing people the fastest, in absolute terms or per capita? Those are two different questions, and they give two different answers. So I pulled the open data and ran the numbers. The headline figure Across the 19 European countries in the 2024 dataset, 17 recorded a net loss of native-born residents . Only two were net positive. So the "brain drain" story is not a handful of outliers, it is the default state of the continent. But the interesting part is who tops the ranking, because it depends entirely on how you measure. Load the data yourself The dataset is public on GitHub (CC BY 4.0). Every number below is reproducible with a few lines of pandas. No download, no API key, it reads the raw CSV straight from the repo: import pandas as pd url = ( " https://raw.githubusercontent.com/DatapulseResearch/ " " brain-drain-eu/main/data/net_migration_native_born_2024.csv " ) df = pd . read_csv ( url ) print ( df . shape ) # (19, 3) print ( df . columns . tolist ()) # ['country', 'net_migration', 'per_1000_residents'] # How many countries lost native-born residents? losers = ( df [ " net_migration " ] < 0 ). sum () print ( f " { losers } of { len ( df ) } countries had a net loss " ) # 17 of 19 net_migration is the raw count for 2024 (negative means a net loss of native-born residents). per_1000_residents is the same flow normalized by population size. The absolute ranking: Germany runs away with it Sort by the raw count and one country dominates: worst_absolute = df . sort_values ( " net_migration " ). head ( 5 ) print ( worst_absolute [[ " country " , " net_migration " ]]) country net _ migration 0 Germany - 91067 ... Germany loses -91,067 native-born residents, far more than anyone else in absolute terms. If you stop reading here, the story writes itself: "Germany, Europe's biggest brain drain." Plenty of coverage did exactly that. The counterintuitive finding: the ranking inve

Every Indian data scientist hits the same wall. You need district-level population data. You go to censusindia.gov.in. You find hundreds of inconsistent Excel files with merged headers, footnote rows, and zero documentation. You spend a full day just loading the data before doing any actual analysis. I fixed that. Once. For everyone. What I built indiaset/census-2011 India's Census 2011 district data, clean, typed, and ready for pandas. 640 districts · 29 columns · 0 missing values Validated against official India total · LGD codes attached Load it in 4 lines from huggingface_hub import hf_hub_download import pandas as pd path = hf_hub_download ( repo_id = " indiaset/census-2011 " , filename = " census_2011_districts_final.parquet " , repo_type = " dataset " ) df = pd . read_parquet ( path ) print ( df . shape ) # (640, 29) What's in it Column Description state_code Census 2011 state code state_name Official state/UT name district_code Census 2011 district code district_name District name as per Census lgd_code LGD permanent district code district_name_lgd District name as per LGD pop_total Total population pop_male Male population pop_female Female population pop_under6_total Children under 6 years pop_sc Scheduled Caste population pop_st Scheduled Tribe population literate_total Literate persons literate_male Literate males literate_female Literate females illiterate_total Illiterate persons workers_total Total workers workers_male Male workers workers_female Female workers non_workers_total Non workers literacy_rate Literate / Total × 100 sex_ratio Females per 1000 males workforce_participation Workers / Total × 100 The validation The most important test - do all 640 district populations sum to India's official total? print ( df [ ' pop_total ' ]. sum ()) # 1210854977 ✅ — exact match, zero discrepancy What the data actually shows Most literate district → Pathanamthitta, Kerala : 88.74% Least literate district → Alirajpur, Madhya Pradesh : 28.77% Literacy gap acro