5 states, 2 working filters: scraping US childcare license registries

Five states, one query language, and an "active licenses only" checkbox that only actually filters two of them. That's the trap in scraping US childcare-license open-data registries: Socrata SODA makes every state's API look identical, but "active" is defined — or not defined at all — differently in every dataset. Quick answer New York, Connecticut, Colorado, Delaware, and Texas all publish their childcare-facility registries through Socrata, and all five accept the same $where query syntax. But only NY and CT ship a server-side status filter this Actor can apply. Colorado and Delaware have no status column in the dataset at all — there's nothing to filter on. Texas does have a status column ( operation_status ), it's just not wired into the active-only filter, so toggling activeOnly doesn't touch Texas rows either way. Treating "active only" as a global switch that behaves the same everywhere will silently hand you closed and revoked facilities in three of the five states while you believe you filtered them out. STATE_CONFIGS : dict [ str , StateConfig ] = { " NY " : StateConfig (..., col_status = " facility_status " , active_where = " facility_status= ' Active '" ), " CT " : StateConfig (..., col_status = " status " , active_where = " status= ' ACTIVE '" ), " CO " : StateConfig (..., col_status = None ), # no status column to filter on " DE " : StateConfig (..., col_status = None ), # no status column to filter on " TX " : StateConfig (..., col_status = " operation_status " ), # status exists, filter isn't wired } Why does "active only" do nothing in three states? Because the filter is applied per-state, not globally, and only two states have both a status column and a configured $where fragment for it: async def _fetch_page (...): params = { " $limit " : str ( page_limit ), " $offset " : str ( offset ), " $order " : config . order_key } if active_only and config . active_where : params [ " $where " ] = config . active_where return await _get_with_retry ( session