今日已更新 498 条资讯 | 累计 20470 条内容
关于我们

标签：#javascript

找到 553 篇相关文章

AI 资讯

How I scraped the CQC Care Register without hitting the API auth wall

The Care Quality Commission regulates 56,000+ healthcare and social care locations in England — care homes, GP surgeries, hospitals, dental practices, home care agencies. If you work in care sector tech, you've probably needed this data at some point. There's a CQC REST API, and I was planning to wrap it. Then I hit the auth wall. The API is now authenticated CQC migrated their API to api.service.cqc.org.uk and added bearer token authentication. You need to register at their developer portal, create an application, and include an Authorization: Bearer <token> header on every request. That's not a dealbreaker for enterprise use cases, but it creates friction for a data product — it means requiring users to register with CQC before they can run your actor. I checked the old API base ( api.cqc.org.uk/public/v1 ) as a fallback. HTTP 403. Fully blocked. The open-data file rescue CQC publishes a monthly open-data file called HSCA_Active_Locations.ods . It's a 23 MB OpenDocument Spreadsheet with every active regulated location in England — all 56,000 of them. Free, no auth, Open Government Licence. The URL is date-stamped and changes each month, but the transparency page always links to the current version. The approach: scrape the transparency page to find the current ODS URL, download the file, parse it, filter rows, push results. No API. No auth wall. The ODS parsing challenge ODS files ( .ods ) are ZIP archives containing XML. The standard tool for parsing them in Node.js is SheetJS ( xlsx package, v0.18.5 — the last Apache 2.0 release). The first surprise: the workbook has three sheets — README , HSCA_Active_Locations , and Dual_Registration_Locations . SheetJS defaults to the first sheet, which is the README with 34 rows. I added logic to find the sheet with the most rows. for ( const name of workbook . SheetNames ) { const probe = XLSX . utils . sheet_to_json ( sheet , { header : 1 }); if ( probe . length > bestCount ) { bestCount = probe . length ; bestSheet = shee

2026-05-29 原文 →
AI 资讯

I Just Wanted to Scrape One Page. Why Did I Write 50 Lines of Puppeteer?

Last Friday at 4:30 PM, my product manager walked over: "Hey, can you grab the titles from the Hacker News homepage and send me an Excel file?" I thought: That's it? Five minutes tops. Two hours later, I was still debugging CSS selectors. How Things Spiraled Out of Control Step 1: Initialize the Project mkdir hacker-news-scraper && cd hacker-news-scraper npm init -y npm install puppeteer Hit enter, waited three minutes. Puppeteer needs to download a full Chromium browser — over 200 MB. I stared at the progress bar and started questioning my life choices. Step 2: Write the Code "It's just a document.querySelectorAll , right?" That's what I thought. Then I opened my editor: const puppeteer = require ( ' puppeteer ' ); ( async () => { const browser = await puppeteer . launch ({ headless : true , args : [ ' --no-sandbox ' , ' --disable-setuid-sandbox ' ] }); const page = await browser . newPage (); try { await page . goto ( ' https://news.ycombinator.com ' , { waitUntil : ' networkidle2 ' , timeout : 30000 }); await page . waitForSelector ( ' .titleline > a ' , { timeout : 10000 }); const titles = await page . evaluate (() => { const items = document . querySelectorAll ( ' .titleline > a ' ); return Array . from ( items ). map ( el => ({ title : el . textContent , url : el . href })); }); console . log ( JSON . stringify ( titles , null , 2 )); } catch ( err ) { console . error ( ' Scraping failed: ' , err . message ); } finally { await browser . close (); } })(); I counted: 27 lines. And this is the minimal version — no User-Agent spoofing, no retry logic, no proxy support, no concurrency control. Add all of that and you're well past 50 lines. Step 3: Run It node index.js Error: Navigation timeout of 30000 ms exceeded . Switched to domcontentloaded , got past that. But then waitForSelector timed out — because .titleline was a relatively new class name. Hacker News had silently changed it from .storylink at some point, and nobody sent me the memo. Step 4: Debug Set head

2026-05-28 原文 →
AI 资讯

Building a Japanese-First Read-Later PWA: From Pocket Shutdown to Launch

When Mozilla shut down Pocket in July 2025, I lost my favorite tool. Worse, none of the English alternatives (Instapaper, Readwise, Matter, Raindrop) had Japanese UI, and their article extraction was mediocre on Japanese pages. So I built one. It's called Readbox — Japanese-first, English-too, read-later as a PWA. Here's what I learned shipping it. The stack Next.js 15 App Router + TypeScript strict (no any ) Supabase (Postgres + Auth + RLS) Stripe (JPY + USD prices, locale-routed) Tailwind CSS Service Worker for PWA install + offline read Three things that bit me 1. Article extraction on Vercel serverless First attempt: Mozilla Readability + jsdom. Doesn't bundle on Vercel because of ESM compatibility issues and the 50MB serverless function size limit. I tried 6 approaches — Webpack externals, dynamic imports, edge runtime — none worked cleanly. Ended up using Jina Reader , which returns clean Markdown/HTML from any URL. Trade-off: third-party dependency, rate limits at scale. But it works today, and it's free. 2. Storing article body on-device I didn't want to host millions of articles' worth of HTML on Supabase (cost + privacy). Solution: extracted HTML lives in the browser's IndexedDB only (via Dexie); only metadata (URL, title, tags, read status) syncs to the server. Trade-off: cross-device sync of body content doesn't work seamlessly. Acceptable for a "read it later" workflow where you usually read on the device you saved on. 3. i18n routing — the silent sitemap killer For Japanese + English from one codebase: app/[locale]/ segment with /en prefix for English (default Japanese has no prefix, to preserve old URLs). Middleware detects cookie / Accept-Language and redirects accordingly. The gotcha (cost me a launch-day hour): middleware matcher excludes _next , api , image extensions — but if you forget .xml/.txt/.webmanifest , sitemap.xml and robots.txt get rewritten to /ja/sitemap.xml (which doesn't exist as a route → 404). Fix: export const config = { matcher

2026-05-28 原文 →
AI 资讯

Building Metadata Capabilities in Apache SeaTunnel: A Committer’s Journey

Recently, Apache SeaTunnel welcomed several talented and highly motivated new Committers, and Wang Xuepeng is one of them. As a long-time contributor, Wang Xuepeng’s promotion to Committer was no coincidence. Over the years, he has quietly contributed a tremendous amount to the community, and everyone has witnessed his dedication. From first stepping into the open-source world to becoming a Committer of an Apache top-level project, he has accumulated plenty of stories and valuable insights along the way. What inspired his journey? What experiences and lessons does he want to share with the community? Let’s take a closer look at this exclusive interview with him! Personal Introduction Interview Transcript How long have you been involved in open source? What attracts you to open source? I started getting involved in open source in 2023. What attracts me most is the sense of achievement when the code I write can actually be used within the industry. When did you start contributing to SeaTunnel? What was the trigger? I joined WhaleOps in 2023, which was also when I first started engaging with open source. Now that you’ve been elected as a SeaTunnel Committer, could you summarize your contributions to the community, including both code and non-code contributions? Most of my major feature PRs have focused on building SeaTunnel’s metadata capabilities. When running SeaTunnel jobs and writing job configurations, users often need to manually enter datasource connection information. For file-based tasks, users also need to manually define field mappings. To address these issues, I designed an SPI interface called MetadataProvider . The interface mainly exposes two methods: Map<String, Object> datasourceMap(String connectorIdentifier, String metaDataDatasourceId); Optional<TableSchema> tableSchema(String metaDataTableId); Previously, some users in the community mentioned that datasource usernames and passwords were stored in Nacos with read-only access permissions. In scenario

2026-05-28 原文 →
← 上一页 下一页 →