标签：#API

A security researcher found that adding a trailing slash to AWS HTTP API paths bypassed Lambda authorizer authentication entirely, enabling unauthenticated wire transfers at a fintech. The root cause is a path normalization mismatch between HTTP API's greedy route matching and its authorization layer. The same vulnerability class appeared in gRPC-Go via CVE-2026-33186. By Steef-Jan Wiggers

Integrations kill product velocity. Every SaaS team knows this. You ship a killer feature, customers love it, then they ask: "Can it sync with Salesforce? What about HubSpot? Zendesk?" Suddenly your roadmap is hostage to building connector after connector. Each one takes 2-3 weeks. Your engineers hate it. Your customers wait. Competitors who solve this faster win deals. The standard response has been iPaaS platforms. They help, but they don't fundamentally change the game. You still need engineers to map fields, handle edge cases, and maintain brittle connections. The real breakthrough isn't just automation , it's making integrations LLM-native from the ground up . What Actually Makes an Integration Layer "AI-Native"? Let's cut through the marketing speak. Every B2B tool now claims to be "AI-powered." Most just added a ChatGPT wrapper to their UI. Real AI-native architecture means three things: 1. LLM-Ready Connectivity via MCP Servers Model Context Protocol (MCP) is Anthropic's standard for connecting LLMs to external data sources. If your integration layer doesn't support MCP servers natively, your AI features will always be bolted on, not built in. MCP servers expose your SaaS data to language models in a structured way. Instead of engineers writing custom API wrappers for every LLM interaction, you get a standardized interface. Claude, GPT-4, and future models can query your integration layer directly. Example: A customer support tool with native MCP integration lets an AI agent pull ticket history from Zendesk, check Stripe subscription status, and update Salesforce records in one conversation flow. No custom code. No brittle middleware. 2. AI-Mapped Data Migration Data migration is where most SaaS deals die. Customer says "we'll switch from ServiceNow to your ITSM if you migrate our 50,000 tickets." Your team estimates 6 weeks. Deal stalls. Traditional migration means: Manual field mapping spreadsheets Custom scripts for data transformation Downtime windows Hi

PostgreSQL LISTEN/NOTIFY for Real-Time Multi-Tenant Events: Ditching Polling and WebSocket Complexity I've shipped real-time features in CitizenApp using three different approaches: naive polling (embarrassing), Redis pub/sub (overkill), and now PostgreSQL's native LISTEN/NOTIFY. The third option is what I should have started with. Most teams reach for Redis or RabbitMQ the moment they need real-time updates. It's the conventional wisdom. But here's the truth: if you're already running PostgreSQL, you have a battle-tested pub/sub system sitting right there. It handles multi-tenancy correctly, scales to thousands of concurrent connections, and eliminates an entire infrastructure dependency—which matters when you're deploying to Render or Vercel where every added service is friction. Why LISTEN/NOTIFY beats the alternatives Polling is dead. HTTP requests every 2-5 seconds for "new notifications"? That's technical debt masquerading as simplicity. It wastes bandwidth, kills your database with unnecessary queries, and users see stale data. Redis is powerful but expensive. Not just in dollars—in operational overhead. You need to manage connection pools, handle failover, monitor memory usage, and keep another service running in production. At CitizenApp's scale (thousands of concurrent tenants), we were paying $50/month for Redis on top of Render just to broadcast notifications that PostgreSQL could handle natively. WebSockets without a broker are a nightmare. If you're running multiple FastAPI workers (and you should be), a WebSocket connection to Worker A doesn't know about events published by Worker B. You need a message broker to fan-out events across processes. Unless you use PostgreSQL LISTEN/NOTIFY, which handles that automatically. PostgreSQL's pub/sub is: Transactional. Notifications only fire after a transaction commits. Tenant-aware. Use channel names like tenant_123_notifications and broadcast only to the right subscribers. Zero extra infrastructure. It's part

Previously, we learned how to send and receive data in Go. Now, we will combine those concepts and build a simple CRUD API. CRUD stands for: C reate R ead U pdate D elete These four operations form the foundation of most backend applications. In this tutorial, we will build a simple task API in Go using only the standar library. By the end, you will understand: how CRUD APIs work how to handle multiple HTTP methods how to store data in memory how to send and receive JSON data how backend APIs manage resources Prerequisites To follow along, you should have: Go installed basic familiarity with Go syntax understanding of the net/http package basic understanding of JSON handling You can confirm if Go is installed by running: go version Step 1 — Create the Project Create a new folder for the project: mkdir go-crud-api cd go-crud-api Now initialize a Go module: go mod init go-crud-api This creates a go.mod file for managing project dependencies. Step 2 — Create the Server File Create a file called main.go . Your project structure should now look like this: go-crud-api/ ├─ go.mod └─ main.go Step 3 — Write the CRUD API Open main.go and add the following code: package main import ( "encoding/json" "net/http" ) type Task struct { ID int `json:"id"` Title string `json:"title"` } var tasks [] Task func tasksHandler ( w http . ResponseWriter , r * http . Request ) { w . Header () . Set ( "Content-Type" , "application/json" ) switch r . Method { case http . MethodGet : json . NewEncoder ( w ) . Encode ( tasks ) case http . MethodPost : var task Task err := json . NewDecoder ( r . Body ) . Decode ( & task ) if err != nil { http . Error ( w , "Invalid JSON" , http . StatusBadRequest ) return } tasks = append ( tasks , task ) json . NewEncoder ( w ) . Encode ( task ) default : http . Error ( w , "Method not allowed" , http . StatusMethodNotAllowed ) } } func main () { http . HandleFunc ( "/tasks" , tasksHandler ) http . ListenAndServe ( ":8080" , nil ) } Now let's unpack what is hap

The row looked perfect. rating: 7 . Valid JSON, right type, no nulls, no missing keys. My schema check waved it through. The page had returned HTTP 200. The selectors hadn't moved. Everything green. A rating of 7 on a 5-star site is impossible. The model invented it, formatted it correctly, and handed it to me with total confidence. That's the failure I want to talk about. Not the scraper that breaks loudly. The one that hands you a clean-looking row that is quietly, plausibly false — and sails past every check you have, because your checks are all looking at the shape of the data, and the lie is in the value . TL;DR HTTP 200, intact selectors, and valid JSON tell you the form is fine. They say nothing about whether the value is true. When an LLM extracts from messy free-text, structured-output mode guarantees you get valid JSON. It does not guarantee the content is real. The model fills uncertain fields rather than leaving them empty — because the schema demands a complete row. A ~60-line value-level sanity gate (ranges, dates, cross-field, reference, language) catches the obvious lies before they hit your database. Real code and real output below. The honest catch: this gate catches rule violations , not plausible lies inside the allowed range . A rating: 4 where the truth is 2 slides right through. I'll be specific about where the gate stops. Two different ways a scraper lies to you I wrote about source drift last week — the case where the page changes underneath you and a 30-line schema check catches the structure shifting. That's an input problem. The source mutated; your agreement with the page broke; you detect it by watching the shape. This is the other end of the pipe. The source is fine. The page is intact, the selectors are correct, the structure is exactly what you expected. The thing that lied to you is the model , on the extraction step, when you asked it to pull structured fields out of a paragraph of human prose. Those two failures feel similar and t

A stack de autenticação em .NET fica sólida quando separamos duas responsabilidades: ✅ Argon2id para guardar senhas (hash irreversível, lento, memória-intensivo) ✅ JWT Bearer para provar identidade depois do login ✅ Validação de iss , aud , exp e assinatura em cada request ✅ Segredos fora do repositório (ambiente / Key Vault) Se o ecossistema .NET já oferece hosting, APIs e pacotes maduros, combinar Argon2 (referência da Password Hashing Competition , testável em argon2.online ) com JWT é o caminho natural para microsserviços e Web APIs. Neste artigo, mostro o fluxo registo → login → token → rotas protegidas com foco no que implementar no dia a dia. ⚠️ Observação importante JWT não substitui Argon2. Nunca coloque senha ou hash no payload do token. Argon2 protege a credencial na base de dados; JWT é sessão assinada com expiração. 🧠 Visão Geral Aspecto Argon2 (senha) JWT (sessão) Foco Resistir a offline cracking Autorizar requests após login Onde vive Coluna password_hash na BD Header Authorization: Bearer Algoritmo Argon2id (OWASP) HMAC-SHA256 ou RSA (config) Ferramenta de estudo argon2.online docs Microsoft JWT Bearer Runtime Biblioteca .NET (ex.: Konscious Argon2) Microsoft.AspNetCore.Authentication.JwtBearer Erro clássico MD5/SHA rápido na senha Token sem validar aud / iss 🧩 O que o Argon2 resolve (camada 1) O Argon2 é o vencedor da Password Hashing Competition — hoje a referência para novas passwords . 1️⃣ Hash irreversível com Argon2id var hash = hasher . Hash ( password ); await store . CreateAsync ( email , hash ); ✅ Salt único por utilizador ✅ Parâmetros m , t , p documentados no próprio hash ✅ Verificação com tempo constante ( FixedTimeEquals ) 2️⃣ Calibrar custo com consciência Em argon2.online podes experimentar memory cost e iterations — útil em laboratório. 📌 Em produção usa biblioteca auditada (.NET), não hashes de utilizadores reais em sites públicos. 3️⃣ O que não fazer na senha ✅ Não “criptografar” senha com AES reversível ✅ Não MD5 / SHA-1 / SHA-256

RecursiveMAS (arXiv 2604.25917) showed that agents sharing internal reasoning state outperform agents that share only final outputs. The average accuracy gain across benchmarks was 8.3 points. The mechanism: each agent passes not just its answer but the latent embeddings from its own reasoning process, and the next agent conditions on both. The paper is a good result. The catch is access. RecursiveMAS requires open-weight models with hidden states exposed at inference time. That rules out Claude, GPT-4o, and Gemini. I built a Claude-native version using the Anthropic extended thinking API. The core idea transfers: instead of passing latent vectors, pass the full thinking text. The paper calls it internal state sharing; the Claude version calls it thinking-block relay. The architecture problem Claude's extended thinking blocks carry an encrypted signature tied to the originating conversation. You cannot pass a signed thinking block into a different agent's messages array. The API rejects it. The workaround: extract the text from the thinking block and inject it as a regular user message. # Extract thinking text from Agent 1 thinking_text = next ( ( b . thinking for b in response . content if b . type == " thinking " ), "" ) # Inject into Agent 2 as regular context, not as a thinking block context = f " Prior agent reasoning: \n { thinking_text } " The signature does not transfer. The reasoning does. relay-structured: what I built first The first architecture was a Planner > Critic > Solver loop where each agent emits a compact mental model JSON instead of raw thinking text. Raw thinking at a 1024-token budget is often compressed and fragmented. The hypothesis was that 150 tokens of structured signal carries more information per token than 1024 tokens of compressed prose. The schema each agent emits: { "interpretation" : "how the agent read the problem" , "key_steps" : [ "step 1" , "step 2" ], "rejected_approaches" : [ "approach tried and discarded" ], "confidence" :

I run Sola, a travel app for people who move differently from the traveller the industry was built for. While building it, I kept hitting the same wall. The data I wanted to query did not exist. Not because nobody collected it, but because the schema underneath the whole industry never had a field for it. So on 27 May 2026 I sat down and audited Booking.com. The homepage form, the currency selector, a Bangkok search results page. I wrote down what it accepts and what it refuses. Then I looked at the new AI travel agents shipping on top of it. Here is what I found, and why it matters to anyone building in this space right now. The form is the spec Booking.com's homepage search bar accepts exactly four inputs: A destination, as a single text field A check-in date and check-out date, as one range An occupancy counter, defaulting to "2 adults · 0 children · 1 room" A search button That is the spec. An online travel agency (OTA) is a CRUD app over this spec, and Expedia, Agoda, and Hotels.com run the same four fields. Airbnb lets you skip the dates. The destination stays a single field everywhere. Think about what a spec encodes. The default occupancy is a couple. Not a solo traveller, not a parent with one child, not three generations, not seven people eating from one host's kitchen. The form cannot accept a circuit ("Bangkok, then Hanoi, then Jakarta" forces three separate searches). It cannot accept an open date ("October, not sure which week"). It has no field for the part of a trip where you sleep at family but spend money in restaurants. When you fill that form, you have not searched. You have submitted to a schema. Most of the world's travellers fail the schema before they fail the search. The data receipts I am a builder, so I went for counts, not adjectives. Everything below rendered on the platform on 27 May 2026. Currencies: 52 offered, about 180 in circulation. Eight currencies sit featured at the top of the dropdown. On the day I ran it the order was EUR, US

Repository: public-apis/public-apis What public-apis actually is public-apis is a community-curated directory of free and public APIs, maintained by contributors together with staff at APILayer. It is not a library, SDK, or gateway: there is no package to import and nothing to run in production. The repository is essentially one very large, structured README that catalogs APIs across roughly fifty categories, from Animals and Anime to Finance, Machine Learning, Security, and Weather. Each entry is a row in a table with five columns: the API, a short description, the authentication model ( apiKey , OAuth , or none), whether it serves over HTTPS, and whether it sets permissive CORS headers. That last detail is the part most engineers undervalue. Why engineers keep coming back to it The star count, now past 438,000, is less interesting than the metadata discipline. When you are prototyping and need a currency-exchange or geocoding endpoint, the Auth/HTTPS/CORS columns let you filter candidates before you ever open a browser tab. "No auth, HTTPS yes, CORS yes" tells you that you can call the endpoint directly from a front-end spike without standing up a proxy or registering for a key. For throwaway demos, hackathons, internal tools, and teaching material, that triage saves real time. The category index doubles as a map of what kinds of public data are actually available, which helps when you are scoping whether an idea is even feasible. How it is maintained Curation is manual and community-driven: changes arrive as pull requests against the README, governed by a contributing guide, with issues and PRs as the moderation surface. The project's primary language is Python, reflecting validation tooling that checks entries rather than any runtime you would consume. There is also a separate companion project that exposes the list itself as an API. The model is simple and has clearly scaled, but "manually curated" is both the strength and the weakness. Limitations worth statin

Every MCP server you run locally executes with your full filesystem and network permissions. That means the GitHub MCP server, the Slack one, that third-party tool you installed from npm last week — all of them can read your SSH keys, .env files, and credential stores by default. Anthropic just open-sourced the fix: sandbox-runtime , the sandboxing layer they built for Claude Code. One-line wrap, no Docker, OS-level enforcement. What actually changed srt (the Sandbox Runtime CLI) enforces filesystem and network restrictions on any process using native OS primitives: macOS : Uses sandbox-exec with dynamically generated Seatbelt profiles Linux : Uses bubblewrap for containerization + network namespace isolation Network filtering : HTTP/HTTPS traffic routes through an HTTP proxy; other TCP goes through SOCKS5 — both enforce your domain allowlists Install it: npm install -g @anthropic-ai/sandbox-runtime Wrap an MCP server in your .mcp.json — change command from npx to srt , move the rest to args : { "mcpServers" : { "filesystem" : { "command" : "srt" , "args" : [ "npx" , "-y" , "@modelcontextprotocol/server-filesystem" ] } } } Then configure what the process is actually allowed to touch in ~/.srt-settings.json : { "filesystem" : { "denyRead" : [ "~/.ssh" ], "allowWrite" : [ "." ], "denyWrite" : [ "~/sensitive-folder" ] }, "network" : { "allowedDomains" : [ "api.github.com" , "*.npmjs.org" ] } } The result: the MCP server can work in your project directory, talk to the domains it needs, and nothing else. Why this matters The threat model is real. An MCP server running compromised code — or simply a server with more ambient access than it needs — can exfiltrate your SSH keys, read your .env files, or phone home to arbitrary hosts. This isn't theoretical; it's the same class of supply-chain risk that exists for any untrusted npm package, except MCP servers are typically long-running processes with broad system access. srt is designed secure-by-default : processes start wit

I built a small web scraping framework in Rust, mostly with an AI doing the typing. It's called ferrous — a Colly-style collector: register CSS selector callbacks, queue URLs, write JSONL. About 700 lines. The pitch I kept hearing, and half-believed, was that Rust and LLMs are a good match now: the borrow checker is a correctness oracle the model can lean on, so the class of bugs that plagues AI-written Python just won't compile. That's true. It's also where the story gets uncomfortable, because the build was green and the code was still wrong. How I worked I'm not a Rust native. ferrous was partly an excuse to get fluent — build something real instead of reading about lifetimes — and partly a test of how far an LLM could carry the typing while I drove. The loop was plain: describe the next change in English, let the model write the Rust, read what came back, run cargo , move on. It kept observations.md as a running design journal, one entry per change, each with a short rationale for the decision it made. That setup has a soft spot, and it's the whole point of this post. When you drive a language you don't fully know, the only reviewer you've got with real authority is the compiler. Everything past that — is this idiomatic, is it the right abstraction, does it actually do what the journal claims — depends on already knowing what correct looks like. Which is exactly the knowledge a learner doesn't have yet. Keep that in mind through the next part, because it's the difference between the one bug I could have caught and the six I couldn't. The bug that the toolchain told me wasn't there I ship two fetch backends. The default goes through the Zyte API; an optional one, gated behind a wreq feature, makes direct requests with browser TLS emulation. Each has an example. After a refactor that added URL resolution — ctx.resolve_and_visit(href) so callbacks stop hand-building absolute URLs — I had the model update the examples to use it. It did, and it wrote up the change in

Built with OpenEUDI — open source (MIT), live on npm. GitHub: https://github.com/openeudi · npm: @openeudi/core and @openeudi/openid4vp Originally published on eidas-pro.com . The Developer's Dilemma You need to verify user identity in your application. Traditionally, that meant integrating a KYC provider — uploading documents, waiting for manual review, handling edge cases. With EUDI Wallets launching in December 2026, there's a new option. This comparison is written for developers who need to choose between — or migrate from — traditional KYC to EUDI Wallet verification. Architecture Comparison Traditional KYC Flow User uploads document → Your server → KYC API → Queue → AI/Manual review ↓ Result (minutes to days) ↓ Webhook to your server EUDI Wallet Flow Your server generates QR → User scans with wallet → Wallet authenticates ↓ Cryptographic VP sent back ↓ Result (2-5 seconds) Side-by-Side Comparison Aspect Traditional KYC EUDI Wallet (OpenEUDI) Verification time Minutes to 48 hours 2-5 seconds User effort Photo upload, selfie, manual entry Scan QR, tap approve Data you receive Full document images, extracted PII Only requested attributes (e.g., age_over_18) Data you store Required to store for compliance No PII storage needed API complexity REST + webhooks + polling REST + SSE (real-time) SDK cost Paid (per verification) Free (OpenEUDI is MIT) Production cost Per-verification pricing Managed service from EUR 49/mo Cryptographic verification You trust the KYC provider You verify the issuer's signature yourself Cross-border Provider-dependent All 27 EU member states Regulatory basis Provider-specific compliance EU Regulation 2024/1183 GDPR burden High (you store PII) Low (no PII retention) Offline capability No Yes (proximity/NFC flow) Integration Effort Traditional KYC (Typical) // 1. Create verification session const session = await kycProvider . createSession ({ type : ' identity ' , country : ' DE ' , documentTypes : [ ' passport ' , ' id_card ' ], redirectUrl

Quick answer: The Google Ads Transparency Center is a public registry of every ad Google runs — but it ships no API and no bulk export . To get the data programmatically you scrape it. A Google Ads Transparency scraper sends the same RPC call the website uses and returns every ad creative for an advertiser as structured JSON. The Apify Actor below does it for $0.0012 per ad (~$1.20 per 1,000), with the TLS fingerprinting, proxy rotation, and pagination handled for you. Google's Ads Transparency Center is one of the most underused datasets in marketing. Launched in 2023 under the EU Digital Services Act and parallel US pressure, it indexes every ad campaign currently running on Search, YouTube, Display, Shopping, Maps, and Play — keyed by advertiser. Google's own counter lists 300,000+ active creatives for a brand like Nike . For your nearest competitor, it's usually 50–500. The catch: there's no download button. Just an interactive UI that paginates 40 creatives at a time. If you want this as a CSV — for a competitor sweep, a trademark audit, or a RAG corpus — you have to extract it yourself. Here's what that actually takes, and how I shortened it to one API call. What is the Google Ads Transparency Center? 🔎 The Google Ads Transparency Center is a public, Google-operated registry that shows the ad creatives any verified advertiser is running, the date range each ad was shown, and roughly where. Google built it to comply with ad-disclosure regulation, so the data is public by design — you're reading the same registry a regulator would. What it gives you per advertiser: Every ad creative currently or recently live (text, image, video) The landing domain each ad clicks through to First-shown / last-shown timestamps and a rough impression count A deep link to each creative inside the Transparency Center What it does not give you: a search-by-keyword mode, region-filtered results from the server, or — crucially — an API. Does the Google Ads Transparency Center have an A

Most apps don't just call one API endpoint. They call a whole chain of them. For example, you might log in, get a token, and then pass that token to another service. Tracking these multi-step chains can get messy quickly. To help fix this, the OpenAPI Initiative created the Arazzo Specification . It gives us a standard way to link different endpoints into clear workflows. But writing these workflow files by hand in a regular text editor is tough. It is very easy to lose track of how data moves from one step to the next. That is why I built Arazzo Visualizer for VS Code. It is a free, open-source extension that makes the Arazzo spec visual and easy to use. Live Interactive Graphs The extension reads your workflow files and turns them into interactive maps on the fly. See Data Flow: Look at exactly how data moves between steps. Catch Errors Early: Spot broken paths before you even run your code. Clean Layouts: Navigate large workflows without getting lost in thousands of lines of text. Built-In Workflow Runner Seeing the map is great, but testing it is even better. The tool has a step-by-step runner built right into your editor. Run a single step or execute the whole chain. See real-time data payloads and HTTP headers. Watch requests happen live to pinpoint bugs fast. Give it a Try The project is fully open source, and you can grab it or check out the code using the links below: Download: Install it directly from the VS Code Marketplace . Source Code: Check out the repository, report bugs, or contribute on GitHub . Deep Dive: Read my full technical breakdown and design on Medium . If you are working with API chains, I would love for you to try it out. Drop your feedback in the comments below! Note: Arazzo v1.1.0 is out with official AsyncAPI support. I am currently updating the VS Code extension to support these new features. Stay tuned for future updates!

This is a complete guide with steps by step process to deploy the gRPC and how to use Azure API Management to import the gRPC API. It cover step‑by‑step guide to deploying a gRPC API on Azure API Management (APIM), grounded in the Microsoft documentation and a real-world deployment workflow. NOTE: This post is published already in GITHUB here. https://github.com/shailugit/apimGrpc/blob/main/README.md The API Management can expose gRPC services, but with important constraints: APIM supports gRPC by importing a .proto file and forwarding calls to a gRPC backend. gRPC requires HTTP/2 end‑to‑end. gRPC APIs are supported in Self-hosted gateway and not supported in APIM v2 tiers. You can't use the test console to test gRPC The major steps claissfied in two major steps Creating a gRPC server Calling the gPRC application using APIM 1. Creating gRPC Application Typical backend deployment steps include the following Create a .NET gRPC server application Create a .NET gRPC client application Test the setup locally Publish the .NET gRPC server to Azure WebApp and verify the service works directly over HTTPS Step-1 As a first step we will be building a .NET gRPC server application. You can skip this step in case you already have gRPC server application. If you would like to view .NET Core sample used for this sample project, please visit here . Step-2 As a second step we will be building a .NET gRPC client application. You can skip this step in case you already have gRPC client. If you would like to view .NET Core client used for this sample project, please visit the below here . Step-3 Once your client and server code is ready here are the steps to Test your application locally Step-4 Deploy the server to Azure WebApp To understand how-to deploy a .NET 6 gRPC app on App Service, please visit here . Please make sure to enable HTTP version, Enable HTTP 2.0 Proxy and add HTTP20_ONLY_PORT application setting as gRPC only work using http2.0 as shown below 2. Calling gRPC from APIM T

You hit "Submit Order" and nothing happens. The spinner just spins. Is it processing? Did the request get lost? You click again. If the API on the other end does not implement idempotency, you just placed two orders. Maybe two charges to your card. This is a solved problem — and the solution is simpler than you think. What Is Idempotency? An operation is idempotent if doing it multiple times produces the same result as doing it once. GET requests are naturally idempotent — fetching a resource does not change it. DELETE is also idempotent in practice. The trouble is POST and PATCH : create an order twice, and you get two orders. An idempotency key is a client-generated unique identifier (usually a UUID) that you send with a mutating request. The server stores this key with the result. If the same key arrives again — whether due to a retry, a network blip, or an impatient user — the server returns the cached result instead of executing the operation again. Implementing Idempotency on the Server Here is a minimal Express implementation backed by Redis: const express = require ( " express " ); const redis = require ( " ioredis " ); const { v4 : uuidv4 } = require ( " uuid " ); const app = express (); const cache = new redis (); app . use ( express . json ()); // TTL for idempotency records: 24 hours const IDEMPOTENCY_TTL = 86400 ; async function idempotencyMiddleware ( req , res , next ) { const key = req . headers [ " idempotency-key " ]; if ( ! key ) return next (); // optional on GET/DELETE const cached = await cache . get ( `idem: ${ key } ` ); if ( cached ) { const { status , body } = JSON . parse ( cached ); return res . status ( status ). json ( body ); } // Intercept the response to cache it const originalJson = res . json . bind ( res ); res . json = async ( body ) => { if ( res . statusCode < 500 ) { await cache . setex ( `idem: ${ key } ` , IDEMPOTENCY_TTL , JSON . stringify ({ status : res . statusCode , body }) ); } return originalJson ( body ); }; next ();

In this blog, I’ll give a clear brief summary of the n8n beginner course . You can watch the full video course on the official n8n website (link in the references below). What is n8n? n8n is a powerful workflow automation platform that combines AI capabilities with business process automation. It offers a node-based visual interface while giving you full control to write custom JavaScript or Python code directly in the canvas. APIs and Webhooks Understanding APIs An API (Application Programming Interface) allows different applications to communicate with each other. Almost every modern app has an API you can connect to. Example: Google Sheets API lets you read or update data in spreadsheets. When working with APIs, we make requests and receive responses . Components of an HTTP Request There are four main components: URL – The unique address of the resource (page, image, data, etc.). Includes: Scheme, Host, Port (optional), Path, Query Parameters (optional). Method – Defines the action you want to perform: GET – Retrieve data POST – Send data PUT / PATCH / DELETE – Update data (less common) Headers – Provide additional context (language, device type, location, etc.). Body – Contains data being sent (used mainly with POST requests). Authentication (Credentials) To prove you’re allowed to make a request: API Key (via query parameter or header) OAuth (most secure common method) HTTP Response Components Status Code – Tells if the request was successful: 200 = Success 401 = Unauthorized 404 = Not Found 500 = Server Error Headers – Metadata about the response (content type, length, expiration, etc.). Body – The actual data returned (usually JSON, HTML, or binary). What are Webhooks? Webhooks are used when an external service needs to notify your workflow automatically (e.g., every time a payment is made in Stripe). You provide a URL that receives a POST request when the event occurs. Nodes in n8n Nodes are the building blocks of every workflow. There are three main categor

Most RAG tutorials I found were either "pip install langchain and you're done" or 50-page academic papers. I wanted something in between — a pipeline I could actually explain in an interview, where I understood every line. So I built one from scratch. No LangChain, no LlamaIndex, no frameworks. Just FastAPI, FAISS, sentence-transformers, and an LLM API. Here's what I built, what worked, and what broke. The architecture PDF --> extract text (pypdf) --> chunk (500 char, 50 overlap) --> embed (MiniLM-L6-v2) | v question --> embed --> FAISS top-k search --> build prompt with chunks --> LLM --> answer + sources Five Python files, ~300 lines total: File Responsibility main.py FastAPI app, 3 endpoints, prompt engineering pdf_loader.py PDF text extraction via pypdf rag.py Chunking + embedding store.py FAISS vector store wrapper llm.py Swappable LLM client (Groq / OpenAI / Anthropic) How the upload works When you POST a PDF to /upload , three things happen: 1. Text extraction — pypdf reads each page and returns the raw text. Pages with no extractable text (scanned images) are skipped. 2. Chunking — each page is split into ~500-character chunks with 50 characters of overlap. The overlap prevents losing context at chunk boundaries. CHUNK_SIZE = 500 CHUNK_OVERLAP = 50 def chunk_pages ( pages ): chunks = [] chunk_id = 0 for text , page_num in pages : start = 0 while start < len ( text ): end = min ( start + CHUNK_SIZE , len ( text )) chunk_text = text [ start : end ]. strip () if chunk_text : chunks . append ( Chunk ( chunk_id = chunk_id , text = chunk_text , page = page_num )) chunk_id += 1 if end == len ( text ): break start = end - CHUNK_OVERLAP return chunks 3. Embedding — each chunk is embedded into a 384-dimensional vector using all-MiniLM-L6-v2 . This runs locally on CPU, no API call needed. Vectors are normalized so we can use inner product as cosine similarity. def embed_texts ( texts ): model = get_embed_model () # lazy-loaded singleton vectors = model . encode ( texts

It is straightforward to configure a minimal API to produce an OpenAPI document at build time . This runs the API during build, requests the OpenAPI document from it, and saves it to disk. The slightly trickier part is to put checks in Program.cs to exclude any startup code that cannot run at build time. This is typically done because configuration key/value pairs are not available at that time. For example: if (! isBuildTime ) { connString = builder . Configuration . GetConnectionString ( "AppDB" ) ?? throw new InvalidOperationException ( "Connection string 'AppDB' is not configured." ); builder . Services . AddDbContext < AppDbContext >( options => { options . UseNpgsql ( connString ); } ); } The question is how to deduce that the API has been launched at build time, i.e. isBuildTime should be true? The official way of doing this is to check that the assembly that invoked the API is "GetDocument.Insider" : var isBuildTime = Assembly . GetEntryAssembly ()?. GetName (). Name == "GetDocument.Insider" ; GetDocument.Insider.dll is the command line tool that automatically runs during build of the API if the .csproj includes the following reference: <PackageReference Include= "Microsoft.Extensions.ApiDescription.Server" Version= "10.0.7" > ... </PackageReference> This package is a shim. It only provides build targets and props and hooks into the build of the API to run the command-line tool dotnet-getdocument . This tool in turn runs the command line tool GetDocument.Insider that we check for. This is a pretty convoluted sequence: Microsoft.Extensions.ApiDescription.Server provides targets that run during build of the API. One of those targets runs the command line tool dotnet-getdocument That in turn runs the command line tool GetDocument.Insider That in turn runs the API and fetches the /openapi/v1/json (or other configured endpoint) to get the OpenAPI document and saves it to disk. Checking in Program.cs if the API was invoked by the assembly GetDocument.Insider.dll t

I came across Scrapling through a recommendation on X and decided to put it through its paces — not against a demo page, but against Lazada Singapore, a production site with Google reCAPTCHA and a custom slider verification. The setup: a single 4GB VPS, no residential proxies, no credits, just open-source tools. Here's the full journey: installation pitfalls, wiring it into an AI agent, choosing the right browser for the job, and the real-world benchmarks that followed. What Is Scrapling? Scrapling is an adaptive web scraping framework for Python (BSD-3, v0.4.8). It handles everything from single HTTP requests to full-scale concurrent crawls. What sets it apart from the BeautifulSoup/Scrapy world: Adaptive element tracking — saves fingerprints of targeted elements and relocates them after site redesigns using similarity scoring. Your scrapers survive CSS changes without maintenance. Three fetchers, one API — HTTP ( Fetcher , curl_cffi), browser ( DynamicFetcher , Playwright Chromium), and stealth ( StealthyFetcher , Chromium + anti-bot patches). Swap with one line. Spider framework — Scrapy-like API with async, concurrent crawling, Ctrl+C pause/resume via checkpoint persistence, multi-session support. MCP server — 14 tools exposed natively for AI coding agents. Your agent can call mcp_scrapling_get , mcp_scrapling_fetch , mcp_scrapling_stealthy_fetch directly. It's open source, pip-installable, and designed to be the backbone of a scraping stack — not just another tool in the toolbox. Installation on a 4GB VPS This is where the real story starts. The VPS has 4GB RAM, 2 vCPUs, 77GB disk, and runs an AI agent gateway (615MB baseline). Every browser installation decision matters. What we installed pip install scrapling[fetchers,ai] # HTTP + Chromium + MCP server scrapling install # Downloads Playwright browsers This pulls in Playwright Chromium, Firefox, and WebKit (~1.3GB disk), plus curl_cffi for HTTP requests and patchright (Playwright fork) for browser automation.