标签：#dev

Building a Self-Healing Data Pipeline with Event-Driven Idempotence Building a Self-Healing Data Pipeline with Event-Driven Idempotence A senior engineer’s sketchbook: a project I shipped to production that turned brittle batch jobs into resilient, observable, and self-healing data pipelines. The core idea is to treat data processing as an event-driven system with strict idempotence guarantees, automated reconciliation, and graceful recovery. The result was a measurable reduction in retry storms, faster time-to-insight for dashboards, and a foundation that scales with data volume without blowing up operator toil. Overview and motivation Problem: A data ingestion workflow relied on nightly batch jobs that often overlapped, causing late-arriving data, duplicate processing, and fragile error handling. Observability was ad-hoc, retries were uncoordinated, and operators spent days triaging failures. Solution: Reframe the pipeline around event streams with idempotent processing, push-based checkpoints, and a lightweight orchestration layer that can recover from partial failures without human intervention. Impact: 40% reduction in data latency for dashboards, 60% fewer retry-induced incidents, and a robust foundation for future scaling. Architecture at a glance Data sources emit events to a durable message bus (Apache Kafka or a cloud equivalent). A set of microservices subscribes to the stream, each performing a deterministic, idempotent transformation. A central idempotence layer guarantees that repeated events do not mutate state or produce duplicate side effects. A reconciliation service audits the target data store against the event log and replays or compensates as needed. Observability stack with per-event tracing, lineage, and anomaly detection. Key design principles Idempotence by default: Every processing step should be safe to replay. Use deterministic keys and avoid non-idempotent side effects without compensation. Exactly-once semantics where feasible: Impleme

Vercel's built-in cron triggers your serverless functions on a schedule. For simple use cases it works. But it has no failure alerts, no execution history on the Hobby plan, and no way to know whether your function actually completed successfully — only that it was called. Where Vercel cron falls short Vercel cron works by invoking one of your API routes on a schedule defined in vercel.json . The invocation is fire-and-forget — if your function times out, throws an error, or returns a non-2xx status, you get no alert. You find out when a user reports something is broken. The specific gaps developers run into: No failure alerts. Vercel does not send an email or webhook if your scheduled function fails. No execution history on Hobby. The free plan does not retain cron execution history. Timeout ceiling. Functions are subject to the same timeout limits as all serverless functions — 10 seconds on Hobby, up to 300 seconds on Pro. HTTP-only. Vercel cron calls an HTTP endpoint on your app. You cannot schedule arbitrary background work outside your deployment. No heartbeat monitoring. Even if your function is called successfully, you have no built-in way to verify it completed its work — only that it was invoked. Minimum 1-hour interval on Hobby. Sub-hourly schedules require a paid plan. If you are hitting any of these limitations, you need an external tool. Comparison at a glance Tool Schedules jobs Failure alerts Heartbeat Uptime monitoring Free tier Vercel built-in ✓ ✗ ✗ ✗ ✓ (1h min) Tickstem ✓ ✓ ✓ ✓ ✓ Upstash QStash ✓ ✓ (retries) ✗ ✗ ✓ Inngest ✓ ✓ ✗ ✗ ✓ cron-job.org ✓ ✓ (basic) ✗ ✗ ✓ Tickstem — cron + heartbeat + uptime in one API key Best for: developers who need scheduling, failure alerts, heartbeat monitoring, and uptime checks without managing multiple tools. Tickstem is an external HTTP cron scheduler with built-in monitoring. You register your Vercel endpoint as a cron job, and Tickstem calls it on your schedule — every minute if needed, regardless of your Vercel

Swap debugging war stories\n\nI have been living in webhook + PHP + email land (Vapi, OpenAI, PHPMailer, CSV logs).\n\nDrop your stack in a comment (even one line). I will reply with the first three places I would look for a silent production failure.\n\nNo sales pitch — trying to meet dev friends who ship unglamorous integration work.\n\nMy builds: elionmusic.com · prayerauthority.com

Stop shipping a 1990s C library to compute planets. Xalen is the pure-Rust, Apache-2.0 replacement for Swiss Ephemeris. If your app does astrology, you already know the dependency. Swiss Ephemeris: a C library from the 1990s, a folder of binary .se1 data files you have to ship and locate at runtime, and a license that is either AGPL or you pay for a commercial seat. For 30 years it was the only serious option, so everyone just swallowed the cost. That era is over. Xalen Ephemeris is a full planetary engine written in pure Rust, with no unsafe in the core engine (the only unsafe lives in the optional FFI, Node and WASM binding crates), released under Apache-2.0. No C toolchain. No data files to ship. No copyleft clause waiting for the day you try to make money. It is built to replace Swiss Ephemeris in production, not to admire it from a distance. Python is live on PyPI and the Rust crates are live on crates.io: # Python pip install xalen # Rust cargo add xalen-ephem xalen-time xalen-ayanamsa xalen-vedic Node and WASM build straight from the repo. Repo: https://github.com/vedika-io/xalen-ephemeris Switching takes one line Xalen ships a pyswisseph-shaped API on purpose. Migrating an existing codebase is a find-and-replace: # before import swisseph as swe # after import xalen.swe as swe jd = swe . julday ( 1990 , 6 , 15 , 10.5 ) xx , ok = swe . calc_ut ( jd , swe . SUN , swe . FLG_SWIEPH | swe . FLG_SPEED ) # same argument order, same SE_/SEFLG_/SIDM_ constants, same tuple layout Your function calls do not change. Your data-file directory disappears. Your license problem disappears. Xalen vs Swiss Ephemeris Line them up and the gap is hard to miss. Swiss Ephemeris is C from the 1990s, shipped as a native library you compile and link, fed by .se1 data files you have to bundle and locate at runtime, under AGPL or a paid commercial license. Xalen is pure Rust with no unsafe in the core engine, thread-safe, with no native dependency and no data files for the analytical eng

Not a job application — a peer search .\n\nI ship PHP/JS/AI production sites. People around me cannot relate to webhook failures at 2am.\n\nI want friends who are better coders than me in some layers.\n\nReply with what you are building: https://dev.to/elionreigns/looking-for-dev-friends-who-actually-get-how-much-work-this-is-3m0c

This is a submission for the GitHub Finish-Up-A-Thon Challenge What I Built An AI interview coach that runs in your browser. No backend. No accounts. No subscriptions. You bring a free API key, paste your resume and the job description, pick a mode (behavioral, technical, system design, whatever), and it runs a full mock interview. Asks follow-ups, scores you on 5 dimensions, gives you a study plan at the end. I built the first version for the Gemma 4 DEV.to Challenge last month. It kinda worked. But I wouldn't have used it myself, and that bothered me. Live: hajirufai.github.io/gemma4-interview-coach Repo: github.com/hajirufai/gemma4-interview-coach Demo What you get now: 🗣️ 6 practice modes — behavioral (STAR method), technical, system design, online assessment sim, certification prep, case studies 🎤 Voice mode — talk into your mic, hear feedback read aloud. Because typing answers in a mock interview is weird. - 📄 Resume + JD aware — paste both, get questions about your actual experience gaps 📸 Screenshot upload — snap a coding problem or whiteboard and discuss it 🌐 4 AI providers with free tiers (Google AI Studio, OpenRouter, NVIDIA NIM, Hugging Face) 🌙 Dark mode, session history, timer, downloadable reports ## The Comeback Story ### Where it was before I threw this together during the Gemma 4 Challenge in May. Classic hackathon energy — built the core chat loop, got 6 mode cards looking nice, slapped on dark mode, shipped it. Then I hit the wall. Google AI Studio was throwing 500 errors during peak hours. The only option was "refresh and hope." No voice input, so you're typing interview answers like it's a customer support chat. And if you had a typo in your API key? Good luck figuring out why nothing's working. It was a demo, not a tool. ### What actually changed I came back with one rule: make this something I'd actually use to prep for my own interview. Voice mode was the big one. I'm prepping for a senior cybersecurity engineering interview right now. Typing

submitted by /u/ReasonableLoss6814 [link] [留言]

A client asked me for a simple thing. Not ChatGPT. Not an agent. Not a multimodal assistant that can explain invoices, generate React components, and write poetry in three languages. Just a small classifier embedded into a website. The job sounded boring in the best possible way: take some text, classify it, return a result, keep it fast. So I started looking at the usual solutions. And then I had one of those moments where you stop reading documentation, lean back, and ask: Are we seriously doing this? Because the answer I kept running into looked like this: download a huge runtime download a huge model initialize a big ML stack then classify one small piece of text In one setup, the path was getting close to something like 250 MB per user . For a simple classifier. On a website. From a server. Every time. No. Sorry. That is insane. The problem The web has a strange habit now. You ask for one small AI feature, and the answer is often: bring the entire construction company. But sometimes I do not need a construction company. I need one person on the construction site. One task. One tool. One result. This is especially true for simple classification, embeddings, semantic search, routing, filtering, ranking, small local decisions. Not every AI problem needs an LLM. Not every website needs a full inference engine. Not every user should pay a 250 MB download tax because we were too lazy to think smaller. So I started digging I wanted something simple: runs in the browser does not require a server for inference small enough to actually ship works with transformer-style models can tokenize text can run BERT-like forward inference can produce embeddings or classification input does not bring ONNX Runtime, Candle, ndarray, or half the internet with it At first I thought: “Surely someone already made the tiny version.” There are great tools out there. Transformers.js is powerful. ONNX Runtime Web is powerful. Candle is powerful. But that was exactly the problem. They are pow

I'll be honest, when I first heard about Astro, I was skeptical. Another JavaScript framework? I already had React, Next.js was doing fine. Why bother? Then I actually used it. I built the website for Zimnovate, my AI-native digital product studio based in Harare, Zimbabwe, with Astro, ran it through Google PageSpeed Insights, and got scores I'd never seen before on a site I actually built myself. That changed everything for me. Let me tell you what Astro is, why it's architecturally different, and why it's worth learning, especially if you care about performance and SEO. What Even Is Astro? Astro is a web framework built around one radical idea: ship zero JavaScript by default . Most modern frameworks (React, Vue, Svelte) are component-based and hydrate the entire page on the client. Even if your page is mostly static content, the user's browser still downloads and runs JavaScript to render it. Astro flips this. It renders your components to pure HTML at build time. JavaScript only runs in the browser when you explicitly need it, and only for the specific components that need it. This isn't just a config option. It's the core architecture. The Architecture: Islands Astro uses a pattern called Islands Architecture . Think of your page as a static ocean with interactive "islands" floating in it. The ocean (static content, headings, text, images) ships as plain HTML. The islands (a navbar with a dropdown, a contact form, a live counter) are the only parts that hydrate with JavaScript. --- // This runs only at build time zero runtime cost const services = await fetch('/api/services').then(r => r.json()) --- <html> <body> <!-- Pure static HTML, no JS needed --> <h1>Zimnovate</h1> {services.map(service => <ServiceCard service={service} />)} <!-- This island hydrates only when visible --> <ContactForm client:visible /> </body> </html> The client:visible directive tells Astro: "only load this component's JavaScript when it scrolls into the viewport." You get full interacti

There is a moment every embedded Linux developer hits eventually. You have spent days building something that works beautifully — a sensor pipeline, a streaming server, an MQTT client — and then you reboot the device and everything is silent. Nothing started. You SSH in, manually run your script, and it all comes back to life. The hardware is fine. Your code is fine. You just have no way of automatically running it. That is the gap systemd fills. It is the init system on virtually every modern Linux distribution, and on embedded Linux systems like the Raspberry Pi it is what decides what runs at boot, what gets restarted if it crashes, and where all the logs go. Once you understand how to write a service file, your applications stop being fragile scripts you need to babysit and start being first-class system services that survive reboots, network drops, and unexpected crashes. This tutorial builds up from the simplest possible service file to a production-ready configuration, explaining every line along the way. By the end you will have a service running your own Python application, logging to the system journal, and automatically restarting itself after failures. See Complete Tutorial in Github: Systemd Services Tutorial What systemd Actually Does Before writing any configuration, it helps to understand what problem systemd is solving, because the design of service files makes much more sense once you see the underlying model. When your Raspberry Pi boots, the Linux kernel starts and immediately hands control to process ID 1 — the very first user-space process. On modern systems, that process is systemd . Everything that happens next — mounting filesystems, bringing up the network, starting your application — is orchestrated by systemd. It reads configuration files called unit files that describe what should be started, when, in what order, and what to do if something goes wrong. A service file is just one type of unit file (there are also unit files for timers, so

On a website of mine I use around ~50 different tiny SVG icons (1-2KB) per page which I store on a seperate subdomain where I host all my static files that I use across multiple websites. I am wondering what would be the best way to serve those. SVG sprites (a single svg file with each icon wrapped in a symbol tag): Not supported from crossorigin unless weird javascript hacks are being used. 1 request per icon : Seems a bit crazy to do 50+ requests for tiny files? Even with modern http2 this seems to slow the site down and feels like it causes unecessary overhead. Bake SVG into html : This feels the fastest, but it adds like 50kb (a bit less when gzipped) to every html page which seems unecessary as well. How do you do it? What is considered best practice? submitted by /u/Nonilol [link] [留言]

Building a Real-Time WebSocket-Based Chat Server with Rust and WASM Building a Real-Time WebSocket-Based Chat Server with Rust and WASM In this tutorial, you’ll build a scalable, real-time chat server using Rust on the backend, WebSocket for bidirectional communication, and WebAssembly (WASM) for a fast, interactive frontend. You’ll learn how to structure a minimal, production-ready system with clean code, testable components, and practical deployment considerations. Overview Goals: Real-time messaging with low latency Safe, fast backend implemented in Rust Frontend capable of connecting via WebSocket and rendering messages efficiently Basic authentication, message persistence, and reconnection handling Testing strategies for end-to-end and unit tests Tech stack: Backend: Rust, Warp or Actix-Web, tokio, tungstenite or tokio-tungstenite for WebSocket Frontend: Rust + WASM (via wasm-bindgen) or a lightweight JS client Persistence: SQLite or PostgreSQL for message history Deployment: containerized (Docker), with a simple reverse proxy (Nginx) in front Prerequisites Rust toolchain installed (rustup, cargo) Basic knowledge of Rust and asynchronous programming Node.js/npm if you choose a JS frontend (optional since you can use Rust WASM) SQLite or PostgreSQL installed locally for testing 1) System design and data model Clients connect via WebSocket and join a chat room. The server maintains in-memory state for active connections and broadcasts messages to all connected clients in the same room. Messages are persisted to a database for history. Reconnection: clients reconnect on network hiccups; server replays recent history upon join. Scalability note: for multiple instances, use a message broker (Redis pub/sub) to broadcast messages between workers. Data model (simplified) Users: id, username Rooms: id, name Messages: id, room_id, user_id, content, timestamp 2) Backend: Rust WebSocket server Key components HTTP upgrade to WebSocket Per-room broadcast hub Connection manag

StudyQuiz has moved forward since the first frontend MVP release. This update focuses less on adding major new features and more on making the app smoother to use, safer to change, and more reliable in production. What’s New Logout functionality Call-to-action section on the user dashboard Edit and delete functionality for questions and answers Guest Mode restrictions for editing and deleting questions and answers Integration tests for core backend workflows Improved Enhanced quiz user experience and feedback Improved quiz creation user experience Improved navigation across the application More reliable page reload behavior for protected routes Fixed Fixed 500 errors when reloading protected frontend pages Fixed production database session configuration issues Improved reliability around authentication-protected routes Why This Release Matters This release is mainly about stability and maintainability. I added integration tests and a GitHub Actions workflow so future changes are checked automatically before they silently break existing behaviour. StudyQuiz now feels closer to a maintainable product rather than just a working prototype. Coming Next The next major focus is slide upload and AI-assisted quiz generation, allowing users to generate structured quizzes more directly from their study materials. Repo: github.com/aissa-laribi/studyquiz Live app: https://www.studyquiz.co

Most SaaS founders build the dashboard first. It looks impressive in demos, investors love screenshots, and it feels like real progress. We did the opposite. Here's why. The real reason approvals fail When I started building TeamAutomation, I interviewed a dozen people about their approval process. Every single one said the same thing — approvals don't fail because people reject them. They fail because nobody follows up. The requester sends the request. The approver gets busy. Nobody wants to be the annoying person who keeps pinging. Days pass. Project blocked. A dashboard showing "pending approvals" doesn't fix this. The approver still has to remember to open it. Nudges are the product We ship automatic reminders at 24 hours, 3 days, and 7 days — directly in Slack where the approver already lives. No new app to open. No new habit to build. The accountability shifts from the requester to the system. That's the whole unlock. What we learned Build the thing that changes behavior first. The dashboard is just reporting. Nudges are intervention. If you're building any kind of workflow tool, ask yourself — what happens when nobody does anything? Your answer to that question is your core feature. What's next Still in early beta. Slack Directory approval pending. Zero users, full transparency. If you're dealing with approval chaos in your team, drop a comment — happy to give early access.

After migrating Spark pipelines to Azure Kubernetes Service, two infrastructure settings interacted destructively: spark.kubernetes.local.dirs.tmpfs=true backed shuffle spill with RAM instead of disk, and a hard podAffinity rule forced all executors onto one node. Together, they caused repeated OOM kills invisible to standard diagnostics. By Pranav Bhasker

Hey everyone, I recently landed a freelance gig to build out a complete SaaS platform from scratch. The client is incredibly generous and is treating this as an opportunity for me to learn and implement things as we proceed with development. I want to make sure I choose the right architecture from day one. I can't reveal the exact idea, but it functions as a two-sided platform connecting service providers with end-users. I need to pick a tech stack that plays to my strengths while keeping infrastructure costs as close to zero as possible. Here is my profile and the project constraints: - My Background: I am a frontend-heavy developer. My backend knowledge is only "okay-ish," so I’m looking for a stack that minimizes complex backend boilerplate and dev-ops headaches. - Timeline: I have 4 to 5 months maximum to build and fully deploy the MVP. - Strict Budget: The monthly budget for deployment, database, and any necessary transactional emailing cannot exceed $40/month for the MVP phase. - Expected Scale: To start, the web app needs to comfortably sustain 100 service providers and roughly 1,000 active users. - Payments and subscription : Stripe - Team Size: It's just me (Solo dev). Based on current trends, I've been leaning toward a meta-framework (like Next.js) paired with a BaaS (like Supabase or Firebase), but I’d love to hear from folks who have recently shipped something similar. Questions for the community: What specific tech stack would you recommend that allows a frontend dev to move fast without getting bogged down in backend setup? - Are there specific databases, ORMs, or auth providers you'd suggest that will confidently keep me under that $40/month limit for my expected user count? - Any hosting or deployment "gotchas" I should watch out for when launching a two-sided platform on a shoestring budget? Appreciate any guidance you can share! Note : Used AI for articulation. submitted by /u/chutneypow [link] [留言]

Hi r/webdev , As web developers, we frequently get stuck fighting the tools we use for e-commerce. You find a great framework, but it forces you to use a specific database, or it only deploys nicely to one cloud platform. I built Storecraft to solve this frustration. It is an open-source, headless e-commerce core that treats both the database and the runtime hosting environment as completely swappable dependencies. Why it makes building easier: Deploy Anywhere: It runs identically on traditional servers (Node), modern runtimes (Deno, Bun), or lightweight edge infrastructure (Cloudflare Workers). Choose Your DB: Switch between Postgres, MongoDB, or SQLite by simply changing your driver configuration. Purely Headless: Connect it to any frontend stack you prefer via a clean, predictable API. Whether you are building a small storefront for a local client or a globally distributed edge application, the engine adapts to your infrastructure—not the other way around. Repository link: https://github.com/store-craft/storecraft website: https://storecraft.app/ I'd love to get your feedback on the setup ergonomics. What are the biggest pain points you usually hit when deploying open-source e-commerce backends for your clients? submitted by /u/hendrixstring [link] [留言]

We run Quackr, a virtual phone number platform that lets developers and individuals receive SMS verifications without exposing a real number. We just published our first inventory transparency report and the data was surprising enough that we thought the dev community would find it useful. The Numbers Right now, 97.6% of our entire virtual phone number inventory is actively rented. 66,214 out of 67,815 numbers assigned and in use across 15+ countries. Over 1,000 numbers available at any given moment but they move fast. That utilisation rate tells you something about how the market has shifted. Virtual numbers are no longer a niche throwaway tool. Developers, businesses, and privacy-conscious users are holding them long term. What Developers Actually Use Virtual Numbers For The obvious use case is SMS verification during testing. Spin up a number, verify an account in staging, move on. But that is not what drives the bulk of demand on our platform. The real volume comes from: Multi-account management — developers and businesses running multiple instances of platforms that require unique phone verification per account. Privacy layers in production apps — applications that need to verify users without collecting their real numbers. A virtual number sits between the user and the platform. Automated verification pipelines — this is where our API and MCP Server come in. If you need to provision numbers programmatically and retrieve OTPs without manual intervention, this is the use case we built for. Geographic flexibility — needing a UK number from Australia, a US number from Ukraine, or any combination that your real SIM cannot provide. The OTP Blocking Problem Something worth knowing if you are building anything that involves SMS verification: platform-level VoIP blocking has become significantly more aggressive over the past two years. WhatsApp, Telegram, Google, and TikTok all run detection on incoming verification requests. A VoIP number gets flagged and the OTP simp

This is a submission for the GitHub Finish-Up-A-Thon Challenge I wired edge-context-mode into my own...

Node.js will change its release schedule starting with version 27 in October 2026, moving from two major releases per year to one. All releases will become Long-Term Support (LTS), removing the distinction between odd and even versions. An Alpha channel for early testing will also be introduced. This decision addresses maintenance challenges and aims to align with user needs. By Daniel Curtis