标签：#Data

After months of development, I'm excited to share DBNavigator, a cross-platform database IDE that I've been building from scratch using Java and JavaFX. ✨ Current Features ✅ PostgreSQL support ✅ MySQL support ✅ Modern Datagrip-inspired UI ✅ Multi-tab SQL editor ✅ Syntax highlighting ✅ Schema explorer ✅ Query execution ✅ Professional dark theme ✅ Cross-platform (Windows, Linux & macOS) This project has been an incredible learning journey in desktop application development, JavaFX UI design, database connectivity, and IDE architecture. I'm sharing it with the developer community because I'd genuinely appreciate your honest feedback. I'd love to hear your thoughts on: UI/UX design Performance Missing features Overall developer experience Architecture and code quality Any bugs or improvements you notice Whether you're a Java developer, DBA, or someone who works with databases every day, your feedback would mean a lot and help shape the next version of the project. ⭐ If you find the project interesting, please consider giving it a star on GitHub. GitHub: DBNavigator Thank you for taking the time to review it. Every suggestion, issue report, and critique is greatly appreciated! 🙌

Framework told "all" of its customers that hackers accessed their names, email addresses, phone numbers, and physical addresses in a data breach.

The Quest Begins (The "Why") Honestly, I still remember the first time I tried to build an autocomplete widget for a side‑project. I had a list of 200 k product names, a simple filter that ran on every keystroke, and the UI felt like wading through molasses. Each keypress triggered a full scan of the list, and with a few users typing at once the browser would start to lag. I was stuck in a loop that felt like the infamous “boss fight” where you keep hitting the same pattern over and over, hoping for a different outcome. I kept asking myself: There has to be a smarter way. Why am I re‑checking the same prefixes again and again? If ten users type “tea”, why do I walk through the whole dictionary ten separate times? That question turned into a mini‑quest, and the treasure at the end was the trie data structure. The Revelation (The Insight) Look, the magic of a trie isn’t that it’s some exotic tree; it’s that it stores words by their shared prefixes . Imagine you have the words “cat”, “car”, “cart”, and “dog”. In a trie you’d have a root node, then a c branch that splits into a → t (for “cat”) and a → r → t (for “cart”), while “dog” lives on its own d → o → g path. Every common prefix is stored once , and you can walk down the tree following the characters of a query to land exactly at the node that represents all words with that prefix. Why does this give us O(L + K) time for autocomplete, where L is the length of the prefix and K is the number of results? Walking the trie follows the prefix character‑by‑character → O(L). From that node we just need to collect all words in its subtree. If we keep a list of words at each node (or run a DFS), we touch each result once → O(K). No extra work for words that don’t share the prefix. Contrast that with the naive filter approach: O(N × L) where N is the total dictionary size. For a large N, the trie is a game‑changer—it’s like switching from swinging a blunt sword to wielding a lightsaber that cuts through the prefix forest in

If you are building anything that touches European business data — onboarding, invoicing, KYB, fraud checks — you will eventually ask the same question I did: which countries can I actually get company data from, for free, without an account? I could not find this written down anywhere, so I worked it out the hard way while building a supplier checker. Here it is. The baseline: VIES The European Commission runs VIES , which validates VAT numbers across all 27 member states plus Northern Ireland ( XI ). It is free, it needs no key, and it is the obvious starting point. Two things about it are worth knowing before you build on it. It answers one question: is this VAT number currently registered. It does not tell you the company is solvent, trading, or that it has not been struck off. A company in liquidation keeps a cleanly resolving VAT number for months, because deregistration and insolvency are run by different authorities on different timetables. Name and address are returned for 25 of the 28 jurisdictions, not all of them. Germany and Spain confirm registration but publish no company name through VIES. I tested three valid numbers for each before accepting that. For those two, a yes/no is genuinely all you can honestly show. Where you can go further, free Ten countries publish enough through a national register to add something meaningful on top of VIES: Country Free register Reports company state Reports VAT-active Romania yes yes yes Poland yes — yes Slovenia yes — yes Estonia yes yes — France yes yes — Greece yes yes — Bulgaria yes yes — Latvia yes yes — Czechia yes — — Finland yes — — Company state means the register tells you whether a business is inactive, in liquidation, bankrupt, insolvent, terminated or struck off. This is the valuable column, and only six countries have it. VAT-active matters more than it sounds. VIES cannot distinguish "this is a real company that is not VAT-registered" from "this number belongs to nobody". Three registers can. Note th

Why Sports Data Is Harder Than Most People Think Building believable cross-era simulations turned out to be less about the engine and more about the data underneath it. Here is what we learned. MicroLeague Dev Blog, Vol. 3 By Eddie Solar When we started building MicroLeague Sports, I assumed the simulation engine would be the hard part. The vision was ambitious enough to justify that assumption. Let fans ask whether the 1996 Bulls beat the 2017 Warriors. Whether the 1985 Bears could slow down Patrick Mahomes. Which Cowboys team was actually the greatest. Teaching software to play those games across eras felt like the mountain. I was wrong about which mountain it was. The engine is hard, but it is a solvable, bounded kind of hard. The data underneath it is a different animal. Like most developers approaching this for the first time, we figured sports data was largely a collection exercise: gather historical teams, player stats, schedules, and box scores, feed it to the model, done. That assumption fell apart almost immediately, and the reason it fell apart is the subject of this article. Sports data is not a collection problem. It is an identity problem. Franchises do not stay the same thing. Players are not one entity. And the historical record does not agree with itself. The Real Problem Is Modeling Identity Over Time Volume 2 covered the era problem: statistics are confounded by the conditions that produced them, so a raw number pulled across decades lies to you. That is a normalization challenge, and it is real. But normalization assumes you already know what you are normalizing. Before you can compare the 1992 Cowboys to the 2023 Chiefs, your system has to have a confident answer to a more basic question: what exactly is a "team," and what exactly is a "player," when your dataset spans a hundred years? Those sound like trivial questions. They are not. They are the questions that ate most of our early engineering time, and getting them wrong quietly corrupts ever

I have been building machin for a while — a Go-flavored, type-inferred language that compiles through C to a single native binary. It has grown a lot recently, and I wanted to answer the obvious question honestly: does it beat Rust and Zig at anything? It does, at two things, decisively. But the first thing I found was not a win. It was my own benchmark quietly lying to me, and the number it was lying about was the best one I had. The benchmark was measuring the order I ran things in machin's repo has had a bench/native-speed suite for months: four compute kernels — recursive fib, a mandelbrot, a sieve, a big integer loop — written in machin, Rust and Zig, producing byte-identical output, so the timing compares the same computation three ways. The published result claimed machin won the integer loop by 20-25% . That claim also shipped inside machin guide , which is what every coding agent reads to learn the language. When I re-ran it, the margin was gone. Not shrunk — gone. So I read the harness instead of the output: for kernel in kernels : for lang in [ machin , rust , zig ]: for _ in range ( 5 ): # all 5 machin, THEN all 5 rust, THEN all 5 zig time ( binary ) It ran every sample of one language before starting the next. On a laptop that heats up and down-clocks during a three-second kernel, that does not measure the languages. It measures who had the misfortune of running last . Zig always went last. Zig always looked slowest. The fix is four lines — interleave the rounds, rotate who starts each one. Here is what my headline number did: intsum 10^9 before (blocked) after (interleaved) machin 2832 ms 3079.7 ms rust 3764 ms 3223.8 ms zig 3556 ms 3189.7 ms "machin +20-25%" machin +3% = a TIE A 20-25% win became a tie. I deleted the claim from the README and from machin guide . The harness now also refuses to declare a winner inside a 3% band, because the worst run-to-run spread I measured was 41% of the min sample. Calling winners inside that is how benchmarks start

Artificial intelligence is rapidly changing how engineering teams respond to production incidents, offering the ability to summarize incident channels, analyze unfamiliar code, suggest remediation steps, generate pull requests, and increasingly assist with diagnosis. By Craig Risi

Classic Machine Learning Through the Eyes of an SRE — Part 3 The random forest is the first ML algorithm that made me feel at home. Not because of the math — because it's an SRE idea wearing a stats costume. Many independent workers. No single point of failure. Majority vote. If one worker goes weird, the fleet absorbs it. We've been building systems this way for decades; the forest just applies it to prediction. The problem it exists to fix Last article: a single decision tree is readable but unstable — small data change, whole tree flips, explanation rewrites itself. That instability is variance, and it's exactly what scared me about trusting one tree in production. The forest's move: grow hundreds of trees, each on a random resample of the data, and — this is the part that matters — force each split to choose from only a random subset of features. That second randomization is the whole difference between a random forest and plain bagging. Bagging alone gives you many trees on resampled data, but if one feature is strongly predictive, every tree grabs it first and they all end up looking alike. Starving each split of features is what makes the trees genuinely different from each other. The randomness isn't sloppiness. It's manufactured disagreement. The instability doesn't get fixed. It gets CANCELLED. Each tree is still jumpy, but they're jumpy in different directions, and the average is calm. What surprised me No new loss function. Each tree still minimizes impurity exactly like a lone tree. The forest adds zero new objectives. The entire gain is a bias-variance bargain: variance drops hard, bias barely moves. You give up readability and get back trustworthiness. Embarrassingly parallel. Trees are independent, so training scales horizontally — throw cores at it. Boosting, its sequential cousin, is the opposite: each model depends on the last. Map-reduce versus a pipeline. The smoothness illusion. A forest's decision boundary looks smooth, almost like regression'

We’ve all been there: you're crushing your workouts, feeling like a beast, and then suddenly— bam . You can’t get out of bed, your resting heart rate is through the roof, and your motivation has evaporated. Welcome to Overtraining Syndrome (OTS) . In the world of sports science, Heart Rate Variability (HRV) is the gold standard for tracking recovery. By analyzing the tiny fluctuations between heartbeats (R-R intervals), we can peek into our Autonomic Nervous System (ANS). Today, we’re going to build a Python-based pipeline to fetch data from the Oura Cloud API , calculate key HRV metrics like SDNN and RMSSD , and use an Isolation Forest model to detect when you're pushing a bit too hard. Whether you're a biohacker or a developer interested in wearable data analysis , this guide will show you how to turn raw health data into actionable recovery insights. The Architecture: From Pulse to Prediction 🏗️ Before we dive into the code, let's visualize how the data flows from your finger to our anomaly detection model. graph TD A[Oura Ring] -->|Sync| B(Oura Cloud API) B -->|Raw R-R Intervals| C{Data Preprocessing} C -->|Filtering Artifacts| D[Feature Extraction] D -->|SDNN & RMSSD| E[Isolation Forest Model] E -->|Normal| F[Keep Training! 🚀] E -->|Anomaly| G[Rest Day Required! 🛑] Prerequisites 🛠️ To follow along, you’ll need a few tools in your tech_stack : Python 3.9+ Scikit-learn : For our machine learning magic. SciPy/NumPy : For the heavy math lifting. Oura Cloud API Access : To get that sweet, sweet biometric data. pip install scikit-learn scipy pandas requests Step 1: Fetching R-R Intervals from Oura 💍 The Oura Ring records "R-R intervals" (the time between successive heartbeats in milliseconds) during sleep. This is much more granular than a simple "Heart Rate" average. import requests import pandas as pd def fetch_oura_hrv_data ( api_token , start_date , end_date ): url = f ' https://api.ouraring.com/v2/usercollection/heart_rate ' headers = { ' Authorization ' : f ' B

Every failure I am about to describe produced a clean run. No exception, no stack trace, no red build. Each one produced a plausible number that I believed for longer than I should have. That is the category of bug I have come to fear most. A crash tells you it crashed. A silently broken dataset tells you nothing at all, and your metrics will politely agree with it. Here are three from the last year, all from my own work, all found late. 1. The dataset that was 92% one category I had a training set of 688 records for a multi-category vision-language task. Thirteen categories. Reasonable size for a fine-tune, already used in a completed training run whose results I had written up. While preparing a stratified split, I joined the records back against the source annotations and actually counted the categories. 630 of 688 were a single category: scene captions. Zero examples of traffic signals. Zero of planning. Zero of uncertainty. Several categories the evaluation explicitly measured had no representation in training at all. The previous fine-tune had shown gains on some of those very categories. I had interpreted this as the model learning the task. The real explanation was duller and more useful: the model had learned the answer format from caption supervision, and format alignment alone was enough to move a multiple-choice score. Nothing category-specific had been learned, because nothing category-specific had been shown. The root cause was upstream and boring. The conversion script I inherited only rewrote file paths and dropped records with missing frames. It faithfully preserved a caption-only selection made further up the chain. It had no opinion about balance because nobody had asked it to have one. What I changed: the composition of a training set is now an artifact I generate and inspect before any run, not a property I assume. A category histogram takes seconds. I had not looked, for months. 2. The 18-hour run that converged perfectly to nothing Large model

Anthropic and OpenAI are racing to scale up while reducing dependence on Nvidia.

You've done it. You're finally done building the website or application you've been working on for quite a while. Proud and elated, you go to share this on your socials or to your buddies — uh oh, what's this now? The preview in WhatsApp shows no headline, your avatar is cropped and dimensions seem wrong. I've been there too. The site looked fine in the browser. The problem was everything outside the browser: link previews, search snippets, and tab icons all use a separate metadata layer most of us skip until something breaks. So, how do you fix this? Use this as a pre-launch checklist — or run it on a site that's already live but sharing badly. What I ran into on my own portfolio When I ran this audit on shwethaadiraj.com , the site rendered fine — but sharing it told a different story. I had pointed both the favicon and Open Graph image at my profile avatar. At tab size the illustration was unreadable; in link previews it got cropped awkwardly. An OG validator then flagged two things I hadn't considered: the image was 512×512 (most platforms expect 1200×630 ), and there was no headline or CTA on the image itself — so Slack and LinkedIn showed a plain square with none of the context from my meta title. I replaced the favicon with a simplified monogram, regenerated the OG image at the correct aspect ratio with my name, tagline, and site URL on it, and re-ran the debuggers. Even then, previews didn't update until I hit Scrape Again — platforms cache OG data aggressively, so fixes on your end won't show up until you bust that cache. None of this required rethinking the app. It was a metadata pass — the kind of work that's easy to defer and annoying to discover at the share button. Before we get into the specifics, here's a primer on what metadata can actually impact: What is metadata for? Metadata, simply put, is data about data. Search engines, crawlers and social sites all parse different metadata from your app. Search & Discovery: The title and description in your

In modern Data Engineering, there is an unspoken fetish for "Real-Time." If you ask any business stakeholder how fast they need their dashboard to update, the default answer will always be: "As fast as possible." This drives well-intentioned engineers to design incredibly complex architectures. We spin up Kafka clusters, implement Flink, and wrestle with latency, late-arriving data, and tumbling windows. All to have data flowing in milliseconds. But the harsh reality is that the vast majority of companies are building Ferraris just to sit in rush-hour traffic. 1. The Actionability Gap (The Golden Question) The biggest mistake when choosing a streaming architecture isn't technical; it's a business mistake. Before implementing real-time pipelines, the only question that matters is: "Does the company have the operational capacity to make a decision in milliseconds?" If you are building a credit card fraud detection system or a live e-commerce recommendation engine, yes, every millisecond counts. But if the data is feeding a financial dashboard that the executive board only reviews during their Monday morning meeting, updating that screen every second is a colossal waste of money and effort. Real-time data has zero value if the human action is batch. 2. The Hidden Complexity and the Cloud Bill Batch processing is forgiving. If a pipeline fails at 3 AM, you trigger a rerun, and by 8 AM, everything is fine. Batch is cheap, predictable, and easy to debug. Streaming, on the other hand, is unforgiving. Handling application state, event duplication (exactly-once semantics), out-of-order events, and sudden traffic spikes requires a senior engineering team dedicated solely to keeping the infrastructure alive. Furthermore, the cloud bill for 24/7 continuous processing is orders of magnitude higher than spinning up your compute clusters on a schedule. 3. "Micro-Batch" Solves 99% of Your Problems There is a perfect middle ground that the hype industry tries to ignore: the micro-ba

GitHub malware advisories no longer stop at npm. Here's how we wired OpenSSF's malicious-packages data into the Advisory Database, and why we built the pipeline paranoid. The post How we took malware advisories beyond npm appeared first on The GitHub Blog .

Connor Moucka pled guilty to hacking and stealing data from more than 165 Snowflake customers, which net him and his accomplices more than $2.5 million in ransom payments.

If your files already live in Amazon S3, the pitch for storage that branches with your database is appealing but the word "migration" makes it sound like a project. It mostly is not. Neon's object storage speaks the S3 API, so the code you already wrote, the AWS SDK calls and presigned URLs, keeps working. What changes is how you point the client and where the bucket comes from, and that is a small, mechanical diff. The actual data move is a copy loop you can run once. The one thing to do up front is confirm the object operations your app actually relies on: the demo here exercises PutObject , GetObject , listing, and presigned URLs, and I flag the S3 features you should check for yourself further down. This post is the practical version: what stays identical, the exact config that changes, a script to copy the objects across, and an honest list of the S3 features that do not have an equivalent so you know what to check before you commit. The repo with the working client is at the end. TL;DR Neon object storage is S3-compatible. Your @aws-sdk/client-s3 code for the common operations, PutObject , GetObject , getSignedUrl , listing, works unchanged (these are what the demo verifies). Confirm anything beyond that, like multipart for large objects, against the current preview. The diff is the client config: point endpoint at the Neon storage endpoint, pin region: 'us-east-2' , set forcePathStyle: true . The bucket is declared in neon.ts instead of created in the console, and credentials are injected per branch. Move the data with a list-and-copy loop between two S3 clients (source AWS, destination Neon). What does not carry over: S3 bucket policies, event notifications and Lambda triggers, storage classes and Glacier transitions, and cross-region replication. Object CRUD and presigning do. The payoff is everything else in this series: once the files are on Neon, they branch with your database. Prerequisites An existing S3 bucket and credentials that can read it A Neon p

If your app stores files and you want real preview environments, you eventually hit the same wall: each preview needs its own storage, so you start provisioning a bucket per environment. That sounds cheap until you write it down. For every ephemeral environment you create a bucket, attach a policy, mint an IAM role or access keys, set CORS, add a lifecycle rule so it eventually cleans up, wire the credentials into the preview's config, and register a teardown step for when the PR closes. Then you find the orphaned buckets the teardown missed, months later, still billing. The reason this is painful is that the bucket is a separate resource from the database, so it needs its own lifecycle. Neon collapses that: the bucket is declared as part of the branch, so it is created and destroyed with the branch and needs no per-environment provisioning at all. This post compares the two approaches and shows the branch version working with no bucket-management code in sight. The repo is at the end. TL;DR Isolated storage per preview usually means provisioning a bucket per environment: policy, IAM, CORS, lifecycle, credential wiring, teardown. It is slow, it drifts, and it leaves orphaned buckets that keep costing money. On Neon the bucket is declared once in neon.ts . Creating a branch brings the bucket (with a copy-on-write copy of the files) and injects scoped credentials; deleting the branch removes it. There is no per-environment bucket to create, no IAM role to mint, and nothing to orphan. Copy-on-write means fifty preview buckets do not cost fifty times the storage, only what each one changes. Prerequisites A Neon project on the platform preview (object storage, us-east-2 ) The Neon CLI, and a CI system that opens/closes preview environments Familiarity with S3 buckets and IAM if you have done the manual version The per-environment bucket, written out Here is what "just give the preview its own bucket" actually expands to, per environment: Create a bucket with a unique nam

The naive way to accept file uploads is to POST them to your API, let the server read the bytes, and write them to object storage. It works until the files get large or the traffic gets real. Now every upload crosses your infrastructure twice, once from the client to your server and once from your server to storage, and your server holds the whole file in memory or on disk while it does. On a serverless function it is worse, because functions have request-size and duration limits that a big upload runs straight into. Presigned URLs are the standard fix, and they predate serverless by a decade. Your server does not move the bytes; it hands the client a short-lived, pre-authorized URL and the client uploads directly to object storage. The server only issues permission and records metadata. On a Neon Function this is the same AWS S3 SDK you already use, pointed at the branch's storage endpoint. This post builds it and tests the whole round trip. The repo is at the end. TL;DR Proxying uploads through a function sends the bytes across it, burning bandwidth and memory and hitting request-size limits. A presigned URL is a time-limited, pre-authorized link to one object key. The client PUTs the bytes straight to storage; the function never touches them. On Neon Functions you generate it with getSignedUrl from @aws-sdk/s3-request-presigner , the same code as any S3-compatible store. I tested the full flow: presign, the client PUT straight to storage returned 200 , a metadata record was saved, and downloading the object returned the exact bytes. One gotcha to pin: the injected AWS_REGION is the storage-cell host, not a region, so set region: 'us-east-2' on the client. Prerequisites A Neon project on the platform preview with a declared bucket (object storage, us-east-2 ) The AWS SDK: @aws-sdk/client-s3 and @aws-sdk/s3-request-presigner Familiarity with S3-style object storage and HTTP PUT Why not just proxy the upload Sending the file through the function has three costs that

Chevron and Williams are big winners in the race to power artificial intelligence as they build out gas-fired power plants and pipelines.

Wiz Research disclosed CosmosEscape, a chain that escaped Azure Cosmos DB's Gremlin sandbox and reached a platform-wide key granting read and write access to every database on the service. Microsoft blocked the entry point within two days but took until July 2026 to remove the key. Practitioners debated shared responsibility and what that rearchitecture actually cost. By Steef-Jan Wiggers