标签：#data

We’ve all been there: staring at a stack of printed lab results or a folder full of cryptic report_final_v2_NEW.pdf files, trying to remember if our cholesterol was higher or lower two years ago. For developers, this isn't just a filing problem—it's a data engineering challenge. In the world of healthcare, data is messy, siloed, and often locked in "unstructured" formats. To build a truly personal Electronic Health Record (EHR) system, we need more than just a folder; we need a RAG (Retrieval-Augmented Generation) pipeline that can parse PDFs, map them to the FHIR (Fast Healthcare Interoperability Resources) standard, and provide natural language insights. In this guide, we’ll leverage Unstructured.io , Milvus , and DuckDB to turn chaotic medical PDFs into a queryable, structured knowledge base. The Architecture: From Raw Pixels to Structured Insights Before we dive into the code, let’s look at how the data flows from a messy lab report to a structured answer. graph TD A[Unstructured PDF Reports] --> B[Unstructured.io Partitioning] B --> C{Data Split} C -->|Textual Context| D[Milvus Vector DB] C -->|Tabular Data| E[DuckDB Structured Storage] D --> F[LangChain RAG Engine] E --> F G[User Query: Is my glucose trending up?] --> F F --> H[FHIR-Formatted Response] Why this stack? Unstructured.io : The gold standard for handling "ugly" PDFs (tables, headers, and nested lists). Milvus : A high-performance vector database built for scale. DuckDB : Perfect for running complex analytical SQL queries on the extracted "structured" parts of our medical data. FHIR Standard : To ensure our data follows global healthcare interoperability rules. Prerequisites Make sure you have your environment ready: pip install langchain milvus unstructured[pdf] duckdb openai Step 1: Extraction with Unstructured.io Medical PDFs often contain complex tables. Standard PDF parsers usually fail here. We’ll use unstructured to partition the document into logical elements. from unstructured.partition.pdf

Quick answer: The PromptEval Prompt Quality Report scored over 1,000 real prompts across 12 use cases. The average was 52 out of 100, and only 8% reached "good" (75+). The strongest single predictor of a good prompt is whether it defines its output format, worth 27 points on average. In 9 of 10 prompts, the weakest dimension was robustness. This is the PromptEval Prompt Quality Report . Over 1,000 real prompts have been scored on PromptEval , submitted by real users across use cases from customer support to healthcare to code. Each was scored from 0 to 100 on four structural dimensions: clarity, specificity, structure, and robustness. Every figure below comes from that set. No prompt text is stored; the analysis is anonymous and aggregate. Only 8% of the 1,000+ scored prompts reached "good" (75 or higher). Fewer than 1% reached "excellent." Source: PromptEval Prompt Quality Report, 2026 How the scores break down Here is how the scores spread across the set. The bar for "good" is 75, the point where a prompt is clear, specified, and holds up under variation. Score range Share of prompts 0 to 40 (failing) 25% 41 to 60 (below par) 31% 61 to 74 (functional but mediocre) 36% 75 to 84 (good) 8% 85 to 100 (excellent) under 1% Roughly 92% of prompts never reach "good," and almost none reach "excellent." This includes prompts from people who clearly know the tools. The gap is not talent. It is a few missing pieces that repeat. What separates a good prompt from a bad one For each structural element, we compared the average score of prompts that had it against those that did not. These are averages across the set, not a controlled experiment, so read them as correlation. But the gaps were large and consistent. The prompt... Avg with Avg without Point gap Defines the output format 58 31 +27 Has explicit constraints (what not to do) 63 41 +22 Assigns a role or persona 57 42 +15 Includes at least one example 64 51 +13 Prompts that define their output format score 27 points higher

Stale data và cache stampede: vì sao TTL một mình không đủ và vì sao origin sập khi key hết hạn Stale data là dữ liệu trong cache đã lỗi thời so với source of truth. Nó xuất hiện vì cache và origin là hai bản sao, và bất kỳ cơ chế đồng bộ nào — TTL, event-based invalidation, versioning — đều có cửa sổ giữa lúc origin đổi và lúc cache biết chuyện. Cái giá phải trả trong production không chỉ là "user nhìn thấy giá cũ vài giây". Khi một key hot vừa hết hạn, hàng nghìn request cùng miss, cùng đâm xuống DB để tính lại — đó là cache stampede (dogpile, thundering herd), và nó đủ sức đưa origin xuống trong vài chục giây. Cơ chế hoạt động Có bốn cơ chế invalidation dùng thật: TTL (time-to-live). Mỗi entry gắn một hạn dùng. Hết hạn coi như miss, đọc lại từ origin. Đơn giản, không cần coordination giữa writer và cache. Nhược điểm: staleness bounded bởi TTL, và tất cả replica của cùng một key hết hạn cùng lúc. Event-based invalidation. Khi origin thay đổi, phát một event (thường qua pub/sub, CDC như Debezium, hoặc gọi trực tiếp DEL) để cache xoá hoặc cập nhật entry. Fresh gần như realtime, nhưng đòi hỏi coupling giữa write path và cache — writer phải biết mọi key phái sinh từ dữ liệu vừa đổi. Versioning (cache key có version). Key gắn version của dữ liệu, ví dụ user:123:v42 . Đổi dữ liệu thì tăng version, key cũ tự nhiên bị bỏ qua, không cần xoá gì. Kỹ thuật này còn được gọi là generational caching; Rails cache dùng cách tương tự với cache_key_with_version . Single-flight (request coalescing). Không phải invalidation, mà là cách xử lý miss: khi N request cùng miss cùng một key, chỉ một trong số đó được phép gọi origin, các request còn lại đợi kết quả của nó. Go có golang.org/x/sync/singleflight implement sẵn pattern này; Facebook memcache dùng "leases" (paper của Nishtala et al., NSDI 2013) cho cùng ý tưởng ở scale phân tán. Kết hợp điển hình: TTL để bounded staleness, single-flight để chặn stampede khi key hết hạn, event-based invalidation để cắt TTL sớm khi có write. Ví dụ si

From a massive DOGE data breach and the hacking of critical energy and water systems to the hack of an FBI surveillance system, here are the most damaging security incidents and data breaches of 2026.

It's early, but the plan is to reduce dependency on Nvidia and Huawei.

SaaS software vendor HubSpot has described how its semantic search platform grew from a proof of concept into an internal service that now manages more than 20 billion vectors across 38-plus teams. The company says the system now supports agents, RAG, and contact deduplication, and that the increase in agent usage has made retrieval quality and latency more important than before. By Matt Saunders

I built a small Redis clone in C: a RESP parser, a command table, an append-only file for persistence. Recently I started building the same thing again in Rust, and rebuilding a project I had already finished has taught me more Rust than any from-scratch tutorial. The reason is simple. The second time, the design is already solved. I know what the AOF has to guarantee, what the command table dispatches, what the parser must reject. So none of my attention goes to what to build. All of it goes to how Rust wants it built. That turns the domain into a constant and the language into the only variable. Every difference I hit is pure signal about Rust, not noise about key-value stores. The first difference shows up before any logic runs. In C, I built the substrate first: my own dynamic strings, my own hashmap, my own linked list. Hundreds of lines before a single command worked. In Rust, Vec , String , and HashMap are just there, so that whole layer disappears and I start at the actual command logic. A standard library quietly decides where your project even begins. The sharper difference is in dispatch. In C it is a switch with argument counts I check by hand: if ( argc != 3 ) return err ( "wrong arg count" ); switch ( cmd ) { case CMD_SET : return do_set ( argv [ 1 ], argv [ 2 ]); case CMD_GET : return do_get ( argv [ 1 ]); /* forget a case and it is a runtime bug */ } In Rust the same dispatch is an enum and a match, and the compiler will not build until every case is handled: match cmd { Command :: Set { key , val } => self .set ( key , val ), Command :: Get { key } => self .get ( key ), } Same dispatch. One version cannot ship the missing-case bug I actually shipped in C. If you already know a project cold, rebuild it in the language you are learning. You stop thinking about the problem and start feeling the language.

InfoQ has opened enrollment for a five-week AI Security & Privacy Engineering cohort for senior engineers and architects in regulated industries, focused on applying security, privacy, threat modeling, observability, and governance practices to production AI systems. By Artenisa Chatziou

The Model Context Protocol team has promoted its Enterprise-Managed Authorisation extension to stable status, adding a centralised way for organisations to control access to MCP servers through their identity provider. The project states the aim is to replace per-server consent prompts with a zero-touch flow in which users sign in once and then access approved servers without further setup. By Matt Saunders

Plenty of SEO audits end with a single tool report. You run Lighthouse, screenshot Search Console coverage, save a "12 issues found" panel, and call it done. The trouble is that most audits finished that way silently revert within three months. Someone publishes a new post, refactors a component, swaps a font, and the issue quietly comes back. Nobody notices. Over the last five days I actually audited my four-language blog (ko/ja/en/zh, 298 posts per language). Five items, all fixed. But what I really want to talk about isn't what I fixed. It's that the five fixes mattered less than the build gates that keep them from ever returning. An audit should be a loop, not an event. Why a one-report audit always comes back Most technical SEO issues aren't "the code is wrong." They're "an invariant was never enforced anywhere." Take a clear rule: a published post must not link internally to a draft. Obvious enough. But if a human has to remember that every time, then the moment a recommendation generator pulls in one draft slug, a 404 is born. The report catches that 404 and shows it to you, but it does nothing to prevent the next one. So I ran the audit as a three-step loop. Measure. Fix the biggest item first. Then turn that item into a checker and nail it to the build . Skip the third step and the first two become a chore you repeat every six months. Once a gate is in place, the same class of problem makes npm run build fail. A pipeline enforces the rule, not human memory. This isn't a new invention. It's the same logic by which tests prevent bug regressions, applied to the content and markup layer. It's just oddly rare in SEO, where most teams leave "SEO checks" as a quarterly manual task. The five items I actually ran over five days Measurement first. Each item got a before/after in numbers, not a vibe that "things feel better" but reproducible figures. (The raw log of all five lives on the improvement history page too.) Date Item Before After Gate 07-02 relatedPosts int

In February, city officials in Cheyenne, Wyoming discovered something in their reclaimed water system that shouldn't have been there: Cupriavidus gilardii , a rare metal-resistant bacterium traced to wastewater discharges from Meta's $800 million data center campus. The contamination shut down Cheyenne's reuse water system for months , and on July 2, the city publicly named Meta's construction entity — a shell company called Goat Systems LLC — as the source. 📖 Read the full version with charts and embedded sources on ComputeLeap → "It's a very, very unpleasant surprise," said City Councilman Pete Laybourn. It shouldn't have been a surprise at all. Cheyenne is just the latest community learning what happens when AI's insatiable demand for compute meets the physical world: contaminated water, noise that residents describe as "living in hell," electricity bills that spike 267%, and — in the most surreal twist — a federal government that deleted its own energy conservation pages while a heatwave slammed the eastern seaboard. The AI industry talks endlessly about parameters, benchmarks, and scaling laws. But the story converging across Reddit, Hacker News, X, and YouTube this week isn't about models. It's about watts, gallons, and the communities living next to the machines. The water problem is worse than you think A Brookings Institution analysis puts the numbers in perspective: a typical data center consumes 300,000 gallons of water every day — equivalent to roughly 1,000 households. Large facilities gulp up to 5 million gallons daily, matching the needs of a town of 50,000. And water demand for data center cooling may rise by 870% as the current build-out continues. The scale is hard to overstate. According to a Consumer Reports investigation , Phoenix-area data centers currently use 385 million gallons annually — a figure projected to explode to 3.7 billion gallons once planned facilities come online. About two-thirds of data centers built since 2022 sit in water-st

A friend messaged me a photo of a sealed booster box last month with one question: "worth it?" He'd already decided, really. The chase card in that set was all over his feed, so the box felt like a good deal. I asked him to send me the pull rates instead of the hype, and we spent twenty minutes turning "worth it?" into something we could actually compute. That exercise is a small, self-contained data problem. It's also a good example of how a clean-looking model can hand you a confident number that doesn't survive contact with reality. If you like building little estimators, this one is worth doing carefully, because the interesting part isn't the formula. It's everything the formula assumes. The formula is the easy part Expected value of a box is a weighted sum. Each card you can pull has a probability and a market value, and you multiply the two across every slot the box gives you. That's it. Undergrad probability. Here's a stripped-down version for a hypothetical set. I'm using made-up numbers so nobody mistakes this for real pull data — the point is the shape of the computation, not the specific set. # One "hit slot" in a box: probabilities cover the full outcome space. # Values are illustrative market estimates in USD. hit_table = [ { " name " : " Alt-art chase " , " p " : 0.0125 , " value " : 180.00 }, { " name " : " Secret rare " , " p " : 0.030 , " value " : 45.00 }, { " name " : " Full-art rare " , " p " : 0.100 , " value " : 8.00 }, { " name " : " Standard hit " , " p " : 0.400 , " value " : 0.55 }, { " name " : " No notable hit " , " p " : 0.4575 , " value " : 0.06 }, ] assert abs ( sum ( row [ " p " ] for row in hit_table ) - 1.0 ) < 1e-9 ev_per_slot = sum ( row [ " p " ] * row [ " value " ] for row in hit_table ) hit_slots_per_box = 36 # e.g. one meaningful slot per pack ev_box = ev_per_slot * hit_slots_per_box print ( f " EV per slot: $ { ev_per_slot : . 2 f } " ) # $4.65 print ( f " EV per box: $ { ev_box : . 2 f } " ) # $167.31 The box costs $150 sea

PostgreSQL provides several planner configuration parameters, such as enable_seqscan and enable_indexscan , that influence how execution plans are generated. These settings affect planning, not the execution of an already-generated plan. With prepared statements, this raises an interesting question. Should planner settings be applied before PREPARE, before EXECUTE, or both? Let's look at a simple example: a "tasks" table with a due date and a "done" status: \ c drop table if exists tasks ; -- a table of tasks with status (done or not) and due date create table tasks ( id bigint generated always as identity primary key , due timestamptz , done boolean ); -- insert 500 tasks, with 1% not done insert into tasks ( due , done ) select now () + interval '1 day' * n , 42 != n % 100 from generate_series ( 1 , 500 ) n ; -- index the todo (partial index) create index on tasks ( due , id ) where done = false ; vacuum analyze tasks ; With a partial index, I indexed only the tasks that are not yet done ( done = false ) because that's my most frequent query pattern: postgres =# explain select id , due , done from tasks where done = false and id > 0 order by due limit 1 ; QUERY PLAN --------------------------------------------------------------------------------------- Limit ( cost = 0 . 13 .. 3 . 60 rows = 1 width = 17 ) -> Index Scan using tasks_due_id_idx1 on tasks ( cost = 0 . 13 .. 17 . 47 rows = 5 width = 17 ) Index Cond : ( id > 0 ) ( 3 rows ) With partial indexes, the condition covered by the index is not even visible in the execution plan because the index itself enforces the condition. Prepared statement I decided to use a prepared statement with all values as parameters. It is probably not a good idea in this case. When a parameter can have only a few different values and you expect different cardinalities for each, you should probably define one query per value, using literals. I'm doing this to illustrate what can happen, with a simple, extreme example: postgres =# pr

AWS recently announced Amazon S3 Annotations, a feature that lets teams attach rich, searchable context such as summaries, classifications, compliance data, or AI-generated insights directly to S3 objects. Annotations can be updated independently of the object and queried across datasets, reducing the need for separate metadata systems. By Renato Losio

Claude models reached GA on Microsoft Foundry with Azure-native billing and governance, but no European data zone exists. Anthropic's own documentation confirms data residency guarantees apply to Bedrock and Vertex AI but not Foundry. European practitioners from banking and healthcare report the offering is unapproved for production. By Steef-Jan Wiggers

The problem with a data breach is not finding evidence. It is connecting it. But let me start where I actually was: 4 AM, last day of the hackathon, staring at this in my terminal. RateLimitError: GroqException - Rate limit reached for model `llama-3.3-70b-versatile` on tokens per day (TPD): Limit 100000, Used 99787, Requested 1616. Please try again in 20m12s. Used 99,787 out of 100,000. My deployment was half done, my demo graph was empty on the server, and the free tier had 213 tokens left. The submission deadline was hours away. I had not slept. I had not eaten. My friends were asleep and I was swapping API keys like a gambler swapping chips. This post is the story of how we got there, and how it ended at 7 in the morning with the best sigh of relief I have ever taken. First, some honesty about how I got here When I joined my first WeMakeDevs hackathon, I did not believe in it. I thought it was one of those ordinary online events. Fake prizes, no follow-through, what would I even get out of it. I joined anyway, mostly out of boredom, got into the Discord, talked to people, made a few connections. I landed in the top 50. A few days later an email showed up: a free Claude Max subscription as a gift. I read it twice. I genuinely could not believe a hackathon had actually delivered something. So when this hackathon opened, I did not hesitate. I messaged my friends and said we are joining as a team this time. Three of us: me (Mehraan), Aqib, and Ubaid. The spark We spent the first evening in our group chat throwing ideas around and shooting most of them down. Then one of my friends dropped a thought that stuck: what happens after a company gets hacked? I started digging into it. The answer is honestly depressing. After a breach, the evidence is everywhere. VPN records. File access logs. The email gateway. Badge readers at the office doors. CCTV. HR notes. Anonymous tips. Each system tells one small piece of the story, and a human analyst has to stitch all of it togeth

Every big social network locks audience data behind auth walls and anti-bot systems. Bluesky went the other way. The AT Protocol is open by design, so public profile data (bios, follower counts, full follower and following lists) is queryable through a documented API without logging in. The whole surface is basically two endpoints: GET https://api.bsky.app/xrpc/app.bsky.actor.getProfile?actor=HANDLE GET https://api.bsky.app/xrpc/app.bsky.graph.getFollowers?actor=HANDLE&limit=100 There's also getProfiles for batching 25 handles per call. Follower lists paginate with a normal cursor , which still works on the graph endpoints. Search is a different story, cursor pagination 403s there now, but that's a topic for another post. For one-off lookups, curl is honestly all you need. Where it gets tedious Bulk. Thousands of profiles, follower exports that run into six figures, weekly snapshots for tracking. Pagination, rate-limit backoff, and stitching the pages together is boring code that has to run reliably. I packaged that part as an Apify actor: Bluesky Profile Scraper . Paste handles or profile URLs, optionally turn on follower/following export, and you get JSON or CSV back with a sourceProfile field linking each follower record to the profile it belongs to. $2 per 1,000 records, runs on a schedule if you want snapshots over time. What people use this for Vetting an influencer's real audience before paying them. Exporting who follows a competitor and what their bios say. Charting follower growth from weekly runs. And enrichment: find who's talking about you with a mentions monitor , then profile those authors to see their actual reach. Bluesky is the only major network right now where any of this is straightforward and stable. Worth using while it lasts.

SQLite Internals, Postgres 19 Checksums, & PL/CBMBASIC Extension Today's Highlights This week, we delve into SQLite's secure deletion and blob updates, explore upcoming data integrity features in PostgreSQL 19, and discover a unique PostgreSQL extension bringing Commodore 64 BASIC to your database. These updates offer insights into database internals, future resilience, and creative extensibility for the SQLite ecosystem. Secure Delete and BLOB Updates in SQLite (SQLite Forum) Source: https://sqlite.org/forum/info/6f3e886a1149c97e0ede9a243281efb05a043705393ea94437ed7c0556315972 This SQLite forum discussion delves into the nuances of secure data deletion and efficient BLOB updates within SQLite databases. Secure deletion is a critical concern for applications handling sensitive data, where simply deleting a row might not zero-out the underlying storage, leaving recoverable remnants. The thread explores methods and implications for ensuring data is truly eradicated when removed, potentially touching on PRAGMA settings or specific file system interactions. Understanding these mechanisms is crucial for developers building secure, embedded applications with SQLite. The conversation also extends to optimizing updates for BLOB (Binary Large Object) data. Efficiently handling large binary data, such as images or documents, in an embedded database like SQLite requires careful consideration to avoid performance bottlenecks and excessive disk I/O. The discussion likely covers strategies for in-place updates, managing free space, and the internal workings of SQLite's storage engine when dealing with variable-length BLOBs. This insight helps developers make informed decisions on schema design and update patterns for improved application performance and data integrity. Comment: This thread offers valuable insights into SQLite's low-level data management, essential for anyone needing to implement robust security or optimize BLOB storage. PostgreSQL 19 to Feature Checksums For All

Introduction I used to think "connecting to a database" was one skill. Turns out it's two: connecting to a database chilling quietly on your own laptop, and connecting to one living in the cloud, behind a login, in this case, an SSL certificate that will not let you in until you treat it with respect. This week I did both. Same tool (Power BI), same dataset, two very different vibes. Grab a coffee, here's the full walkthrough local PostgreSQL first, then Aiven's cloud version, side by side, screenshots and all. Part 1: Local PostgreSQL → Power BI Step 1 : Create a schema Nothing fancy, just giving my table a home: CREATE SCHEMA powerbi ; Step 2 : Import the dataset Right-click the new schema → Import Data in DBeaver, point it at your CSV, and let the wizard do its thing. Step 3 : Check the table landed properly A quick peek at the columns to make sure nothing got mangled on the way in. Step 4 : Connect Power BI In Power BI Desktop: Get Data → Database → PostgreSQL database. In the Server field, type localhost (or 127.0.0.1 ) and your database name. localhost Choose Import , hit OK, and log in with your local username and password. Click Load . That's it. That's the whole local experience. Part 2: Aiven PostgreSQL (Cloud) → Power BI Now for the part that actually taught me something. Step 1 : Grab your connection details Everything you need lives on Aiven's Overview page: Host, Port, Database name, User, SSL mode. Your service URI will look something like this (don't worry, this isn't a real password, Aiven masks it in the console): postgres : // avnadmin : •••••••• @ pg - xxxxxxxx - yourproject . c . aivencloud . com : 22016 / defaultdb ? sslmode = require Step 2 : Import the dataset into Aiven Same DBeaver wizard as before, just pointed at the Aiven connection instead of local. CREATE SCHEMA powerbi ; Step 3 : Aiven's certificate. Download the CA cert from the Overview page: Now here's the part that actually tripped me up: Power BI's PostgreSQL connector doesn't ha

Why Your SOE Resume Needs a Structural Overhaul Chinese state-owned enterprises (SOEs) often have deep hierarchical structures and a culture of collective achievement. But Western tech companies want to see individual impact, autonomy, and data-driven results. Continuing to lead with your former employer's prestige or your rank (e.g., "Senior Engineer Grade 7") wastes valuable space. The solution: reshape every section to answer the question "What did you personally accomplish with data?" The Core Shift: From Hierarchy to Impact In a Chinese SOE resume, it's tempting to list departments you led or teams you oversaw. In a Western senior data scientist resume, focus on the problems you defined, the algorithms you deployed, and the revenue, cost savings, or user metrics that improved. For example, instead of "Led the data analytics team of 10 people," write "Designed and deployed a demand-forecasting model that reduced inventory costs by 15% (¥12M annually)." Three Resume Sections That Require Full Rewriting Professional Summary: From 'Accomplished Engineer' to 'Data Science Leader' Start with your total years of experience, your technical stack, and the types of business problems you solve. Example: "Senior Data Scientist with 10+ years applying machine learning to supply chain and logistics. Expertise in Python, TensorFlow, and Spark. Reduced operational costs by 15-30% through predictive models deployed at [SOE name]." Work Experience: From Role Descriptions to Metric-Driven Bullets For each role, list 3-5 bullets. Every bullet should have a verb, a task, a technology (if relevant), and a quantified result. Avoid vague phrases like "responsible for." Use specific numbers: "Improved forecast accuracy from 70% to 85% by building an ensemble of ARIMA and XGBoost models." Education & Certifications: Emphasize Transferable Skills Your Chinese degree is fine, but add relevant certifications (AWS, TensorFlow, Coursera) to show adaptability. Consider a "Technical Skills" se