标签：#Storage

Security researchers at Wiz recently examined S3-compatible object storage services across six popular neoclouds, revealing significant security gaps compared to Amazon S3. While S3 has become the de facto standard for object storage, most services lack several of AWS's security protections. By Renato Losio

Companies including Tesla and Base Power are vying for a piece of the rapidly growing market for home batteries. One technology has made it all possible.

This English version is an AI translation of my original article on Qiita (in Japanese) . Background I've been experimenting with Zenoh via its Elixir bindings, Zenohex , not for its usual pub/sub use case but for its put / get storage feature. It mostly worked, except every so the state I picked back up was one step behind. Digging into why turned into a fun rabbit hole, so here's the writeup. Reproducing it To keep things simple, strip out the GenServer part entirely and just loop put immediately followed by get on the same key: { :ok , session_id } = Zenohex . Session . open ( config ) Enum . each ( 1 .. 2000 , fn i -> payload = Integer . to_string ( i ) :ok = Zenohex . Session . put ( session_id , key , payload ) { :ok , replies } = Zenohex . Session . get ( session_id , key , 3_000 , consolidation: :latest ) case Enum . find ( replies , & match? (% Zenohex . Sample {}, &1 )) do % Zenohex . Sample { payload: ^ payload } -> :ok % Zenohex . Sample { payload: other } -> IO . puts ( "stale! put #{ payload } but got #{ other } " ) nil -> IO . puts ( "no reply at all" ) end end ) Out of 2000 iterations, a small fraction print stale! — about 78 (3.9%) in one run. The interesting part: querying again immediately afterward almost always returns the correct value (the fastest I measured was a single extra get about 1ms later). So it's not that the value disappears — there's just a small window of lag before the write is actually visible. Why Zenohex.Session.put/4 is a thin Rustler wrapper around zenoh-rust's put . Looking at the NIF implementation : fn session_put ( ... ) -> rustler :: NifResult < rustler :: Atom > { ... publication_builder .apply_opts ( opts ) ? .wait () // <- only waits for the local publish to be queued ... Ok ( rustler :: types :: atom :: ok ()) } .wait() only waits for the local session to finish handing the message off — not for the remote side (the zenohd router backing the storage) to actually receive and apply it. session_get , on the other hand,

"We don't have access to the data on the hardware/servers," Iron Mountain told Ars.

Form Energy has landed Google and Crusoe as customers. Now, it has raised $750 million to expand manufacturing to deliver its massive, 100-hour batteries.

Base Power’s $1 billion round will help the startup ramp production of its home batteries.

If you just want the recommendation: for the ordinary AI-generated image an inference job hands back — a 2 to 8 MB PNG — do one plain object PUT into your S3-compatible storage and stop there, because multipart upload only earns its complexity when a single artifact is big enough that losing a transfer halfway through costs you real money to redo, which for my team starts somewhere north of 100 MB. Everything below is about that threshold, and about the operations bill you pick up the moment you cross it. I run the platform roadmap for a team that renders a few hundred thousand images a month, and I count pages before I count features, so read the rest with that bias in mind. Should I use multipart upload for large AI-generated images, or a single object PUT? Multipart solves two narrow problems: a payload too awkward for one HTTP round trip, and a transfer you refuse to restart from byte zero. A 6 MB PNG has neither problem. The shape of the flow is always the same wherever you run it. You start a multipart upload and get back an upload id, you push each part under that id, you collect the returned ETag and part number for every one of them, and you send the finished list back in a complete call that stitches the object together server-side. Parts have to be at least 5 MiB on Amazon S3 and on every S3-compatible store I've tested against, with the final part exempt, which already tells you the feature was designed for objects measured in hundreds of megabytes rather than for a batch of thumbnails. Where it genuinely pays off in an image pipeline is the long tail: a 4-gigapixel tiled upscale, a nightly ZIP export of a customer's whole render history, a raw latent archive somebody in research wants kept for a year. Those are the jobs where a dropped connection at 80% is a real incident and not a shrug. For everything else, one put is one line of code and one thing to monitor. There's a second cost that people underrate, and it's the one I'd argue about in a design re

Use one key prefix per user, delete from the application every key you can name, and leave lifecycle rules to sweep the old temporary images nobody will ever ask for again. That's the whole design, and I've watched teams get it wrong in the same two ways for years: they either try to make the storage layer clever enough to know what a user is, or they hand the entire deletion problem to a lifecycle policy and then wonder why an account-deletion request took nine days to actually remove anything. I design data layers for a living, so I'm going to be blunt about the durability and consistency side of this rather than the upload-a-file-in-five-minutes side. Why the key layout matters more than the backend you pick Object storage has no folders. There's a flat keyspace and a delimiter convention, and every "folder" you see in a console is the UI grouping keys that share a prefix — which is good news, because it means the layout is yours to design and costs nothing to enforce. The layout I keep landing on is users/{userId}/generations/{yyyy-mm}/{uuid}.png , with a sibling users/{userId}/scratch/ prefix for renders that only exist so the browser can show a preview. Four properties come out of that shape, and they're the reason I don't get creative here. Listing a tenant's images is a single prefix query rather than a metadata scan, which matters because object stores generally don't let you search metadata server-side — you filter by prefix or you keep an index in your own database. Deleting an account becomes "enumerate one prefix, delete what's under it," so the compliance clock is something you control. The month segment keeps any single listing page from growing without bound, and it gives you a cheap way to write an age-based rule later. And the opaque UUID means the key never leaks a filename, a prompt, or an email address into a URL that might end up in a log or a referrer header. One thing I'd push back on if I saw it in review: don't put the user's email or usern

Use a private bucket with short-lived presigned URLs when an avatar belongs to exactly one user, and reach for a public CDN-backed bucket only when the images are genuinely public and you'd rather pay for cache hits than for signatures. For a Node.js SaaS that is the entire decision, and everything after it is plumbing: which S3-compatible provider you point at, how long a signature should live, and what happens to the stored object on the day a user deletes their account. Avatars are small. That removes half the hard problems. The half that's left is the half I get paged for, because an avatar key is written by an untrusted client, read on nearly every page render, cached in three places you don't control, and referenced from a database row that has its own opinion about which object is current. So the questions I ask a storage vendor aren't about upload throughput. They're about whether a partial write can ever be visible to a reader, what the durability number is actually measuring, and how I reconcile the bucket with my user table after a failed deploy. I've never watched a team lose avatar bytes. I've watched several lose track of which bytes were current, which is the same outage with a friendlier root-cause section. How should a Node.js SaaS store private user avatars in object storage? Three moves, in this order. Create one private bucket for the whole tenant base, write each avatar under a key that carries a random component, and mint a presigned GET at display time instead of persisting any URL. Store the key in your database, on the user row, and nothing else, because keys are stable and signatures expire — a URL you saved last Tuesday is a support ticket waiting to happen. Serving the image then costs you one signing call per render, which you can cache in Redis for slightly less than the signature's own lifetime. That random component does more work than it looks like it does. Overwriting a fixed path like users/8821/avatar.png puts you in a read-modify

Google's biggest solar and battery project stands in sharp contrast with xAI's nearby unpermitted power plant.

Need an ultrafast drive for video editing or a rugged option to back up your photos in the field? We’ve got a solution for every situation.

Azure Storage SFTP is Microsoft's managed file transfer service on top of Azure Blob Storage, convenient, but billed continuously per endpoint (roughly $0.30/hour, ~$220/month) and tied to Azure AD. SFTPGo is an open-source file transfer server offering SFTP, FTP/S, and WebDAV with pluggable storage backends (local disk, Azure Blob, S3-compatible, GCS) and no per-endpoint charge. This guide deploys SFTPGo with Docker Compose and Traefik, sets up user auth (password + SSH key + 2FA), connects S3-compatible object storage, and covers the migration path from Azure Storage SFTP. By the end, you'll have a self-hosted file transfer server with the same capabilities at zero endpoint cost. Azure Storage SFTP → SFTPGo Mapping Azure Storage SFTP SFTPGo Equivalent Notes SFTP Endpoint SFTPGo SFTP Server Configurable port, default 2022 Azure Blob Storage Azure Blob backend Native support; point at the same container, no migration needed Azure AD Authentication LDAP/OIDC plugin External identity provider via plugin Local Users Web UI / REST API user management Hierarchical Namespace Virtual directories No HNS requirement Azure Monitor Built-in logging + webhooks/syslog Prerequisite: Linux server with Docker + Compose, a DNS A record for your domain, and (if migrating) an existing Azure Storage account with SFTP enabled plus the Azure CLI installed locally. Deploy with Docker Compose 1. Create the project directories: $ mkdir -p ~/sftpgo/ { data,config } $ cd ~/sftpgo 2. Create the environment file: $ nano .env DOMAIN = sftp.example.com LETSENCRYPT_EMAIL = admin@example.com 3. Create the Compose manifest: $ nano docker-compose.yml services : traefik : image : traefik:v3.6 container_name : traefik command : - " --providers.docker=true" - " --providers.docker.exposedbydefault=false" - " --entrypoints.web.address=:80" - " --entrypoints.websecure.address=:443" - " --entrypoints.web.http.redirections.entrypoint.to=websecure" - " --certificatesresolvers.letsencrypt.acme.httpchallenge=tr

WAL: vì sao Postgres bắt buộc ghi log trước data file, và lý do pg_wal/ đầy đĩa làm cluster ngừng nhận write WAL (Write-Ahead Log) là cơ chế durability lõi của Postgres: mọi thay đổi đối với heap, index, free-space map, visibility map đều phải được ghi xuống WAL và fsync trước khi data file tương ứng được phép flush ra đĩa . Nguyên tắc này, mô tả trong Postgres docs chương "Reliability and the Write-Ahead Log", là cái cho phép một transaction đã COMMIT thoả ACID-D dù OS crash hoặc mất điện ngay sau đó. Dev gặp WAL trong việc thật không phải vì cú pháp khó: gặp khi pg_wal/ đầy đĩa do một replication slot bị quên dọn, Postgres dừng nhận write với PANIC: could not write to file ... No space left on device , hoặc khi crash recovery sau OOM kéo mười mấy phút làm health check fail và load balancer cắt traffic. Cơ chế hoạt động Postgres không ghi thẳng vào data file mỗi khi có INSERT / UPDATE . Trang 8KB (heap page, index page) sống trong shared_buffers ; mỗi thay đổi tạo ra một WAL record mô tả delta đó (record type, relfilenode, block number, payload), append vào wal_buffers — một vùng shared memory nhỏ trước khi xuống đĩa. Tại thời điểm COMMIT , backend gọi XLogFlush() để write + fsync WAL tới hết byte chứa commit record; chỉ sau khi fsync trả về, Postgres mới ghi commit bit vào pg_xact và reply OK về client. Data page bẩn ở lại trong shared_buffers ; checkpointer sẽ flush chúng ra data file sau, không gắn với từng commit. WAL được tổ chức thành segment file kích thước cố định trong $PGDATA/pg_wal/ , mặc định 16MB mỗi segment (cấu hình lúc initdb --wal-segsize ). Vị trí trong WAL là LSN (Log Sequence Number) — số 64-bit, in dạng XXXX/XXXXXXXX , thực chất là byte offset từ đầu WAL của cluster. LSN tăng đơn điệu và là "đồng hồ" duy nhất Postgres tin cậy cho thứ tự ghi. -- Quan sát LSN tiến lên sau mỗi ghi SELECT pg_current_wal_lsn (); -- vd: 0/1A2B3C40 INSERT INTO t SELECT g FROM generate_series ( 1 , 1000 ) g ; SELECT pg_current_wal_lsn (); -- 0/1A2BE018 SELECT pg_wal_ls

Honda wants in on the lucrative energy storage market. This week it began producing batteries destined for data centers, not driveways.

Base Power is skipping the PJM's troubled interconnection queue by placing its batteries at people's homes, offering backup services in exchange.

SeaweedFS is an open-source, distributed object storage system with an S3-compatible API, a filer for POSIX-style hierarchical access, and a small footprint. This guide deploys SeaweedFS using Docker Compose with the master, volume, filer, S3, and admin services behind Traefik for automatic HTTPS on separate admin and S3 domains. By the end, you'll have SeaweedFS serving S3-compatible object storage securely at your domains. Prerequisite: Two DNS A records pointing at the server — storage.example.com (admin dashboard) and s3.storage.example.com (S3 API). AWS CLI installed on your local machine for testing. Set Up the Directory Structure 1. Create the project directory: $ mkdir seaweedfs && cd seaweedfs 2. Generate an access key and a secret key (run twice and save both): $ openssl rand -hex 16 3. Create the environment file: $ nano .env STORAGE_DOMAIN = storage.example.com LETSENCRYPT_EMAIL = your-email@example.com ADMIN_PASSWORD = yourpassword 4. Create the S3 identities file: $ nano s3-config.json { "identities" : [ { "name" : "admin" , "credentials" : [ { "accessKey" : "YOUR_ACCESS_KEY" , "secretKey" : "YOUR_SECRET_KEY" } ], "actions" : [ "Admin" , "Read" , "Write" , "List" , "Tagging" ] } ] } Deploy with Docker Compose 1. Create the Compose manifest: $ nano docker-compose.yml services : traefik : image : traefik:v3.7.0 container_name : traefik restart : unless-stopped ports : - " 80:80" - " 443:443" volumes : - /var/run/docker.sock:/var/run/docker.sock:ro - ./letsencrypt:/letsencrypt command : - --providers.docker=true - --providers.docker.exposedByDefault=false - --entrypoints.web.address=:80 - --entrypoints.websecure.address=:443 - --entrypoints.web.http.redirections.entrypoint.to=websecure - --entrypoints.web.http.redirections.entrypoint.scheme=https - --entrypoints.web.http.redirections.entrypoint.permanent=true - --certificatesresolvers.letsencrypt.acme.email=${LETSENCRYPT_EMAIL} - --certificatesresolvers.letsencrypt.acme.storage=/letsencrypt/acme.json - --

Imagine a user trying to upload a 100MB video or a high-resolution photo to your app. If you use the standard Rails file upload, that file travels from the user's browser to your Rails server, and then your server sends it to S3 or Google Cloud. This is a terrible way to do it. While that 100MB file is transferring, your Rails worker (Puma) is frozen. It can't handle other users. If three people upload large files at once, your whole app will stop responding. In 2026, the professional way to handle this is Direct Uploads . With Direct Uploads, the file goes directly from the user's browser to your cloud storage (S3, R2, etc.). Your Rails server only handles a tiny bit of metadata. It is faster for the user and much safer for your server. Here is how to set it up in Rails 8. STEP 1: Configure Your Storage First, make sure you aren't using the local disk for production. You need a cloud provider like AWS S3 or Cloudflare R2. In your config/storage.yml : amazon : service : S3 access_key_id : <%= ENV['AWS_ACCESS_KEY_ID'] %> secret_access_key : <%= ENV['AWS_SECRET_ACCESS_KEY'] %> region : us-east-1 bucket : my-app-uploads # Crucial for Direct Uploads! public : true Note: You must configure CORS in your S3/R2 dashboard to allow requests from your domain. If you don't do this, the browser will block the upload. STEP 2: The Rails Form Rails makes the backend part incredibly easy. You just add one attribute to your file field: direct_upload: true . <!-- app/views/users/_form.html.erb --> <%= form_with ( model: user ) do | f | %> <div class= "field" > <%= f . label :avatar %> <%= f . file_field :avatar , direct_upload: true %> </div> <%= f . submit "Save Profile" %> <% end %> When you add direct_upload: true , Rails automatically includes a JavaScript library that handles the "handshake" with S3. STEP 3: Adding a Progress Bar (The UX Win) Direct uploads can take a few seconds. If nothing happens on the screen, the user will think your app is broken. We can use the built-in Ac

Electricity demand from AI data centers is pushing everyone — including automakers like GM and Ford — into the energy storage business.

Electricity demand from AI data centers is pushing everyone — including automakers like GM and Ford — into the energy storage business.

Ambrosia Energy wants to build power plants in less than 12 months while undercutting natural gas. It hopes to build gigawatts worth by 2030.