标签：#s3

You self-host S3-compatible storage on bare metal by installing a single Rust binary on a Linux server and pointing any S3 client at it. RustFS installs with one script, listens on port 9000 (S3 API) and 9001 (console), and is Apache 2.0 licensed. Single-node mode is production-ready today; multi-node clustering is still under testing. Every command below is copied verbatim from the official source cited beside it. This sandbox has no Docker daemon, so none of the commands were executed here; they are marked accordingly. Key Stats Fact Source RustFS installs with one command and runs as a systemd service on x86_64 or aarch64 Linux RustFS docs (Linux quick-start) Default S3 API port is 9000; console port is 9001 RustFS GitHub README Default credentials are rustfsadmin / rustfsadmin and must be changed RustFS README + docs RustFS is Apache 2.0 licensed and S3-compatible RustFS GitHub README Single-node mode is production-ready; distributed mode is still under testing RustFS README Feature & Status What is self-hosted S3-compatible storage? A self-hosted S3-compatible storage server is a program you run on your own hardware that speaks the Amazon S3 API. Applications using AWS SDKs, the aws CLI, or MinIO's mc can talk to it without code changes, because the bucket, object, and credential model matches S3. The difference from a cloud bucket is ownership: the disks, the network path, and the uptime are yours. RustFS is one such server, written in Rust and licensed under Apache 2.0. It exposes the S3 API on port 9000 and a web console on 9001, and it stores objects on the local filesystem. Because it is S3-compatible, the same client code that targets AWS S3 also targets a RustFS node. That compatibility is the whole point of self-hosting here: you get an S3 endpoint without renting one. Why run object storage on bare metal? Running object storage on bare metal means installing the server directly on a Linux machine instead of in a container or a managed cloud. The appeal

Security researchers at Wiz recently examined S3-compatible object storage services across six popular neoclouds, revealing significant security gaps compared to Amazon S3. While S3 has become the de facto standard for object storage, most services lack several of AWS's security protections. By Renato Losio

Canva redesigned session revocation infrastructure to support 100M active sessions while reducing database lookups. The architecture uses Amazon S3 for durable revocation records and distributes compact, in-memory indexes to application gateways. Canva said the design improved deployment speed, reduced database infrastructure requirements, and cut the revocation cache memory footprint by 87.5%. By Leela Kumili

If you are staring at An error occurred (AccessDenied) when calling the GetObject operation: Access Denied , this guide walks through every cause in the order you should check them, with the exact fix for each. I am a cloud associate and I debug this error often enough that I keep a mental checklist. Here it is, written down. Quick answer: the 8 most common causes of S3 Access Denied In Amazon S3, "Access Denied" means the request was authenticated but not authorized, or an explicit deny blocked it. In practice it is almost always one of these, roughly in order of frequency: The IAM identity (user or role) is missing the required s3: permission. The bucket policy does not allow the action, or explicitly denies it. S3 Block Public Access is on and you expected public/anonymous access. SSE-KMS : you have S3 permission but not kms:Decrypt on the encryption key. Missing s3:ListBucket , which turns a "key not found" into a 403. Cross-account access where only one side grants permission. An explicit Deny somewhere wins (SCP, permissions boundary, VPC endpoint policy, or bucket policy). Object ownership / ACLs after a cross-account upload. If you only remember one thing: an explicit Deny anywhere in the chain always beats an Allow . Start by finding a deny, then work down the list. Step 0: Confirm which identity is actually making the request Before touching any policy, confirm who you are. Most "but I have admin" cases are the wrong principal. aws sts get-caller-identity Check the Arn in the output. If it is a role you did not expect (an EC2 instance profile, a CI role, an assumed role), you have been debugging the wrong identity's permissions the whole time. This single command saves more time than any other step. Step 1: Does the IAM identity policy allow the action? S3 needs the specific action for the specific resource. The two resource types trip people up: Bucket-level actions ( s3:ListBucket , s3:GetBucketLocation ) target the bucket ARN: arn:aws:s3:::my-bucket Obj

The naive way to accept file uploads is to POST them to your API, let the server read the bytes, and write them to object storage. It works until the files get large or the traffic gets real. Now every upload crosses your infrastructure twice, once from the client to your server and once from your server to storage, and your server holds the whole file in memory or on disk while it does. On a serverless function it is worse, because functions have request-size and duration limits that a big upload runs straight into. Presigned URLs are the standard fix, and they predate serverless by a decade. Your server does not move the bytes; it hands the client a short-lived, pre-authorized URL and the client uploads directly to object storage. The server only issues permission and records metadata. On a Neon Function this is the same AWS S3 SDK you already use, pointed at the branch's storage endpoint. This post builds it and tests the whole round trip. The repo is at the end. TL;DR Proxying uploads through a function sends the bytes across it, burning bandwidth and memory and hitting request-size limits. A presigned URL is a time-limited, pre-authorized link to one object key. The client PUTs the bytes straight to storage; the function never touches them. On Neon Functions you generate it with getSignedUrl from @aws-sdk/s3-request-presigner , the same code as any S3-compatible store. I tested the full flow: presign, the client PUT straight to storage returned 200 , a metadata record was saved, and downloading the object returned the exact bytes. One gotcha to pin: the injected AWS_REGION is the storage-cell host, not a region, so set region: 'us-east-2' on the client. Prerequisites A Neon project on the platform preview with a declared bucket (object storage, us-east-2 ) The AWS SDK: @aws-sdk/client-s3 and @aws-sdk/s3-request-presigner Familiarity with S3-style object storage and HTTP PUT Why not just proxy the upload Sending the file through the function has three costs that

If you just want the recommendation: for the ordinary AI-generated image an inference job hands back — a 2 to 8 MB PNG — do one plain object PUT into your S3-compatible storage and stop there, because multipart upload only earns its complexity when a single artifact is big enough that losing a transfer halfway through costs you real money to redo, which for my team starts somewhere north of 100 MB. Everything below is about that threshold, and about the operations bill you pick up the moment you cross it. I run the platform roadmap for a team that renders a few hundred thousand images a month, and I count pages before I count features, so read the rest with that bias in mind. Should I use multipart upload for large AI-generated images, or a single object PUT? Multipart solves two narrow problems: a payload too awkward for one HTTP round trip, and a transfer you refuse to restart from byte zero. A 6 MB PNG has neither problem. The shape of the flow is always the same wherever you run it. You start a multipart upload and get back an upload id, you push each part under that id, you collect the returned ETag and part number for every one of them, and you send the finished list back in a complete call that stitches the object together server-side. Parts have to be at least 5 MiB on Amazon S3 and on every S3-compatible store I've tested against, with the final part exempt, which already tells you the feature was designed for objects measured in hundreds of megabytes rather than for a batch of thumbnails. Where it genuinely pays off in an image pipeline is the long tail: a 4-gigapixel tiled upscale, a nightly ZIP export of a customer's whole render history, a raw latent archive somebody in research wants kept for a year. Those are the jobs where a dropped connection at 80% is a real incident and not a shrug. For everything else, one put is one line of code and one thing to monitor. There's a second cost that people underrate, and it's the one I'd argue about in a design re

AWS Lambda can now reference deployment packages directly in customer-owned S3 buckets, removing the per-Region code storage quota and raising the managed default from 75 GB to 300 GB. Per-function package limits are unchanged, and UpdateFunctionCode is still required after replacing an object. Terraform provider support remains an open enhancement request. By Steef-Jan Wiggers

Originally published at blog.whynext.app . It started as an ordinary cleanup problem. Users upload media files (recordings and images) through presigned URLs. The server issues an upload URL, the client uploads straight to S3, then calls a commit API to say "register this key as a real asset." The problem is what happens when someone gets a presign but never commits. The app crashes, the network drops, the user leaves the screen, and the bucket is left with an object that isn't registered anywhere. I wanted a lifecycle rule to clean these up, but there was no way to write one. Committed and uncommitted objects were mixed under the same prefix, so any rule that says "delete old things" would delete real assets too. A daily upload quota kept the pile from growing fast, but the fact remained: there was no path to reclaim the space. The design: what isn't committed lives in tmp The backbone of the fix is key namespace separation. presign issues a temporary key under the tmp/ prefix. When commit passes validation (existence check via HEAD, Content-Type, size limit), it promotes the object to its final key with CopyObject and deletes the tmp original. Objects whose commit never arrives stay in tmp/ , and a lifecycle rule expires them after 7 days. Now the lifecycle rule only has to look at tmp/ . Real assets are outside its blast radius from the start. The clients didn't need to change. I read all three upload flows to confirm this: every one of them uses the key returned in the commit response for its follow-up calls, so the server can change the key shape without them noticing. Commits for old-format keys already in flight at deploy time still go through the existing path. One trap here. This bucket has versioning enabled. On a versioned bucket, expiration doesn't delete an object. It only adds a delete marker, and the original bytes stay behind as a noncurrent version. Without a paired noncurrent_version_expiration (1 day), the cleanup runs and not a single byte is rec

This continues from Part ③ . We have the system set to refresh data daily, but unexplained errors can sometimes occur. Symptoms The segments were not properly configured as expected. Checking System → Schedule showed that the automatic refresh schedule itself was fine. The data source refresh also appeared to have no issues. Opening the Unify screen, everything looks like it succeeded at first glance — but checking the Performance screen reveals that something is failing during unification. Drilling in further, the failure is happening at the Customer Profiles step. Resolution Steps Attempt to manually re-run just the "Customer Profiles" task → Error persists Redo everything manually from the data source refresh step onwards This resolved the issue successfully. Takeaway Since this is not a misconfiguration issue, patient repetition of the refresh process is enough to recover. Hopefully this saves someone some troubleshooting time.

AWS recently announced Amazon S3 Annotations, a feature that lets teams attach rich, searchable context such as summaries, classifications, compliance data, or AI-generated insights directly to S3 objects. Annotations can be updated independently of the object and queried across datasets, reducing the need for separate metadata systems. By Renato Losio

SeaweedFS is an open-source, distributed object storage system with an S3-compatible API, a filer for POSIX-style hierarchical access, and a small footprint. This guide deploys SeaweedFS using Docker Compose with the master, volume, filer, S3, and admin services behind Traefik for automatic HTTPS on separate admin and S3 domains. By the end, you'll have SeaweedFS serving S3-compatible object storage securely at your domains. Prerequisite: Two DNS A records pointing at the server — storage.example.com (admin dashboard) and s3.storage.example.com (S3 API). AWS CLI installed on your local machine for testing. Set Up the Directory Structure 1. Create the project directory: $ mkdir seaweedfs && cd seaweedfs 2. Generate an access key and a secret key (run twice and save both): $ openssl rand -hex 16 3. Create the environment file: $ nano .env STORAGE_DOMAIN = storage.example.com LETSENCRYPT_EMAIL = your-email@example.com ADMIN_PASSWORD = yourpassword 4. Create the S3 identities file: $ nano s3-config.json { "identities" : [ { "name" : "admin" , "credentials" : [ { "accessKey" : "YOUR_ACCESS_KEY" , "secretKey" : "YOUR_SECRET_KEY" } ], "actions" : [ "Admin" , "Read" , "Write" , "List" , "Tagging" ] } ] } Deploy with Docker Compose 1. Create the Compose manifest: $ nano docker-compose.yml services : traefik : image : traefik:v3.7.0 container_name : traefik restart : unless-stopped ports : - " 80:80" - " 443:443" volumes : - /var/run/docker.sock:/var/run/docker.sock:ro - ./letsencrypt:/letsencrypt command : - --providers.docker=true - --providers.docker.exposedByDefault=false - --entrypoints.web.address=:80 - --entrypoints.websecure.address=:443 - --entrypoints.web.http.redirections.entrypoint.to=websecure - --entrypoints.web.http.redirections.entrypoint.scheme=https - --entrypoints.web.http.redirections.entrypoint.permanent=true - --certificatesresolvers.letsencrypt.acme.email=${LETSENCRYPT_EMAIL} - --certificatesresolvers.letsencrypt.acme.storage=/letsencrypt/acme.json - --

Introduction In the previous articles, we explored AWS networking concepts like: VPC Subnets Internet Gateway Security Groups Route 53 Now, let's move to one of the most popular AWS services: Amazon S3 (Simple Storage Service) Amazon S3 is one of the easiest AWS services to learn and one of the most widely used services in real-world applications. Almost every application stores some kind of data: Images Videos Documents Log files Backups Static websites Amazon S3 helps us store and retrieve these files securely from anywhere in the world. In this article, we will understand: What Amazon S3 is Buckets and Objects Benefits of S3 Storage Classes Versioning Basic security concepts Static website hosting overview Why Do We Need Storage? Imagine you are running an online photo-sharing application. Users upload: Profile pictures Photos Videos Where should these files be stored? Keeping them directly inside application servers is not a good idea because: Storage is limited. Scaling becomes difficult. Replacing servers may cause data loss. This is where Amazon S3 helps. What is Amazon S3? Amazon S3 stands for: Simple Storage Service It is a cloud-based object storage service provided by AWS. S3 allows us to: Store data Retrieve data Manage data from anywhere using the internet. Amazon S3 is designed to be: Highly available Durable Scalable Secure Cost-effective Real-World Example Think of Amazon S3 as a digital warehouse. Suppose you own an e-commerce company. Inside your warehouse, you store: Product images Invoices Customer documents Videos Similarly, Amazon S3 stores digital files safely in the cloud. Buckets and Objects Amazon S3 stores data using two concepts: Bucket A bucket is a container used to store files. Think of a bucket like a folder. Examples: company-documents customer-images application-logs Bucket names must be globally unique. Object Anything stored inside a bucket is called an Object . Examples: invoice.pdf profile.jpg backup.zip video.mp4 Objects can co

If you’ve used S3, MinIO, or any cloud storage API, it’s easy to assume object storage is just a “cloud folder system.” That assumption is wrong — and it leads to confusion when you start working with distributed systems. Object storage is not a file system. It’s closer to a distributed key-value system with strong durability guarantees and a very specific access model . Once you understand that shift, a lot of cloud infrastructure starts to make more sense. The mental model most people start with When people first see object storage, they imagine something like this: /photos/cats.png /photos/dogs.png A hierarchical file system: folders subfolders files inside directories This is how traditional systems like ext4 or NTFS work. But object storage doesn’t actually work this way. The actual model: key → object Object storage is much simpler at its core: key → value Example: key : photos/cats.png value : <binary data> There are no real folders. “folders” are just string prefixes used for organization. That’s it. Why this design exists This model isn’t accidental. It solves real distributed system problems. Traditional file systems struggle when you try to: scale across many machines replicate data reliably handle partial failures coordinate metadata changes at scale Object storage avoids many of these problems by simplifying the model. Instead of supporting complex file operations, it focuses on: store object retrieve object delete object list objects by prefix Nothing more. The most important design choice: immutability In most object storage systems: Objects are not modified in place. If you “update” a file, what actually happens is: upload a new object replace the key pointer old object becomes orphaned (eventually cleaned up) This is a huge shift from file systems. Why this matters Immutability makes distributed systems easier because: no concurrent write conflicts on the same object replication becomes simpler caching becomes safer failure recovery is easier to rea

I recently read Jérémie Rodon's excellent article On-Demand Archives on S3 , where he describes an elegant Rust solution for zipping 3,000 × 5MB files from S3 within a single Lambda function. His approach is impressive: streaming a ZIP archive through a custom Rotating Slab Buffer, saturating bandwidth with concurrent downloads, all within 512MB of RAM. The result: 3 minutes 35 seconds . I thought it was a good challenge to reach better performance. His article ends with an open invitation: "do you think you can do better with your favorite language?" Well, my favorite language is not Rust nor Go nor.. however, I'm fluent in serverless ;) so I took a different angle entirely. A Different Approach: Why Not Parallelize the Problem? Jérémie's constraint was a single Lambda. That's elegant, but it means you're bound by one machine's network bandwidth (~600 Mbps). No matter how perfect your streaming is, physics wins: 15GB at 600 Mbps ≈ 200 seconds minimum. My question was: what if we break that single-machine bottleneck? The key insight is that ZIP files in STORE mode (no compression) have deterministic byte offsets . Each entry is exactly 50 + len(filename) + filesize bytes (local header + ZIP64 extra field + data). If you know all filenames and sizes upfront, you can pre-calculate exactly where every file will land in the final archive, before downloading a single byte. This means independent workers can each build their portion of the zip in parallel, and S3's multipart upload lets them write their chunks independently (parts can be uploaded in any order by different processes sharing the same upload ID). Architecture Planner Lambda → Step Functions Distributed Map → N Worker Lambdas → Finalizer Lambda │ │ │ │ │ │ CreateMultipartUpload │ │ │ UploadPart (parallel) │ CompleteMultipartUpload ▼ ▼ ▼ ▼ ▼ S3 Output Bucket Planner : Lists all source files, computes zip byte offsets, initiates multipart upload, divides work into balanced batches (equal data volume per worker)