标签：#s3

AWS recently announced Amazon S3 Annotations, a feature that lets teams attach rich, searchable context such as summaries, classifications, compliance data, or AI-generated insights directly to S3 objects. Annotations can be updated independently of the object and queried across datasets, reducing the need for separate metadata systems. By Renato Losio

SeaweedFS is an open-source, distributed object storage system with an S3-compatible API, a filer for POSIX-style hierarchical access, and a small footprint. This guide deploys SeaweedFS using Docker Compose with the master, volume, filer, S3, and admin services behind Traefik for automatic HTTPS on separate admin and S3 domains. By the end, you'll have SeaweedFS serving S3-compatible object storage securely at your domains. Prerequisite: Two DNS A records pointing at the server — storage.example.com (admin dashboard) and s3.storage.example.com (S3 API). AWS CLI installed on your local machine for testing. Set Up the Directory Structure 1. Create the project directory: $ mkdir seaweedfs && cd seaweedfs 2. Generate an access key and a secret key (run twice and save both): $ openssl rand -hex 16 3. Create the environment file: $ nano .env STORAGE_DOMAIN = storage.example.com LETSENCRYPT_EMAIL = your-email@example.com ADMIN_PASSWORD = yourpassword 4. Create the S3 identities file: $ nano s3-config.json { "identities" : [ { "name" : "admin" , "credentials" : [ { "accessKey" : "YOUR_ACCESS_KEY" , "secretKey" : "YOUR_SECRET_KEY" } ], "actions" : [ "Admin" , "Read" , "Write" , "List" , "Tagging" ] } ] } Deploy with Docker Compose 1. Create the Compose manifest: $ nano docker-compose.yml services : traefik : image : traefik:v3.7.0 container_name : traefik restart : unless-stopped ports : - " 80:80" - " 443:443" volumes : - /var/run/docker.sock:/var/run/docker.sock:ro - ./letsencrypt:/letsencrypt command : - --providers.docker=true - --providers.docker.exposedByDefault=false - --entrypoints.web.address=:80 - --entrypoints.websecure.address=:443 - --entrypoints.web.http.redirections.entrypoint.to=websecure - --entrypoints.web.http.redirections.entrypoint.scheme=https - --entrypoints.web.http.redirections.entrypoint.permanent=true - --certificatesresolvers.letsencrypt.acme.email=${LETSENCRYPT_EMAIL} - --certificatesresolvers.letsencrypt.acme.storage=/letsencrypt/acme.json - --

Introduction In the previous articles, we explored AWS networking concepts like: VPC Subnets Internet Gateway Security Groups Route 53 Now, let's move to one of the most popular AWS services: Amazon S3 (Simple Storage Service) Amazon S3 is one of the easiest AWS services to learn and one of the most widely used services in real-world applications. Almost every application stores some kind of data: Images Videos Documents Log files Backups Static websites Amazon S3 helps us store and retrieve these files securely from anywhere in the world. In this article, we will understand: What Amazon S3 is Buckets and Objects Benefits of S3 Storage Classes Versioning Basic security concepts Static website hosting overview Why Do We Need Storage? Imagine you are running an online photo-sharing application. Users upload: Profile pictures Photos Videos Where should these files be stored? Keeping them directly inside application servers is not a good idea because: Storage is limited. Scaling becomes difficult. Replacing servers may cause data loss. This is where Amazon S3 helps. What is Amazon S3? Amazon S3 stands for: Simple Storage Service It is a cloud-based object storage service provided by AWS. S3 allows us to: Store data Retrieve data Manage data from anywhere using the internet. Amazon S3 is designed to be: Highly available Durable Scalable Secure Cost-effective Real-World Example Think of Amazon S3 as a digital warehouse. Suppose you own an e-commerce company. Inside your warehouse, you store: Product images Invoices Customer documents Videos Similarly, Amazon S3 stores digital files safely in the cloud. Buckets and Objects Amazon S3 stores data using two concepts: Bucket A bucket is a container used to store files. Think of a bucket like a folder. Examples: company-documents customer-images application-logs Bucket names must be globally unique. Object Anything stored inside a bucket is called an Object . Examples: invoice.pdf profile.jpg backup.zip video.mp4 Objects can co

If you’ve used S3, MinIO, or any cloud storage API, it’s easy to assume object storage is just a “cloud folder system.” That assumption is wrong — and it leads to confusion when you start working with distributed systems. Object storage is not a file system. It’s closer to a distributed key-value system with strong durability guarantees and a very specific access model . Once you understand that shift, a lot of cloud infrastructure starts to make more sense. The mental model most people start with When people first see object storage, they imagine something like this: /photos/cats.png /photos/dogs.png A hierarchical file system: folders subfolders files inside directories This is how traditional systems like ext4 or NTFS work. But object storage doesn’t actually work this way. The actual model: key → object Object storage is much simpler at its core: key → value Example: key : photos/cats.png value : <binary data> There are no real folders. “folders” are just string prefixes used for organization. That’s it. Why this design exists This model isn’t accidental. It solves real distributed system problems. Traditional file systems struggle when you try to: scale across many machines replicate data reliably handle partial failures coordinate metadata changes at scale Object storage avoids many of these problems by simplifying the model. Instead of supporting complex file operations, it focuses on: store object retrieve object delete object list objects by prefix Nothing more. The most important design choice: immutability In most object storage systems: Objects are not modified in place. If you “update” a file, what actually happens is: upload a new object replace the key pointer old object becomes orphaned (eventually cleaned up) This is a huge shift from file systems. Why this matters Immutability makes distributed systems easier because: no concurrent write conflicts on the same object replication becomes simpler caching becomes safer failure recovery is easier to rea

I recently read Jérémie Rodon's excellent article On-Demand Archives on S3 , where he describes an elegant Rust solution for zipping 3,000 × 5MB files from S3 within a single Lambda function. His approach is impressive: streaming a ZIP archive through a custom Rotating Slab Buffer, saturating bandwidth with concurrent downloads, all within 512MB of RAM. The result: 3 minutes 35 seconds . I thought it was a good challenge to reach better performance. His article ends with an open invitation: "do you think you can do better with your favorite language?" Well, my favorite language is not Rust nor Go nor.. however, I'm fluent in serverless ;) so I took a different angle entirely. A Different Approach: Why Not Parallelize the Problem? Jérémie's constraint was a single Lambda. That's elegant, but it means you're bound by one machine's network bandwidth (~600 Mbps). No matter how perfect your streaming is, physics wins: 15GB at 600 Mbps ≈ 200 seconds minimum. My question was: what if we break that single-machine bottleneck? The key insight is that ZIP files in STORE mode (no compression) have deterministic byte offsets . Each entry is exactly 50 + len(filename) + filesize bytes (local header + ZIP64 extra field + data). If you know all filenames and sizes upfront, you can pre-calculate exactly where every file will land in the final archive, before downloading a single byte. This means independent workers can each build their portion of the zip in parallel, and S3's multipart upload lets them write their chunks independently (parts can be uploaded in any order by different processes sharing the same upload ID). Architecture Planner Lambda → Step Functions Distributed Map → N Worker Lambdas → Finalizer Lambda │ │ │ │ │ │ CreateMultipartUpload │ │ │ UploadPart (parallel) │ CompleteMultipartUpload ▼ ▼ ▼ ▼ ▼ S3 Output Bucket Planner : Lists all source files, computes zip byte offsets, initiates multipart upload, divides work into balanced batches (equal data volume per worker)