Why I wrote this
After I got one app working on MinIO, I thought I understood the self-hosted storage problem.
That confidence lasted exactly until the next deployment.
The new constraint looked similar: no S3, no managed object storage, media-heavy app, private/generated files, FastAPI backend, browser-facing URLs, and enough user uploads that "just put it in Postgres" would be a crime.
MinIO was the obvious first thought because it had already worked. It is S3-compatible, easy to run, and the boto3 migration story is basically one endpoint_url. But by this point I was also thinking about the enterprise side of the decision. MinIO is still open source, but serious enterprise packaging, support, and newer commercial product positioning live behind MinIO's paid AIStor/subscription world. For a client-owned deployment where the whole premise was "we want control and predictable cost", I wanted to at least look at the other serious self-hosted option.
That is how SeaweedFS entered the chat.
Note — SeaweedFS: a distributed storage system originally designed around fast file lookup and efficient small-file storage. It can expose a filesystem-like filer, volume servers, and an S3-compatible API. That last part is what made it usable for my app without rewriting every upload/download path.
MinIO felt like "S3, but on my server."
SeaweedFS felt like "a storage system that happens to speak S3."
That difference matters.
The terms, because I will forget them
If I come back to this in six months, this is the section I want near the top.
Object storage: storage for blobs addressed by keys, not rows or normal filesystem paths. Example key: users/42/gallery/image.webp. The slash is part of the name; it is not necessarily a real folder.
Bucket: a top-level container for objects. S3 calls it a bucket. S3-compatible systems usually keep the same word.
S3-compatible: not AWS S3, but speaks enough of the same API that tools like boto3, AWS CLI, and pre-signed URLs can work.
boto3: the Python AWS SDK. In this project it is not "AWS code" anymore; it is just the S3 protocol client.
Endpoint URL: the base URL boto3 talks to. For AWS it is implicit. For MinIO/SeaweedFS it is explicit, like http://seaweed:8333.
SigV4: AWS Signature Version 4, the signing algorithm used for authenticated S3 requests and pre-signed URLs.
Pre-signed URL: a temporary URL that contains a signature. Anyone with the URL can access that one object until it expires.
Path-style URL: https://media.example.com/bucket/key.
Virtual-hosted-style URL: https://bucket.media.example.com/key.
Filer: SeaweedFS's metadata/file layer. It gives a directory-like view over files whose bytes live on volume servers.
Volume server: SeaweedFS component that stores actual file content.
Master: SeaweedFS coordinator that tracks volumes and where file ids live.
S3 gateway/API: the SeaweedFS-facing door that lets normal S3 clients upload and download objects.
That is the whole mental model: my app talks S3, SeaweedFS translates that into its own filer/volume world, and the browser receives either public URLs or pre-signed URLs.
Why not just stay on MinIO?
I still like MinIO. For a single machine, simple object storage, and a codebase that already uses boto3, it is almost unfairly convenient. One binary, one console, one bucket, done.
But this app had a slightly different smell:
Lots of generated media. Images, audio, and possibly video. That means many files, many reads, and a lot of cache behavior.
A need for public and internal paths. The backend should talk to storage privately. The browser should see a clean HTTPS media domain.
A future where "single disk with object API" might stop being enough. I did not need a giant cluster on day one, but I wanted a path that did not force another migration the moment storage became more serious.
SeaweedFS looked attractive because it is not only trying to mimic S3. Its architecture has masters, volume servers, filers, and an S3 gateway. That is more moving parts, yes. But it also means the S3 API is not the whole product.
Note — filer: in SeaweedFS, the filer gives you a directory/file abstraction and metadata layer. The actual bytes live on volume servers. The S3 API sits in front of that world so S3 clients can still PUT and GET objects.
The official SeaweedFS README talks a lot about fast small-file access, volume-id lookup, and using existing metadata stores for filer metadata. That is not something I cared about in the first hour. I cared about "does boto3 upload work?" But later, those design choices started making the system feel more powerful than MinIO for the kind of messy media workload I was dealing with.
MinIO vs SeaweedFS in my head
This is not a benchmark. This is the practical decision table I wish I had while choosing.
MinIO:
best when I want S3-compatible object storage quickly
mental model server -> bucket -> object
setup feel simple
operations easier to explain
downside feels narrower if I later need a bigger storage architecture
SeaweedFS:
best when I expect lots of files and want room to grow storage topology
mental model master -> filer -> volume -> optional S3 API
setup feel more nouns, more config, more thinking
operations more powerful but less beginner-friendly
downside I need to understand the pieces before debugging productionIf the app only needs "store some files and give me S3 URLs", MinIO is the clean answer.
If the app is media-heavy and might eventually need multiple storage nodes, filer behavior, public/internal endpoints, and more control over how files are organized, SeaweedFS becomes interesting.
I did not pick SeaweedFS because it was easier. I picked it because the future shape of the app looked less tiny than the first deployment.
The boto3 config was familiar
The pleasant surprise: the app code still stayed S3-shaped.
The client factory looked almost the same as MinIO. The important bits were endpoint_url, SigV4, and path-style addressing:
import boto3
from botocore.config import Config
async def get_s3_client():
endpoint_url = settings.AWS_S3_ENDPOINT_URL
if not endpoint_url:
raise ValueError("AWS_S3_ENDPOINT_URL is not configured")
config = Config(
signature_version="s3v4",
s3={"addressing_style": "path"},
)
return boto3.client(
"s3",
aws_access_key_id=settings.AWS_ACCESS_KEY_ID,
aws_secret_access_key=settings.AWS_SECRET_ACCESS_KEY,
region_name=settings.AWS_REGION,
endpoint_url=endpoint_url,
config=config,
)Note — path-style addressing:
https://storage.example.com/bucket/keyinstead ofhttps://bucket.storage.example.com/key. Self-hosted S3-compatible systems usually want path-style unless you have wildcard DNS and TLS set up for every bucket hostname.
Uploads stayed boring, which is exactly what I wanted:
async def upload_to_s3_file(file_obj, s3_key, content_type, bucket_name, s3_client=None):
if s3_client is None:
s3_client = await get_s3_client()
def _upload():
s3_client.upload_fileobj(
Fileobj=file_obj,
Bucket=bucket_name,
Key=s3_key,
ExtraArgs={"ContentType": content_type},
)
await asyncio.to_thread(_upload)
presigned_url = await generate_presigned_url(s3_key, bucket_name=bucket_name)
return s3_key, presigned_urlThat asyncio.to_thread is doing real work. boto3 is blocking. FastAPI is async. If you call boto3 directly inside the request handler, one slow upload/download can block the event loop and make unrelated requests feel haunted. Offloading the blocking storage call into a thread keeps the async server breathing.
The storage module boundary
The smartest thing in this migration was not SeaweedFS. It was keeping storage behind one module.
The rest of the app should not know whether the backend is AWS S3, MinIO, SeaweedFS, or a directory behind nginx. Most code should only need verbs like:
await upload_to_s3_file(file_obj, key, content_type, bucket)
url = await generate_presigned_url(key)
public_url = await generate_public_s3_url(key)
await delete_s3_object(key)That sounds boring, but boring boundaries save migrations.
The module owns:
credentials
endpoint_url
public_url
bucket name
path-style addressing
presigned URL generation
blocking boto3 calls
Redis caching
ContentType
delete behaviorThe feature code owns:
what file is being uploaded
who is allowed to see it
what key/name it should have
what DB row points to itWhen those boundaries blur, every storage migration becomes a treasure hunt. I prefer boring now. Boring is portable.
The part SeaweedFS forced me to understand
With MinIO, I had already learned that pre-signed URLs are fragile. The host is part of the SigV4 signature. Change the host after signing and you get SignatureDoesNotMatch.
SeaweedFS made that lesson more explicit because I wanted two endpoints:
AWS_S3_ENDPOINT_URL=http://seaweed:8333
AWS_S3_PUBLIC_URL=https://media.example.com
AWS_BUCKET_NAME=app-mediaThe backend container should use the internal endpoint:
http://seaweed:8333The browser should use the public HTTPS endpoint:
https://media.example.comMy first instinct was to generate the URL internally, then rewrite the host to the public domain. That is exactly the trap from the MinIO migration. It looks like a harmless string operation, but it breaks the signature.
So I split the clients.
One client is for backend storage operations:
async def get_s3_client():
return boto3.client(
"s3",
endpoint_url=settings.AWS_S3_ENDPOINT_URL,
aws_access_key_id=settings.AWS_ACCESS_KEY_ID,
aws_secret_access_key=settings.AWS_SECRET_ACCESS_KEY,
region_name=settings.AWS_REGION,
config=Config(signature_version="s3v4", s3={"addressing_style": "path"}),
)Another client is specifically for pre-signed URLs:
async def get_s3_presign_client():
endpoint_url = (
settings.AWS_S3_PUBLIC_URL
or settings.AWS_S3_ENDPOINT_URL
).rstrip("/")
return boto3.client(
"s3",
endpoint_url=endpoint_url,
aws_access_key_id=settings.AWS_ACCESS_KEY_ID,
aws_secret_access_key=settings.AWS_SECRET_ACCESS_KEY,
region_name=settings.AWS_REGION,
config=Config(signature_version="s3v4", s3={"addressing_style": "path"}),
)That is the clean version of what I learned the painful way:
Sign the URL with the hostname the browser will actually request.
If the browser requests https://media.example.com/app-media/file.webp, the pre-signing client should also use https://media.example.com. Not http://seaweed:8333. Not an IP. Not a Docker service name. The public host.
Why hostnames break signatures
This is the thing I keep needing to re-explain to myself.
A pre-signed URL is not just "a URL with a token". It is a signed representation of a specific HTTP request. The signature is calculated from pieces like:
HTTP method
path
query params
headers
host
expiry
secret keySo if boto3 signs this:
GET http://seaweed:8333/app-media/image.webpand the browser actually requests this:
GET https://media.example.com/app-media/image.webpSeaweedFS receives a request whose host does not match the signed host. The storage server recalculates the signature from the request it received, compares it to the signature in the URL, and says no.
This is why the fix is not "replace the hostname in the string".
The fix is:
Use internal endpoint for backend upload/download.
Use public endpoint for presigned URL generation.
Make nginx/proxy route public endpoint to the storage service without changing path/query.That is the shape.
The public URL helper
Not every file needs a pre-signed URL. Some generated assets can be public, and for those I wanted a fast path that did not sign anything:
import urllib.parse
async def generate_public_s3_url(s3_key: str) -> str | None:
if not s3_key:
return None
bucket_name = settings.AWS_BUCKET_NAME
encoded_key = urllib.parse.quote(s3_key, safe="/")
public_endpoint = (
settings.AWS_S3_PUBLIC_URL
or settings.AWS_S3_ENDPOINT_URL
).rstrip("/")
return f"{public_endpoint}/{bucket_name}/{encoded_key}"This is not a security feature. This is the opposite: it assumes the object is public or the endpoint is allowed to serve it without a signature. But it matters because not all media in an app has the same privacy level.
Private generated content gets pre-signed URLs.
Public thumbnails, static generated samples, and assets meant to be indexed can use clean public URLs.
One storage backend. Two access patterns.
Caching pre-signed URLs without lying to yourself
Pre-signing is cheap, but not free. Also, signing on every list endpoint is wasteful if you are rendering dozens of media items. So I cached pre-signed URLs in Redis, always with a TTL shorter than the actual expiry:
async def generate_presigned_url(s3_key, expires_in=None, bucket_name=None):
if not s3_key:
return None
expires_in = expires_in or settings.S3_PRESIGNED_EXPIRES
cached = await redis_cache.get_presigned(s3_key)
if cached:
return cached
s3_client = await get_s3_presign_client()
bucket = bucket_name or settings.AWS_BUCKET_NAME
url = s3_client.generate_presigned_url(
ClientMethod="get_object",
Params={"Bucket": bucket, "Key": s3_key},
ExpiresIn=expires_in,
)
cache_ttl = min(max(expires_in - 60, 0), 86400)
await redis_cache.set_presigned(s3_key, url, ttl=cache_ttl)
return urlThe expires_in - 60 part is the boring bit that prevents edge-of-expiry bugs. You do not want to hand the frontend a URL that technically has three seconds left because Redis still had it. Shave a minute off and sleep better.
I also learned from the MinIO migration that cache keys should be treated as infrastructure-coupled. If you change storage hostnames, old pre-signed URLs can survive in Redis and point at the wrong world. The fix is either clear the cache during migration or store enough metadata to reject stale URLs.
The smoke tests I want before trusting storage
I do not trust storage because config "looks right" anymore. I trust storage after a script puts bytes in and gets the same bytes out.
def smoke_storage():
client = get_s3_client()
bucket = os.environ["AWS_BUCKET_NAME"]
key = "smoke/test.txt"
data = b"hello-storage"
client.put_object(
Bucket=bucket,
Key=key,
Body=data,
ContentType="text/plain",
)
got = client.get_object(Bucket=bucket, Key=key)["Body"].read()
assert got == data
url = client.generate_presigned_url(
ClientMethod="get_object",
Params={"Bucket": bucket, "Key": key},
ExpiresIn=300,
)
print(url)
client.delete_object(Bucket=bucket, Key=key)Then I test the public URL path too:
curl -I "https://media.example.com/app-media/smoke/test.txt"And if it is pre-signed:
curl -I "$PRESIGNED_URL"The failure messages tell you where to look:
AccessDenied credentials or bucket policy
NoSuchBucket bucket was never created or wrong bucket env
NoSuchKey object key mismatch
SignatureDoesNotMatch host/path/query changed after signing
Connection refused endpoint/port/proxy problem
SSL error public URL/TLS/proxy certificate problemThis little table alone would have saved me hours.
The production-ish environment shape
The env vars that mattered were basically these:
AWS_ACCESS_KEY_ID=...
AWS_SECRET_ACCESS_KEY=...
AWS_BUCKET_NAME=app-media
AWS_REGION=us-east-1
# Internal service URL. Backend uses this for direct storage operations.
AWS_S3_ENDPOINT_URL=http://seaweed:8333
# Public browser URL. Presigned URLs should be generated against this host.
AWS_S3_PUBLIC_URL=https://media.example.com
S3_PRESIGNED_EXPIRES=3600
REDIS_URL=redis://redis:6379/0The names still say AWS_, which is a little funny because AWS is exactly what I was not using. I kept them because boto3 and old code already spoke that language. Sometimes consistency beats semantic purity.
Future-me note: if public files are broken, inspect AWS_S3_PUBLIC_URL. If backend uploads are broken, inspect AWS_S3_ENDPOINT_URL. If pre-signed URLs generate but fail in browser, inspect the host in the generated URL before touching the upload code.
Why SeaweedFS felt harder
SeaweedFS was harder because it has more nouns.
MinIO gives you server, bucket, object.
SeaweedFS gives you master, volume, filer, S3 gateway, bucket, collection, replication, filer metadata, public endpoint, internal endpoint, and sometimes IAM depending on how you deploy it.
That does not make it bad. It makes it less fake-simple.
With MinIO, I mostly thought about S3 compatibility.
With SeaweedFS, I had to think about how the storage system itself was shaped. Where does metadata live? Which endpoint should the app use? Which endpoint should the browser use? What happens when I add another volume server? Am I using the S3 layer as the main interface, or do I eventually want filer-level access too?
This is where SeaweedFS started feeling more powerful. It was not trying to be only a replacement for S3. It was giving me a storage architecture I could grow into.
Note — volume server: the SeaweedFS component that stores actual file content. The master knows which volume has which file id. This is part of why SeaweedFS can make file lookup efficient: once you know the volume id, finding the bytes is direct and cacheable.
The cost of that power is operational complexity. You have to understand more before you can debug confidently. When something fails, "is the S3 API up?" is not the only question. You may need to ask whether the filer can talk to volumes, whether the public endpoint is signing correctly, whether the bucket exists, whether the credentials map to the right permissions, and whether your reverse proxy is preserving the host the signature expects.
What I would check during an incident
If media suddenly breaks, I would debug from the outside inward.
1. Is the browser URL sane?
Does it use https?
Does it contain the expected bucket?
Does the hostname match AWS_S3_PUBLIC_URL?
Does it still point to an old MinIO/S3 host?2. Does the public endpoint answer?
curl -I https://media.example.com3. Can the backend talk to internal storage?
curl -I http://seaweed:83334. Can boto3 list buckets from inside the backend environment?
client = await get_s3_client()
print(client.list_buckets())5. Is Redis handing out stale URLs?
Clear presigned URL cache after endpoint changes.
Never trust old cached URLs after storage migration.6. Is nginx preserving path and query string?
Pre-signed URLs need the exact path and query params. A proxy that drops query strings, normalizes paths, or redirects to a different host can break signatures.
This debugging order matters because storage failures can look like frontend bugs. The image tag fails. The video player spins. The gallery looks empty. But the real bug might be four layers down in a signature host mismatch.
Would I use it again?
Yes, but not automatically.
For a small app on one VM where the only requirement is "replace S3 quickly", I would still reach for MinIO first. It is simpler, friendlier, and easier to explain to the next developer.
For a media-heavy app with lots of generated files, a future need for scaling storage horizontally, and a client who wants self-hosted infrastructure without buying into a commercial object-store subscription, I would seriously consider SeaweedFS again.
It is lower-level in the best and worst ways.
The best way: it gives you more control over the storage architecture than "one S3-compatible box with a console".
The worst way: it makes you responsible for understanding that architecture.
The thing I liked most is that my application code did not have to become SeaweedFS-specific. It still spoke S3 through boto3:
s3_client.upload_fileobj(
Fileobj=file_obj,
Bucket=bucket_name,
Key=s3_key,
ExtraArgs={"ContentType": content_type},
)That is the trick I keep coming back to. The app should know as little as possible about the storage backend. Put the weirdness in one module. Make the rest of the code say "upload object", "generate URL", "delete object", and move on with its life.
MinIO taught me that S3-compatible storage can save a migration.
SeaweedFS taught me that S3 compatibility is just the front door. Sometimes the building behind it is much bigger.