Why I wrote this
Until this project, "object storage" and "S3" were the same word in my head. Bucket, IAM keys, boto3, done. I had never once thought about what happens underneath, because AWS's whole pitch is that I don't have to.
Note — object storage: unlike a filesystem (folders, files, paths) or a database (rows, queries), object storage stores blobs of data ("objects") in flat containers called buckets, addressed by a string key like
users/42/avatar.png. No real directories — the slashes in keys are just a naming convention. It's the standard way to store user uploads, media, backups, and anything too big or too numerous for a database.
Then a client told me the app had to run entirely on their own Linux servers. Not "prefer self-hosted", not "AWS but in a specific region" — no AWS account at all, no managed anything, their VMs, their disks. The app was a FastAPI backend generating a lot of media (images, audio, video), and every one of those files was going through S3 with pre-signed URLs.
Note — pre-signed URL: a temporary URL that grants access to a single private object without exposing your credentials. The server signs it using its secret key, embedding an expiry time and a cryptographic signature into the query string. Anyone holding the URL can fetch that one object until it expires — no login needed. This is how you serve private media directly from storage without routing every byte through your backend.
So I had a weekend to figure out what replaces S3 when S3 is not allowed.
What I looked at
I didn't do a deep bake-off, I'll be honest. I had a deadline. But I did spend an evening reading before committing, and this is roughly how each option died:
Ceph came up first because everyone says Ceph. It's the heavyweight distributed storage system — it can expose block storage, a filesystem, and an S3-compatible API (through a component called RADOS Gateway) all from the same cluster. Then I read the deployment docs and closed the tab. Ceph assumes a cluster of machines and someone whose job is to babysit it. I had two VMs and my job was shipping the app. Deploying Ceph here would have been solving a problem I didn't have while creating several I couldn't afford.
SeaweedFS looked genuinely interesting — it's designed around the Facebook Haystack paper, optimized for billions of small files, and the benchmarks are impressive. But its S3 API is a gateway bolted onto its own volume/filer architecture, and I didn't want to learn a new mental model (volumes? filers? masters?) just to serve media files for one app.
Just the filesystem + nginx was the tempting lazy option. Save uploads to a directory, let nginx serve them. But I'd lose pre-signed URLs — which the app leaned on heavily for private media — plus multipart uploads for large videos, and every line of existing storage code. What looks like the simple option was actually the biggest rewrite.
Note — multipart upload: the S3 mechanism for uploading large files in parallel chunks, with the ability to retry individual chunks on failure. For anything above ~100MB it's the difference between "upload failed at 98%, start over" and "retry chunk 47". boto3's
upload_fileobjuses it automatically above a size threshold — another thing you get for free by staying S3-compatible.
MinIO won for one boring reason: it speaks the actual S3 API. Same boto3, same calls, same v4 signatures. It ships as a single Go binary (or one Docker image), stores objects on a plain disk path, and includes a web console where the client could see their files. My storage code wouldn't know the difference. When the deciding factor is "how much of my working code do I get to keep", S3-compatible wins by default.
The migration was one config value (almost)
All my storage code already went through one helper module — every S3 call in the codebase funnels through storage.py. That turned out to be the luckiest architectural accident of the project. The only real change was giving boto3 an endpoint_url:
import boto3
from botocore.config import Config
from urllib.parse import urlparse
def get_s3_client():
endpoint = settings.AWS_S3_ENDPOINT_URL # e.g. http://minio:9000
access_key = settings.AWS_ACCESS_KEY_ID
secret_key = settings.AWS_SECRET_ACCESS_KEY
use_ssl = True
if endpoint:
parsed = urlparse(endpoint)
use_ssl = parsed.scheme == "https"
endpoint = endpoint.rstrip("/") # boto3 dislikes trailing slashes
botocore_cfg = Config(
signature_version="s3v4",
s3={"addressing_style": "path"},
retries={"max_attempts": 3, "mode": "standard"},
)
return boto3.client(
"s3",
endpoint_url=endpoint,
aws_access_key_id=access_key,
aws_secret_access_key=secret_key,
config=botocore_cfg,
use_ssl=use_ssl,
)The trick is that endpoint_url is None when you're on real AWS (boto3 falls back to the standard AWS endpoints), and points at your MinIO host otherwise. Same function, both worlds.
Two things in that config I only learned by hitting errors:
signature_version="s3v4" — MinIO only supports AWS Signature Version 4. Ancient code or old SDK defaults sometimes still use v2. If your requests come back with signature errors against MinIO, check this first.
Note — Signature v4 (SigV4): the algorithm AWS uses to authenticate API requests. Instead of sending your secret key, the client computes an HMAC signature over the request (method, host, path, headers, timestamp) and sends that. The server recomputes it and compares. This is also exactly what's inside a pre-signed URL — which becomes very important later in this post.
addressing_style: "path" — AWS defaults to virtual-hosted style, where the bucket name becomes part of the hostname: my-bucket.s3.amazonaws.com/key. For that to work on a self-hosted box, you'd need wildcard DNS (*.minio.example.com) resolving to your server, which nobody sets up for a single-node deployment. Path style puts the bucket in the path instead — minio:9000/my-bucket/key — and just works with one hostname.
Everything downstream of the client factory stayed untouched, because it's all just S3 calls:
def generate_presigned_url(s3_client, bucket, key, method="get", expires_in=36000):
client_method = "get_object" if method == "get" else "put_object"
return s3_client.generate_presigned_url(
ClientMethod=client_method,
Params={"Bucket": bucket, "Key": key},
ExpiresIn=expires_in,
)
def upload_fileobj(s3_client, fileobj, bucket, key, content_type=None):
extra_args = {"ContentType": content_type} if content_type else {}
s3_client.upload_fileobj(
Fileobj=fileobj, Bucket=bucket, Key=key,
ExtraArgs=extra_args or None,
)Setting ContentType on upload is one of those details that doesn't matter until it does: it's what makes the browser render an image inline instead of downloading it as a mystery file, and what makes <video> tags work at all.
I also wrote a dumb smoke test — create client, list buckets, put an object, read it back, delete it — and I now run it against every environment before believing anything works:
def run():
client = storage.get_s3_client()
print("Buckets:", [b["Name"] for b in client.list_buckets()["Buckets"]])
bucket, key, data = "app-media", "smoke_test/test.txt", b"hello-minio"
client.put_object(Bucket=bucket, Key=key, Body=data, ContentType="text/plain")
assert client.get_object(Bucket=bucket, Key=key)["Body"].read() == data
client.delete_object(Bucket=bucket, Key=key)
print("smoke test OK")The first time I ran it, it failed — wrong bucket name in the env file. Which is exactly why it exists. Thirty seconds at the bottom of the stack instead of an hour of debugging failed uploads three layers up. Unlike S3, MinIO doesn't auto-create anything: the bucket has to exist before the first put_object, either created through the web console (port 9001) or MinIO's mc CLI.
The part that actually cost me a day
Here's what nobody warns you about. MinIO was listening on http://<server-ip>:9000. The frontend was served over HTTPS. Every pre-signed URL the backend handed out was an http:// URL pointing at a raw IP — and browsers refuse to load HTTP content from an HTTPS page. On top of that, I was leaking the storage server's IP address to every single user in every media URL.
Note — mixed content: when an HTTPS page tries to load a resource (image, video, script) over plain HTTP. Browsers block it — silently for media, loudly in the console — because it undermines the security guarantee of the padlock. There is no setting to allow it for your users; you have to serve everything over HTTPS.
My first instinct was: easy, just rewrite the URL's host to the real HTTPS domain before returning it to the frontend. I did that. Everything broke with SignatureDoesNotMatch, and it took me embarrassingly long to understand why: the signature in a pre-signed URL is computed over the request including the host and path (that's the SigV4 from earlier). Change any of it and the signature no longer matches, and the storage server correctly rejects the request. The URL is a sealed envelope — you can't edit the address on it without breaking the seal.
The fix that worked: don't modify the signed URL at all. Wrap it. The backend rewrites MinIO URLs into a proxy URL on its own HTTPS domain, with the original pre-signed URL — signature intact — tucked into a query parameter:
def rewrite_minio_url_to_https(url: str) -> str:
# Can't touch the presigned URL itself — signature covers host+path.
# So we encode the whole thing and let /storage/proxy fetch it for us.
if not url or url.startswith("https://"):
return url
encoded = urllib.parse.quote_plus(url)
return f"{settings.BACKEND_URL}/storage/proxy?url={encoded}"quote_plus matters here: the pre-signed URL contains its own query string (?X-Amz-Signature=...&X-Amz-Expires=...), and if you don't URL-encode it before embedding it in another URL, everything after the first & gets interpreted as parameters of the outer URL and the signature gets truncated. Encode the whole thing, decode it on the other side.
The proxy endpoint decodes the URL, fetches it server-side, and streams the response back to the browser:
@app.api_route("/storage/proxy", methods=["GET", "HEAD"])
async def storage_proxy(request: Request, url: str):
decoded_url = urllib.parse.unquote_plus(url)
parsed = urllib.parse.urlparse(decoded_url)
# Without this check you've built an open proxy that will
# happily fetch any URL on the internet on your server's behalf.
allowed = urllib.parse.urlparse(settings.AWS_S3_ENDPOINT_URL).netloc
if parsed.netloc != allowed:
raise HTTPException(status_code=403, detail="forbidden host")
forward_headers = {"accept": "*/*"}
if range_hdr := request.headers.get("range"):
forward_headers["range"] = range_hdr # video seeking = 206 responses
async with httpx.AsyncClient(timeout=30.0) as client:
upstream = await client.get(decoded_url, headers=forward_headers)
passthrough = {
k: v for k in ("content-type", "content-length", "cache-control",
"accept-ranges", "content-range")
if (v := upstream.headers.get(k))
}
return Response(content=upstream.content,
status_code=upstream.status_code,
headers=passthrough)Two details in there that cost real debugging time:
The host allowlist. A proxy endpoint that fetches arbitrary URLs from a query parameter is an SSRF hole — someone can point it at your internal services (?url=http://localhost:8000/admin) and your server will fetch it with its own network access. Checking the target against the one configured MinIO endpoint closes that.
Note — SSRF (Server-Side Request Forgery): an attack where you trick a server into making HTTP requests on your behalf, typically to internal systems the attacker can't reach directly. Any endpoint that accepts a URL and fetches it is a candidate. Always allowlist the destinations.
Forwarding the Range header. Without it, video worked — but seeking didn't. The player would start from zero every time you dragged the timeline. Browsers implement video seeking by requesting byte ranges (Range: bytes=1048576-), and the server answers with 206 Partial Content and just that slice. If the proxy swallows the header, the upstream sends the whole file from byte zero and the player gives up on seeking. Same story for accept-ranges and content-range on the way back — the player needs those headers to know that seeking is even possible.
Is this proxy the "right" architecture? No. The proper fix is putting MinIO itself behind nginx with a real TLS certificate on its own subdomain, so pre-signed URLs are HTTPS from birth and browsers hit storage directly with no extra hop. But the cert situation on the client's box was not getting sorted that week, and the proxy shipped that day. It also quietly fixed CORS and stopped leaking the server IP, so I've made peace with it. If traffic ever justifies it, the nginx route is documented and waiting.
One more bug from the same migration, because it's a good one: I cache pre-signed URLs in Redis so I'm not re-signing on every request (signing is cheap, but at scale, every avoided call counts). After switching endpoints, users started getting broken media — the cache was still full of perfectly valid-looking amazonaws.com URLs pointing at buckets that no longer served anything. Cached URLs that outlive the infrastructure they point to. The fix is validating cache entries against the current endpoint before trusting them:
cached = await redis_cache.get_presigned(s3_key)
if cached:
stale = "amazonaws.com" in cached or not cached.startswith(current_endpoint)
if stale:
await redis_cache.delete(f"presigned:{s3_key}") # re-sign fresh
else:
return cachedGeneral lesson filed away: whenever infrastructure changes, ask what's still sitting in caches pointing at the old world.
Would I pick it again?
For this project, yes, without hesitation. The constraint was hard (data stays on their servers, full stop), the codebase was already S3-shaped, and the scale was one beefy VM — not a use case that needs a distributed storage story. The app also serves a lot of media, and not paying S3 egress on every video view is a real number, not a theoretical one.
Note — egress: cloud-speak for data leaving the provider's network, i.e. bytes served to your users. AWS charges roughly $0.09/GB for it. Uploads (ingress) are free; it's the downloads that get you. A media-heavy app's egress bill scales directly with its popularity, which is a rude thing to discover during a good month.
But I want to be honest about what I gave up, because I felt it. S3 quietly gives you eleven nines of durability — it stores every object redundantly across multiple physical facilities, and you never think about it. A single MinIO node is exactly as durable as the disk it sits on. Backups, monitoring, disk-full alerts at 2 AM — those are mine now. (MinIO does support erasure coding — spreading data plus parity blocks across multiple drives so it can survive drive failures, RAID-like but at the object level — but that needs multiple drives, which this deployment didn't have. Nightly volume backups were the honest answer at this scale.)
There's also no ecosystem. No lifecycle rules quietly moving old files to cheaper storage, no bucket events triggering functions, no CDN a checkbox away. MinIO can approximate some of it (bucket notifications, ILM policies), but you're assembling it yourself. And MinIO is AGPL-licensed — fine for running it as a service inside a deployment like this, but if you're embedding it in software you distribute, read the license before your lawyer does.
If your constraint is "not AWS" rather than "not cloud", look at Cloudflare R2 or Backblaze B2 first — both speak the S3 API, both are drastically cheaper on egress (R2's is literally free), and neither makes you the on-call storage admin. If you're drowning in billions of tiny files, SeaweedFS is built for exactly that. If you have real multi-node durability requirements and a team to match, that's what Ceph is for. And if a directory of files behind nginx genuinely covers your needs, do that and don't feel bad about it.
The thing I actually took away from this isn't about MinIO at all. It's that the thin storage.py wrapper I wrote out of habit — one module, every S3 call goes through it — turned a forced infrastructure swap into a config change plus one proxy endpoint. I didn't design it for portability. But I'll pretend I did in the retro.