The Client Said On-Prem, So I Learned Linux Deployments

Why I wrote this

After the storage migration, I ran into the next very normal sentence that secretly means "your weekend is gone":

Can we deploy this on our own Linux server?

Not "can we deploy it to AWS". Not "can we use ECS". Not "please make a Terraform module and call it a day". The ask was two separate apps, two separate client-owned machines, and the same hard constraint underneath: keep it off managed AWS infrastructure.

This changed the shape of the problem immediately. ECS was out. RDS was out. ALB was out. CloudWatch was out. All the boring managed pieces I usually lean on became my problem again: process restarts, logs, ports, database connectivity, Redis, TLS, health checks, static assets, long-running requests, file permissions, and the tiny Linux details that wait until 2 AM to become interesting.

Note — on-prem: in this context, "on-prem" did not mean a polished datacenter with a platform team and runbooks. It meant Linux VMs controlled by the client, with their security rules, their access level, their firewall, their domain setup, and their appetite for giving me privileges. Which was, in one case, very little.

The funny part is that both deployments were basically the same product shape: a React/Vite frontend, a FastAPI backend, Postgres, Redis, media generation, payment/webhook endpoints, and admin routes. But the two servers allowed completely different deployment strategies.

One gave me Docker. The other gave me a shell account and no sudo.

Same app category. Completely different battle.

The mental model I wish I had before starting

Cloud deployment makes you think deployment is mainly "where do I push this image?" On-prem corrected me very quickly. Deployment is not one thing. It is a stack of small responsibilities that cloud services usually absorb for you.

This is the checklist I now keep in my head:

Runtime: how does the app process start? uvicorn, gunicorn, node, serve, something else?

Process supervision: if the process dies, who restarts it? Docker restart policy, systemd, PM2, or nobody?

Networking: which port does the app bind to, and is that port private or public?

Reverse proxy: what receives public traffic and forwards it inward? Usually nginx.

TLS: where does HTTPS terminate? nginx, a hosting panel, Cloudflare, or inside the app?

Persistence: where do Postgres, Redis, uploaded media, and generated files live?

Configuration: where do secrets and environment variables come from?

Observability: where do logs go, and how do I know the app is alive?

Recovery: after a reboot, what comes back automatically?

Note — process supervision: a process supervisor is the thing that keeps your app process alive. It starts it, restarts it after crashes, and usually gives you logs. ECS does this in AWS. Docker can do it with restart: unless-stopped. systemd does it at the OS level. PM2 can do it from a user account. A terminal tab does not do it. A terminal tab is a dare.

Once I started thinking in those boxes, the two deployments stopped feeling like chaos and started feeling like different answers to the same questions.

Docker deployment:

runtime              container CMD / entrypoint
process supervision  Docker restart policy
networking           Docker bridge network + exposed internal ports
reverse proxy        nginx container
TLS                  nginx/certbot
persistence          Docker volumes
logs                 docker compose logs
recovery             docker compose services restart on bootCopy

No-sudo deployment:

runtime              uvicorn + static frontend server
process supervision  PM2
networking           loopback ports only
reverse proxy        whatever the host already provided
TLS                  outside my user account
persistence          existing DB/storage paths
logs                 pm2 logs
recovery             pm2 save, if startup integration existsCopy

That second list is not pretty, but it is honest. And honest architecture is easier to debug than imaginary architecture.

The deployment I wanted: Docker, nginx, Postgres, Redis

The first machine let me use Docker, so I built the thing the way I wish every small on-prem deployment could be built: containers for the app boundary, compose for orchestration, nginx at the edge, and health checks everywhere.

The backend image was a fairly normal FastAPI production container. Build wheels in one stage, run a slim runtime in the second stage, drop privileges to a non-root user, then start Gunicorn with Uvicorn workers:

FROM python:3.12-slim AS builder

WORKDIR /build
RUN apt-get update && apt-get install -y --no-install-recommends \
    build-essential gcc g++ libpq-dev libffi-dev libssl-dev \
    && rm -rf /var/lib/apt/lists/*

COPY requirements.txt .
RUN pip install --upgrade pip setuptools wheel && \
    pip wheel --no-cache-dir --wheel-dir /wheels -r requirements.txt && \
    pip wheel --no-cache-dir --wheel-dir /wheels gunicorn uvicorn[standard]

FROM python:3.12-slim

ENV PYTHONDONTWRITEBYTECODE=1 \
    PYTHONUNBUFFERED=1 \
    PORT=8000 \
    WEB_CONCURRENCY=4 \
    GUNICORN_TIMEOUT=300

WORKDIR /app
RUN apt-get update && apt-get install -y --no-install-recommends \
    libpq5 curl ca-certificates ffmpeg \
    && rm -rf /var/lib/apt/lists/*

COPY --from=builder /wheels /wheels
RUN pip install --no-cache-dir --no-index --find-links=/wheels /wheels/*.whl \
    && rm -rf /wheels

RUN groupadd -r app && useradd -r -g app -u 1000 app
COPY --chown=app:app ./app ./app
COPY --chown=app:app docker-entrypoint.sh /usr/local/bin/docker-entrypoint.sh
RUN chmod +x /usr/local/bin/docker-entrypoint.sh

USER app
EXPOSE 8000

HEALTHCHECK --interval=15s --timeout=3s --start-period=20s --retries=3 \
  CMD curl -f http://127.0.0.1:${PORT}/health || exit 1

ENTRYPOINT ["/usr/local/bin/docker-entrypoint.sh"]
CMD ["gunicorn", "app.main:app"]Copy

That ffmpeg line looks random until you remember media apps are never just "web apps". Someone will upload a video, the backend will need to inspect it, transcode it, or generate something from it, and suddenly your clean Python image needs system packages.

The entrypoint did the part I usually want systemd or ECS to do for me: calculate worker count, set timeouts, and start the backend in a way that can survive real traffic:

#!/usr/bin/env bash
set -euo pipefail

if [ -z "${WEB_CONCURRENCY:-}" ]; then
  CPU_COUNT=$(nproc 2>/dev/null || echo 1)
  WORKERS_PER_CORE=${WORKERS_PER_CORE:-2}
  MAX_WORKERS=${MAX_WORKERS:-12}
  CALCULATED_WORKERS=$((CPU_COUNT * WORKERS_PER_CORE + 1))

  if [ "$CALCULATED_WORKERS" -gt "$MAX_WORKERS" ]; then
    WEB_CONCURRENCY=$MAX_WORKERS
  else
    WEB_CONCURRENCY=$CALCULATED_WORKERS
  fi
fi

exec "$@" \
  -k uvicorn.workers.UvicornWorker \
  -w "${WEB_CONCURRENCY}" \
  -b "0.0.0.0:${PORT:-8000}" \
  --timeout "${GUNICORN_TIMEOUT:-300}" \
  --graceful-timeout "${GUNICORN_GRACEFUL_TIMEOUT:-30}" \
  --max-requests "${GUNICORN_MAX_REQUESTS:-10000}" \
  --max-requests-jitter "${GUNICORN_MAX_REQUESTS_JITTER:-1000}" \
  --worker-tmp-dir /dev/shm \
  --access-logfile - \
  --error-logfile -Copy

Note — Gunicorn + Uvicorn: Uvicorn is the ASGI server that actually speaks async Python to FastAPI. Gunicorn is the process manager around it. Running gunicorn -k uvicorn.workers.UvicornWorker gives you multiple worker processes, graceful restarts, timeouts, and process supervision inside the container.

Compose then became the map of the whole app:

services:
  backend:
    build:
      context: .
      dockerfile: Dockerfile
    env_file: .env
    expose:
      - "8000"
    restart: unless-stopped
    depends_on:
      db:
        condition: service_healthy
      redis:
        condition: service_healthy
    healthcheck:
      test: ["CMD", "curl", "-fsS", "http://127.0.0.1:8000/health"]
      interval: 20s
      timeout: 10s
      retries: 3

  db:
    image: postgres:16
    restart: unless-stopped
    environment:
      POSTGRES_USER: app
      POSTGRES_PASSWORD: app
      POSTGRES_DB: app
    volumes:
      - pgdata:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U app -d app"]

  redis:
    image: redis:7-alpine
    restart: unless-stopped
    command: >
      redis-server
      --maxmemory 2gb
      --maxmemory-policy allkeys-lru
      --appendonly yes
      --appendfsync everysec

volumes:
  pgdata:Copy

The important thing here is not that compose is glamorous. It is not. The important thing is that the runtime is explicit. If the backend dies, restart it. If Postgres is not healthy, do not start the backend yet. If Redis goes away, make it obvious. If logs grow forever, rotate them. If the server reboots, bring everything back.

Cloud platforms hide this machinery so well that you forget it exists. On a Linux box, it very much exists.

Why each Docker piece existed

This is the part I want future-me to reread before casually deleting something from a compose file.

backend existed for the API. It should not publish a public port directly. It should expose 8000 only to the Docker network and let nginx decide what the outside world can reach.

db existed because the app needed Postgres close to it. Managed RDS was not allowed, so Postgres became my responsibility. The volume mattered more than the container. Containers are replaceable; database files are not.

Note — Docker volume: a Docker-managed persistent directory mounted into a container. If you delete and recreate the Postgres container but keep the volume, the database survives. If you delete the volume, congratulations, you have invented a restore drill.

redis existed for cache, queues, rate-ish state, and pre-signed URL caching. Redis is often treated as disposable until some part of the app quietly starts depending on it. I used append-only persistence because losing cache is fine, but losing every transient app state after a restart can become annoying fast.

nginx existed because public traffic should not hit app servers directly. nginx handles TLS, upload size, compression, static assets, proxy headers, admin route guards, and long timeouts. It is boring in the exact way production needs.

certbot existed because certificates expire. It is easy to get HTTPS working once. The real question is whether it will still work 91 days later when you are not thinking about it.

Note — health check: a command that tells the platform whether a service is alive. For an API, this is usually curl /health. For Postgres, pg_isready. For Redis, redis-cli ping. Health checks are not decoration; they are how your deployment decides whether to route traffic, restart services, or wait before starting dependents.

The most useful Docker command was not up. It was this:

docker compose psCopy

Because it answered the question I actually had during deploy: "What does the machine think is running?"

nginx became the load balancer I did not have

Without an ALB, nginx became the front door: TLS termination, static files, API proxy, admin route protection, upload limits, long request timeouts, and health checks.

The app had image and video generation paths, so default proxy timeouts were a trap. A "normal" 60 second timeout looks fine until a user kicks off a video workflow and nginx gives up while the backend is still doing exactly what it was asked to do.

worker_processes auto;
worker_rlimit_nofile 1000000;

events {
    worker_connections 65536;
    multi_accept on;
}

http {
    sendfile on;
    tcp_nopush on;
    tcp_nodelay on;

    keepalive_timeout 300s;
    client_header_timeout 300s;
    client_body_timeout 300s;
    send_timeout 300s;
    client_max_body_size 200M;

    upstream backend_upstream {
        server backend:8000 fail_timeout=5s;
        keepalive 16;
    }

    server {
        listen 80;
        server_name example.com www.example.com;

        location = /health {
            proxy_pass http://backend_upstream/health;
        }

        location /api/ {
            proxy_pass http://backend_upstream/api/;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
            proxy_read_timeout 300s;
            proxy_send_timeout 300s;
        }
    }
}Copy

Note — reverse proxy: nginx is not running the app. It receives the public request, decides where it should go, forwards it to an internal service, and streams the response back. In cloud terms, it is doing a slice of what an ALB, CDN, TLS manager, and routing layer might normally do together.

This deployment was not painless, but it was legible. docker compose ps showed the state. docker compose logs -f backend showed the app. docker compose restart backend did what it said. I could rebuild one service without touching the others.

That is the good version of on-prem.

My Docker deploy runbook

This is the version I wish I had written before doing it the first time.

# 1. SSH in and go to the repo
ssh app@example-server
cd /srv/app

# 2. Pull code
git pull

# 3. Check env before touching containers
test -f backend/.env
grep -E "DATABASE_URL|REDIS_URL|FRONTEND_URL|BACKEND_URL" backend/.env

# 4. Build images
docker compose build backend frontend

# 5. Start dependencies first if needed
docker compose up -d db redis
docker compose ps

# 6. Apply migrations if the app has them
docker compose run --rm backend alembic upgrade head

# 7. Start app + edge
docker compose up -d backend frontend nginx

# 8. Watch logs while hitting the site
docker compose logs -f backend nginxCopy

The important part is not memorizing these exact commands. It is the order: get the code, confirm config, build, bring up stateful dependencies, migrate, start app services, then watch logs during real requests.

If I skip the config check, I eventually pay for it in the logs.

The preflight checks I now do

Before calling any on-prem deploy "done", I now check boring things out loud:

df -h
free -h
uptime
docker system df
docker compose psCopy

Note — df -h: shows disk usage. Disk-full bugs are some of the least glamorous production bugs. Postgres cannot write, Docker cannot pull layers, nginx cannot write logs, and suddenly everything looks broken in unrelated ways.

Then app checks:

curl -i http://127.0.0.1:8000/health
curl -i https://example.com/health
curl -i https://example.com/api/v1/bootstrapCopy

Then user-flow checks:

home page loads
login works
admin route behaves correctly
one media upload/generation works
one payment/webhook route is reachable
logs are not screamingCopy

This sounds obvious. It is only obvious after you have deployed something that looked healthy until the first real user clicked the one route you did not test.

The deployment I got next: no sudo

The second server looked similar from far away and completely different from the shell.

No Docker.

No systemd unit files.

No nginx edits.

No installing packages into /usr/bin.

No "just open this port".

I had a user account, a project directory, Node, Python, and enough permissions to run processes as myself. Which sounds bad until you realize that a web app is, at the end of the day, a few long-running processes and some files. If I could not make the machine into a platform, I could make a tiny platform inside my home directory.

That meant PM2.

Note — PM2: PM2 is a Node.js process manager, but it can run any executable. It is not a replacement for Docker or systemd, but when you only have user-level permissions, it gives you process naming, restarts, logs, environment variables, and startup resurrection if the host has been configured for it.

The frontend was a Vite app:

{
  "scripts": {
    "dev": "vite --port 5174",
    "build": "tsc -b && vite build",
    "preview": "vite preview"
  }
}Copy

For production, I did not want to run vite dev. That is a dev server. So the pattern became:

cd ~/app/frontend
npm ci
npm run build
npx serve -s dist -l 127.0.0.1:5174Copy

The backend was FastAPI:

cd ~/app/backend
python3 -m venv .venv
. .venv/bin/activate
pip install -r requirements.txt
uvicorn app.main:app --host 127.0.0.1 --port 8000Copy

Running those commands manually proves the app can start. It does not make a deployment. A deployment has to survive SSH disconnects, crashes, restarts, and "wait, which terminal did I run that in?"

So PM2 became the wrapper:

module.exports = {
  apps: [
    {
      name: "app-api",
      cwd: "/home/app/app/backend",
      script: ".venv/bin/uvicorn",
      args: "app.main:app --host 127.0.0.1 --port 8000",
      interpreter: "none",
      env: {
        ENVIRONMENT: "production",
        PYTHONUNBUFFERED: "1"
      },
      max_memory_restart: "900M",
      autorestart: true,
      watch: false
    },
    {
      name: "app-web",
      cwd: "/home/app/app/frontend",
      script: "npx",
      args: "serve -s dist -l 127.0.0.1:5174",
      interpreter: "none",
      env: {
        NODE_ENV: "production"
      },
      autorestart: true,
      watch: false
    }
  ]
};Copy

Then deploys became boring enough to repeat:

git pull

cd backend
. .venv/bin/activate
pip install -r requirements.txt

cd ../frontend
npm ci
npm run build

cd ..
pm2 startOrReload ecosystem.config.cjs --update-env
pm2 save
pm2 statusCopy

Not as clean as Docker. Not as correct as systemd. But it worked inside the permissions I actually had, which is the only deployment environment that matters.

PM2 terms I keep forgetting

I keep having to relearn PM2 because I do not use it every week, so here is the tiny glossary.

Process name: the human label, like app-api. Use names. Do not rely on process IDs.

cwd: the working directory. This matters because relative imports, .env files, and frontend dist paths often assume a current directory.

script: the executable PM2 runs. For Python, this can be .venv/bin/uvicorn. For frontend static serving, it can be npx.

args: everything after the command.

interpreter: "none": tells PM2 not to wrap the command in Node. Without this, PM2 may try to treat non-JS commands like JavaScript and you get nonsense.

--update-env: tells PM2 to reload environment variables when restarting. Without this, you can change .env or shell vars and PM2 may keep the old world alive.

pm2 save: writes the current process list so it can be resurrected later. If the host has PM2 startup configured, this is what makes your current app list survive reboot.

The commands I actually use:

pm2 status
pm2 logs app-api --lines 100
pm2 restart app-api --update-env
pm2 restart app-web --update-env
pm2 delete app-api
pm2 save
pm2 monitCopy

Note — pm2 monit: a terminal dashboard showing CPU, memory, restarts, and logs for PM2 processes. It is not observability in the fancy sense, but when you have no CloudWatch, it is a small lantern.

The no-sudo deploy runbook

This is the version for "I have a shell account and I need to be useful."

# 1. SSH in
ssh app@example-server
cd ~/app

# 2. Pull latest code
git pull

# 3. Backend dependencies
cd backend
python3 -m venv .venv
. .venv/bin/activate
pip install --upgrade pip
pip install -r requirements.txt

# 4. Smoke start backend before PM2
uvicorn app.main:app --host 127.0.0.1 --port 8000
# Ctrl-C if it starts cleanly

# 5. Frontend build
cd ../frontend
npm ci
VITE_API_BASE_URL=https://api.example.com/api/v1 npm run build

# 6. Reload managed processes
cd ..
pm2 startOrReload ecosystem.config.cjs --update-env
pm2 save
pm2 status

# 7. Warm and verify
curl -i http://127.0.0.1:8000/health
pm2 logs app-api --lines 80Copy

The key move is starting the backend manually once before letting PM2 supervise it. If the app cannot start in the foreground, PM2 will not fix it. PM2 will just restart the same broken command very enthusiastically.

The weird bugs were all boundaries

The code was rarely the hard part. The hard part was boundaries.

The frontend needed the right API base URL at build time. With Vite, environment variables are baked into the build. Change VITE_API_BASE_URL after npm run build, and the already-built JS does not care. You need to rebuild.

VITE_API_BASE_URL=https://api.example.com/api/v1 npm run buildCopy

The backend needed to bind privately. If nginx or the hosting panel is the public entrypoint, the app should not casually bind to 0.0.0.0 unless you mean to expose it. On the PM2 server I preferred loopback:

uvicorn app.main:app --host 127.0.0.1 --port 8000Copy

Warmup mattered. FastAPI startup was intentionally lightweight, but the first real request still had to pay for DB connections, Redis connections, config cache, GeoIP reader load, and whatever else had been lazily initialized. So I kept a tiny post-deploy warmup script:

#!/usr/bin/env bash
set -euo pipefail

BASE_URL=${1:-http://127.0.0.1:8000}

for i in 1 2 3 4 5; do
  echo "warmup attempt $i"
  curl -fsS -w "HTTP %{http_code} - %{time_total}s\n" \
    "${BASE_URL}/bootstrap" -o /dev/null || true
  sleep 0.2
doneCopy

This is not magic. It just moves the first-request pain from an unlucky user to me, immediately after deploy, while I am still watching logs.

Logs had to be part of the workflow. In ECS I would check CloudWatch. Here it was:

pm2 logs app-api
pm2 logs app-web
pm2 monitCopy

And for Docker:

docker compose logs -f backend
docker compose logs -f nginx
docker compose psCopy

I learned to never call a deploy successful until I had tailed logs through one login, one API call, one media request, and one admin route. Otherwise you have not deployed the app. You have deployed optimism.

Things I would not forget next time

I am writing this section mostly for myself, because this is exactly the kind of operational detail I forget once the incident is over.

Do not expose app ports publicly unless you mean it. Bind internal services to 127.0.0.1 or Docker-only networks. Public traffic should go through nginx or the host's reverse proxy.

Do not run Vite dev in production. vite --host feels convenient and then you remember it is a development server. Build static assets and serve dist, or run a real SSR server if the app has one.

Do not treat .env as a mystery box. Print the non-secret shape of the environment before deploy: which DB host, which Redis URL, which frontend URL, which backend URL. Never print actual secrets into logs.

Do not assume reboot recovery. Test or at least inspect it. Docker restart policies help. PM2 needs pm2 save and usually startup integration. A process currently running is not the same as a process that will come back after a reboot.

Do not ignore upload size and timeout defaults. Media apps need larger client_max_body_size and longer proxy timeouts. Otherwise the backend can be healthy while nginx kills the request.

Do not call it done before testing the boring flows. Login, admin, media upload, payment/webhook, cache-dependent endpoint, one mobile page.

I used to think this kind of checklist was overkill. Then I had to debug production on someone else's Linux box, with someone else's permissions, while trying to remember which layer owned TLS. I am very pro-checklist now. Character development, but make it curl.

Would I do it this way again?

If sudo is available, I would use Docker again. No contest.

Docker gave me a repeatable runtime, named services, network isolation, health checks, restart policies, and clean logs. The compose file became living documentation. Six months later, someone can read it and understand what the machine is supposed to be running.

If sudo is not available, I would still use PM2 again, but with a very clear mental label: this is a survival deployment, not a platform.

PM2 will keep a process alive. It will not give you container isolation. It will not own system ports. It will not configure nginx. It will not install Postgres. It will not solve backups. But if the real constraint is "you may run userland processes and nothing else", PM2 is a perfectly respectable little raft.

The lesson I took from both deployments is not "Docker good, PM2 bad". It is that deployment strategy is mostly permission strategy.

Cloud gives you APIs. Docker gives you a local platform. Sudo gives you the machine. No sudo gives you a home directory and your manners.

And sometimes that is still enough.