Skip to main content

Deployment

GreekManage ships in three environments, each defined by a different set of files.

EnvDefined inPurpose
Shared infradocker-compose.ymlredis + gotenberg only — depended on by the two stacks below
E2Edocker-compose.e2e.ymlIsolated stack for Playwright runs; the only full app stack in the repo
Stagingdocker-compose.staging.yml + Cloudflare TunnelProduction-like, pre-prod gate
Productionk8s/ manifestsCustomer-facing

The LAN dev stack was retired in v0.66.16 — staging at dev.greekmanage.com is the single non-production environment. docker-compose.minipc.yml and nginx-ssl.conf were removed in v0.73.50 as dead config left over from it.

Production topology​

Manifests​

All in k8s/ (manifests grouped by component into subdirs):

PathResource
namespace.yamlgreekmanage namespace
postgres/deployment.yaml + postgres/pvc.yaml + postgres/service.yamlPostgres 18 + pgvector (digest-pinned), 10 Gi PVC, ClusterIP service
redis/deployment.yaml + redis/service.yamlRedis 8 alpine (digest-pinned, emptyDir — ephemeral cache only)
backend/deployment.yaml + backend/service.yamlDjango + Daphne, 2 replicas, read-only rootfs (writable /tmp emptyDir)
frontend/deployment.yaml + frontend/service.yamlNginx + static SPA, 2 replicas, read-only rootfs
gotenberg/deployment.yaml + gotenberg/service.yamlGotenberg doc→PDF sidecar, read-only rootfs (writable /tmp emptyDir)
ingress.yamlnginx-ingress with cert-manager annotations
network-policies.yamlDefault-deny + per-pod allow rules (Pod-to-pod traffic restriction)

Non-secret env (allowed hosts, log level) and secrets (DB password, JWT keys, Anthropic key, S3 creds) are templated by your own ConfigMap/Secret manifests. The repo's manifests don't ship a stub configmap.yaml / secret.yaml — use sealed-secrets or external-secrets in production.

Pod hardening baseline​

Every pod runs runAsNonRoot: true with seccompProfile: RuntimeDefault, and every container sets allowPrivilegeEscalation: false and capabilities.drop: [ALL]. All app pods run a read-only root filesystem — backend and Gotenberg mount a small emptyDir at /tmp for scratch writes (Python tempfile, gunicorn worker temp, reportlab/Pillow, LibreOffice conversions); the backend also sets TMPDIR=/tmp. Postgres keeps a writable rootfs (its data dir is a PVC). After changing the backend securityContext, roll it out and smoke-test a write path (a certificate/tax-receipt PDF export and a media upload) to confirm nothing needs a writable location outside /tmp.

Resource limits​

PodCPU reqCPU limMem reqMem lim
backend200m500m256Mi512Mi
frontend100m250m128Mi256Mi
postgres200m500m256Mi512Mi
redis100m250m128Mi256Mi

Tune for your tenant size; defaults work for a single national org of ~200 chapters.

Probes​

readinessProbe:
  httpGet: { path: /api/health/, port: 8000 }
  initialDelaySeconds: 10
  periodSeconds: 5

livenessProbe:
  httpGet: { path: /api/livez/, port: 8000 }
  initialDelaySeconds: 15
  periodSeconds: 10

Two endpoints, deliberately. /api/health/ returns 200 only if DB and Redis are reachable, so it belongs on readiness — it removes a pod from rotation. /api/livez/ returns 200 whenever the process is serving, with no dependency checks, so a Postgres failover or Redis restart does not restart every pod at once. Pointing liveness at the dependency-gated endpoint converts a brief blip into a full outage.

Production runs on ECS Fargate, where the ALB target group is the live copy of this decision — see Observability → Health checks. The manifests above are legacy reference and nothing deploys from them.

Network policies​

  • Backend: ingress from frontend + celery; egress to postgres + redis + external HTTPS (LLM, payments, email)
  • Postgres: ingress from backend + celery only
  • Redis: ingress from backend + celery + beat only
  • Frontend: ingress from world (via ingress controller); egress to backend + DNS

Apply​

kubectl apply -f k8s/namespace.yaml
kubectl apply -f k8s/postgres-pvc.yaml
kubectl apply -f k8s/postgres-deployment.yaml
kubectl apply -f k8s/postgres-service.yaml
kubectl apply -f k8s/redis-deployment.yaml
kubectl apply -f k8s/redis-service.yaml
kubectl apply -f k8s/configmap.yaml
kubectl apply -f k8s/secret.yaml
kubectl apply -f k8s/backend-deployment.yaml
kubectl apply -f k8s/backend-service.yaml
kubectl apply -f k8s/frontend-deployment.yaml
kubectl apply -f k8s/frontend-service.yaml
kubectl apply -f k8s/ingress.yaml
kubectl apply -f k8s/network-policies.yaml

For repeated deploys, wrap in a Helm chart or Kustomize overlay (not yet checked into the repo — open issue #TBD).

Migration on deploy​

Run as a one-off Job before pod rollout:

apiVersion: batch/v1
kind: Job
metadata:
  name: migrate
  namespace: greekmanage
spec:
  template:
    spec:
      containers:
      - name: migrate
        image: greekmanage-backend:0.62.1
        command: ["python", "manage.py", "migrate", "--noinput"]
        envFrom:
        - configMapRef: { name: greekmanage-config }
        - secretRef: { name: greekmanage-secrets }
      restartPolicy: OnFailure

Apply, wait, then patch the backend Deployment to the new image tag.

Image building​

All Docker base images are digest-pinned — pinning the SHA prevents silent base-image drift and is enforced by npm run security:full (Trivy scans the resolved digest, not the floating tag).

Backend (backend/Dockerfile):

FROM python:3.13-slim-trixie@sha256:…
RUN apt-get update && apt-get install -y --no-install-recommends \
      gcc=4:14.2.0-1 libpq-dev=17.10-0+deb13u1 xmlsec1=1.2.41-1+b1 \
      pkg-config=1.8.1-4 libjpeg-dev=1:2.1.5-4 zlib1g-dev=1:1.3.dfsg+really1.3.1-1+b1 \
      libwebp-dev=1.5.0-0.1 libheif1=1.19.8-1 libmagic1t64=1:5.46-5 \
      ca-certificates=20250419 curl=8.14.1-2+deb13u4 gnupg=2.4.7-21+deb13u1 \
      postgresql-client-18=18.4-1.pgdg13+1
COPY requirements.txt .
RUN pip install --no-cache-dir --require-hashes -r requirements.txt
COPY . /app
WORKDIR /app
CMD ["gunicorn", "greekmanage.wsgi", "-b", "0.0.0.0:8000"]

postgresql-client-18 ships in the backend image so pg_dump versions match the Postgres 18 server (mismatched client/server pg_dump refuses to run).

The GIT_SHA build arg​

The Dockerfile's last instruction bakes the commit the image was built from:

ARG GIT_SHA=""
ENV GIT_SHA=${GIT_SHA}

It is published on /api/health/ as build — see Observability → the deploy-verification marker. Three things about it are deliberate:

  • It is last. The value changes on every commit and an ARG/ENV invalidates every layer below it, so placing it anywhere earlier would defeat the pip and apt layer caches on every single build.
  • It defaults to empty, so an ad-hoc docker build . still works; the backend reports "unknown" rather than failing to boot.
  • Callers pass the real commit, never the image tag. cd-production.yml uses github.sha, not its image_tag input — that input is free text, and one release shipped as 5ddcfe99-retry1 to get around ECR tag immutability. The staging scripts pass STAGING_IMAGE_TAG, which is a git rev-parse --short of the deployed ref.

Verify an image before deploying it:

docker run --rm <image> printenv GIT_SHA

PowerPoint → PDF conversion for learning documents no longer runs LibreOffice inside the backend image. It is delegated to a Gotenberg sidecar (gotenberg/gotenberg:8, digest-pinned) — a gotenberg service in Docker Compose and a k8s/gotenberg/ Deployment + Service. The backend/Celery worker POSTs to it at settings.GOTENBERG_URL (default http://gotenberg:3000); a NetworkPolicy restricts ingress to the backend and celery pods. This keeps ~1.5 GB of LibreOffice and its CVE surface out of the backend image.

Reproducible backend dependencies (#454)​

backend/requirements.txt is not hand-edited. It is the fully-resolved, hash-pinned output of pip-tools' pip-compile, generated from backend/requirements.in (the top-level pins — edit this one). pip install --require-hashes in the Dockerfile then refuses to install anything whose downloaded artifact doesn't match a hash recorded at compile time, closing the gap where two builds from the same commit could silently resolve different transitive versions, or a compromised/yanked package could slip in unnoticed.

To add, remove, or re-pin a top-level dependency:

  1. Edit backend/requirements.in (never requirements.txt directly).
  2. Regenerate the lockfile using the same Python version as the Dockerfile's base image, so the resolved wheels match what the image will actually install — run pip-compile inside that image rather than your host Python: 
    docker run --rm -v "$PWD/backend:/app" -w /app \
      python:3.13-slim-trixie@sha256:ffb752e139c0a19692a43af8d8523b274222dd68eebad5d583b45c2201c6e30a \
      bash -c "pip install pip-tools && pip-compile --generate-hashes --output-file=requirements.txt requirements.in"
    (Match the digest to whatever backend/Dockerfile's FROM line currently pins — the two must stay in sync.)
  3. Commit the regenerated requirements.txt alongside the requirements.in change.
  4. Verify: build the image twice (docker build --no-cache) and confirm docker run --rm <image> pip freeze is byte-identical between the two — that's the actual reproducibility guarantee, not just "the build succeeded once."

Constraints pip-compile must keep satisfied: every resolved version has to stay inside requirements.in's existing ranges — pip-compile jumping a transitive dependency to a new major version is a sign a top-level constraint needs tightening, not something to accept silently. The security-pin block at the bottom of requirements.in (urllib3, ujson, twisted, idna) holds floors for specific CVEs; don't loosen those without checking the CVE is actually fixed at the new floor.

Re-pinning the apt package versions above: run the apt-get install block unpinned in a container from the same base-image digest, then apt-cache policy <pkg> each package and copy back the Candidate: version. Debian's trixie/pgdg repos generally only serve the current point release, so an old pinned version can roll off the repo and the next build will fail loudly on a 404 rather than silently installing something newer — that's the intended failure mode, not a bug; re-resolve and re-pin when it happens.

Frontend (frontend/Dockerfile):

# Stage 1 — dev
FROM node:22-alpine3.23@sha256:… AS dev
WORKDIR /app
COPY package*.json ./
RUN npm ci
COPY . .
CMD ["npm", "run", "dev"]

# Stage 2 — build
FROM node:22-alpine3.23@sha256:… AS build
WORKDIR /app
COPY package*.json ./
RUN npm ci
COPY . .
ARG VITE_API_URL
ENV VITE_API_URL=$VITE_API_URL
RUN npm run build

# Stage 3 — prod
FROM nginx:1.29-alpine@sha256:… AS production
COPY --from=build /app/dist /usr/share/nginx/html
COPY nginx.conf /etc/nginx/conf.d/default.conf

Build:

docker build -t ghcr.io/<org>/greekmanage-backend:0.62.1 backend/
docker build -t ghcr.io/<org>/greekmanage-frontend:0.62.1 \
  --target production --build-arg VITE_API_URL=https://app.greekmanage.com \
  frontend/
docker push ghcr.io/<org>/greekmanage-backend:0.62.1
docker push ghcr.io/<org>/greekmanage-frontend:0.62.1

Staging environment​

docker-compose.staging.yml + Cloudflare Tunnel:

  • Separate Postgres on 5434, separate Redis DB 2
  • Production-like images (no volume mounts, frozen at build)
  • HTTPS via Cloudflare Tunnel (no public IP needed)
  • DJANGO_DEBUG=False
  • Suitable for E2E + manual QA of release candidates

Domain + TLS​

  • DNS: A record for app.<customer>.com → ingress controller's external IP / load balancer
  • TLS: cert-manager + Let's Encrypt via HTTP-01 challenge automatically
  • WebSocket support: nginx-ingress passes the /ws path through to backend Daphne

Storage configuration​

Per-org StorageConfig model points to the bucket the org uses. Most orgs use the platform-managed bucket; enterprise customers can configure their own.

→ Storage status (org admin view)

Object-storage backend (media)​

Media files (avatars, documents, photos, generated PDFs) are served through the S3 API via django-storages. The same code path drives three backends, selected by env:

BackendUSE_S3AWS_S3_ENDPOINT_URLNotes
Native AWS S3 (production)true(unset)boto3 derives the endpoint from AWS_S3_REGION_NAME.
Cloudflare R2 / MinIOtrue (or default-on when endpoint is set)the service URLCustom endpoint; MinIO may also need AWS_S3_ADDRESSING_STYLE=path.
Local filesystem (default)false / unset—Fallback when S3 is not enabled; also forced during the test suite.

USE_S3 defaults on whenever AWS_S3_ENDPOINT_URL is set, so existing MinIO/dev setups keep working unchanged. Native AWS S3 has no custom endpoint, so production sets USE_S3=true explicitly.

Production (native AWS S3) env:

USE_S3=true
AWS_STORAGE_BUCKET_NAME=greekmanage-media
AWS_S3_REGION_NAME=us-east-1     # the bucket's actual region
# AWS_S3_ENDPOINT_URL is intentionally left UNSET for native S3
AWS_QUERYSTRING_AUTH=true        # serve private objects via presigned URLs (recommended)
# Credentials are OPTIONAL — omit them to use the ECS task role / EC2 instance
# profile (preferred). Only set static keys for MinIO/R2 or non-role hosts:
# AWS_ACCESS_KEY_ID=…
# AWS_SECRET_ACCESS_KEY=…

Prefer an IAM role over static keys. When AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY are unset, boto3 resolves credentials via its provider chain — the ECS task role (or EC2 instance profile), which needs no long-lived secret. Attach a role granting only s3:GetObject, s3:PutObject, s3:DeleteObject on arn:aws:s3:::greekmanage-media/* (plus s3:ListBucket on the bucket). Also use AWS_QUERYSTRING_AUTH=true (presigned URLs, private bucket) rather than a public bucket.

Backups​

celery-backups deployment runs the backups Celery queue. Two scheduled tasks:

  • Daily: incremental DB snapshot to S3
  • Weekly: full DB snapshot to S3 (retained 1 year)

→ Backups & export (platform admin)

Disaster recovery​

  1. Backup: continuous (Postgres point-in-time) + daily snapshot + weekly archive
  2. Replication: cross-region read replica recommended (not in default manifests)
  3. Monitoring: alarms on backup failures, replication lag, certificate expiry
  4. Runbook: stored outside GreekManage (so it's accessible when the platform is down)
  5. Test: quarterly restore-to-staging exercise

Mobile app distribution​

  • Capacitor 8 wraps the same React frontend
  • Fastlane automates TestFlight + Play Store uploads from frontend/ios/ and frontend/android/
  • Native config in those directories (Firebase, etc.)
  • App icons updated separately from web branding (requires native rebuild + store re-submission)

Uploading is not releasing — and the two platforms differ​

This asymmetry is readable in the lanes, and it is what .github/workflows/mobile-drift.yml measures against:

Lane callDoes the upload reach users?
Androidupload_to_play_store with release_status: "draft"No. The build sits as a draft until a human promotes it in the Play Console.
iOSupload_to_testflight with skip_submission: trueYes, for internal testers — skip_submission skips App Store review, not TestFlight.

frontend/mobile-release.json therefore records two different things per platform:

  • sha / version / uploadedAt — the last upload, written by scripts/record-mobile-release.sh from the beta lanes.
  • published — the last build that actually reached users, written by scripts/record-mobile-publish.sh. It is null when nothing has been released. The iOS lane sets it at upload time because that channel releases on upload; the Android lane deliberately does not.

Commit whichever the scripts change, as part of the release. The drift check measures main against published, so an unrecorded release reads as ever-growing drift, and a draft recorded as released reads as zero drift forever while users sit on an older build.

:::warning The check measured uploads, and was dead for its whole life Before v0.73.20 it compared against uploadedAt, which reported the Android 0.71.11 draft — a build nobody could install — as perfectly current. It also never ran at all: its report was built by a shell string whose continuation lines sat at column 0, terminating the YAML block scalar, so GitHub could not parse the file. 33 runs, 33 startup_failures, 0 jobs, 0 issues opened.

The logic now lives in scripts/ci/mobile-drift.js with node:test cases, and scripts/ci/workflows-parse.test.js — run by Fast checks on every PR — fails on any workflow with that shape. :::

Observability​

Currently minimal — see Observability for the gap analysis and roadmap.