CI / CD pipeline

Eleven workflows live in .github/workflows/; the six documented below are the ones you interact with. (The others are fleet plumbing — build-runner-ami, fleet-canary, fleet-queued-jobs — plus e2e-flake-hunt and mobile-drift.) Most CI jobs run on a self-hosted ephemeral EC2 spot fleet rather than GitHub-hosted runners — see Where jobs run, which explains a failure mode specific to that setup.

Overview

Where jobs run

Runner	Jobs
Self-hosted fleet ([self-hosted, greekmanage, linux-x64])	backend, frontend, frontend-coverage, security, e2e, nightly ZAP
ubuntu-latest	Fast checks, Required Checks, iac-scan, android-build, terraform-plan, docs-site, cd-production
minipc ([self-hosted, minipc, staging])	cd-staging

The fleet is ephemeral EC2 spot: one instance per job, terminated after it. Compute genuinely scales to zero between CI bursts, but an individual instance is not billed only for its job — see the floor below. Infrastructure lives in terraform/modules/runners.

Placing a job is a cost decision, and job duration dominates it. GitHub bills each job rounded up to the whole minute at $0.008/min, so a long job on ubuntu-latest is expensive out of proportion to how important it is. frontend-coverage is the worked example: at 22 minutes and ~454 runs/month it came to ~$84/mo — more than the entire workload the fleet was built to absorb — for a report that gates nothing.

On the fleet the rule inverts: short jobs carry a fixed floor, and only long ones pay for duration. minimum_running_time_in_minutes (12) is measured from the instance's launch, not from when it goes idle, and the scale-down reaper runs every 5 minutes — so effective lifetime is max(job_end, launch + 12) rounded up to the next tick.

Cold start matters to that sum. The runner only registers ~115s after launch, so job_end is about launch + 1.9 min + runtime. The floor therefore binds for any job under roughly 10 minutes of runtime, and every one of those costs the same 15 minutes of instance time ($0.019 at c7a.xlarge spot) — a 90-second job and a 9-minute one are priced identically. Past ~10 minutes, cost starts tracking runtime again.

The practical consequence: short jobs are cheap on ubuntu-latest and expensive on the fleet; long jobs are the opposite. That is why iac-scan (37s) stays on ubuntu-latest, and why needlessly triggering a short fleet job costs as much as triggering a long one.

The counterweight is spot reclaim, which abandons a job and reports it failed. That argues for keeping long, blocking jobs off the fleet — but a long non-blocking job is the ideal fleet candidate, because a reclaim costs only a re-run of something nothing depends on.

:::warning A stalled fleet does not fail — it hangs

A job pinned to a self-hosted label does not fall back to GitHub-hosted. If no runner picks it up, the job queues indefinitely rather than failing, and timeout-minutes does not begin counting until a job is picked up. GitHub does not cancel a queued job for roughly 24 hours.

This used to be mergeable-while-untested: Fast checks was the only required status check, so a stalled fleet left a PR showing its one required check green with the tests never having run. The Required Checks aggregator (see Required checks) closed that — it needs: every fleet job, so a queued dependency keeps the aggregator queued too, and the PR is not mergeable rather than green-and-untested.

Three things guard against a stalled fleet:

1. Required Checks and Fast checks both stay on ubuntu-latest permanently. A job pinned to a self-hosted label would queue forever, and the required check is the one job whose silence blocks every PR in the repo.

2. The aggregator turns the hang into a blocked merge instead of a false green.

3. fleet-queued-jobs.yml sweeps every 2h for jobs queued past STALE_MINUTES (75 — longer than job_retry's five attempts), and opens or updates an issue labelled fleet-stranded-job, closing it automatically once nothing is stranded. It runs on ubuntu-latest so it works precisely when the fleet is what is broken, and it is GitHub-side only — no AWS credentials, no coupling to the runner module's SQS schema.

   Guard 3 was inert from its introduction (#904) until v0.73.25: it has no actions/checkout by design, and gh issue / gh label resolve the repo from git remotes rather than taking it in the URL the way gh api does, so every run died at the first gh issue list and the alert was never once filed. It now sets GH_REPO. If you are adding an issue-filing step to any checkout-less workflow, set GH_REPO: ${{ github.repository }} or use actions/github-script (as security-nightly.yml does).

:::danger The CloudWatch alarms do not cover this The webhook queue-depth alarm was previously listed here as the third guard. It is not one. #787 established that when instances launch but never register, every alarm stays OK: scale-up succeeds and logs a created instance, SQS drains so -runner-queue-backlog never fires, the DLQ stays empty, and gh api .../actions/runners reports offline JIT registration stubs that make total_count actively misleading. No user_data or runner log stream is produced at all, because the CloudWatch agent installs after the apt phase that is failing — that absence is itself the diagnostic.

So the aggregator and the fleet-queued-jobs sweep are the mechanisms that surface this failure mode — as an unmergeable PR and a filed issue respectively, never as a page, and the sweep only after a job has been stranded past 75 minutes. Use aws ec2 get-console-output --instance-id <id> and read the user-data: lines around cloud-init modules:final; StateReason: Client.UserInitiatedShutdown means the scale-down reaper, not a spot interruption. :::

If you see a PR that touches backend/ with no backend job, that is the symptom. :::

:::tip Debugging fleet problems Use EC2 console output and the scale-up / scale-down Lambda logs. The Actions UI is blind to all of it — instance-level failures present as a job that simply stops, with no failed step and no error message.

aws ec2 get-console-output --instance-id <id> --query Output --output text
aws logs tail /aws/lambda/greekmanage-scale-up  --since 15m
aws logs tail /aws/lambda/greekmanage-scale-down --since 15m

Instances are ephemeral — capture console output before it ages out. :::

ci-v2.yml — the main CI gate

Trigger: push to main, or PR targeting main. Concurrency is grouped per PR/ref with cancel-in-progress.

Jobs, in dependency order:

1. Fast checks (job id changes) — three cheap always-run steps merged into one job, because GitHub bills a one-minute minimum per job:

   • dorny/paths-filter — decides which downstream jobs run, and which Django apps changed for scoped testing
   • Gitleaks — secret scan of the working tree (--no-git), honouring .gitleaks.toml
   • scripts/check-docs.sh — verifies CLAUDE.md ↔ README ↔ CHANGELOG agree

   The path filter runs first so its outputs still exist if Gitleaks later fails the job.

2. backend — python manage.py test (Django, not pytest), scoped to changed apps when the filter identified them, otherwise the full suite. Postgres (pgvector) and Redis run as service containers.

3. frontend — npx vitest run, npm run build, then npm run knip for orphan/dead-code detection (blocking).

4. android-build — npx cap sync android + ./gradlew assembleDebug, with a dummy google-services.json generated for CI. Push-to-main only, not on PRs: that cuts roughly 454 runs a month to about 60 while still catching a frontend change that breaks the Capacitor build. The tradeoff is that a break is caught one merge later.

5. security — Bandit (Python SAST), Semgrep (custom rules in .semgrep.yml), pip-audit, npm audit (against both frontend/ and docs-site/) and a Trivy filesystem scan. Blocking on HIGH/CRITICAL.

   Unconditional — it has no if:. It used to be gated on backend == 'true' || frontend == 'true', which meant a terraform-only, env/-only, scripts/-only or docs-site-only PR skipped every SAST and dependency scan while the docs claimed security scans could not be skipped. The docs-site audit is here as well as in docs-site.yml: that workflow is path-filtered to docs-site/**, so its audit only ran on PRs that already touched the docs site. npm audit resolves from package-lock.json alone, so the extra step needs no npm ci.

6. e2e — Playwright. Runs migrations and seed data, starts Django plus a Vite preview server, then a @smoke fast-fail gate followed by the full suite (~230 tests). Traces and the HTML report upload on failure.

   DAST is deliberately not on the PR critical path — ZAP runs nightly against main, and locally via npm run security:full before opening a PR.

7. Required Checks (job id required-checks) — the aggregator that branch protection requires. if: always(), needs: every job above, and fails if any dependency reported failure or cancelled. See Required checks.

security-nightly.yml — nightly DAST

Trigger: cron 0 6 * * * (06:00 UTC / 02:00 ET), plus manual dispatch — with no inputs. A full/baseline choice used to sit there and nothing in the job ever read it, so picking "baseline" silently ran the full scan anyway; it was removed. Baseline scanning lives in scripts/security-scan.sh.

Builds a stack on a fleet runner, seeds it, and runs a full ZAP active scan (considerably more aggressive than a baseline passive scan) against the API and frontend. Opens a GitHub issue on failure.

This is also the sanctioned fallback for npm run security:full when a workstation cannot run it (CLAUDE.md). Dispatch it against your branch:

gh workflow run security-nightly.yml --ref <your-branch>
gh run view <id> --log | grep -E "FAIL-NEW: [0-9]"   # two lines, both must be 0

:::warning Read what it scanned, not just the conclusion Two things make a green here mean something:

• The Verify schema endpoint is reachable step must log /api/schema/ returned 200 — that guard exists because a scan against an unreachable surface passes while covering nothing.
• Both halves must report FAIL-NEW: 0. WARN counts differ from a local security:full run and that is expected, not a regression: CI serves the SPA through vite preview, which sends no security headers, while local and deployed serve it through nginx, which does. Do not compare WARN counts across harnesses.

It also failed for a period at Run migrations and seed data with ImproperlyConfigured: ENCRYPTION_KEY must be set in settings — its env block never had one, and seed_e2e_data writes EncryptedTextFields. It died before ZAP started, so the scan covered nothing while the job went red for an unrelated-looking reason. :::

terraform-plan.yml — infrastructure diff

Trigger: PR touching terraform/**.

Assumes a read-only AWS role via OIDC (AWS_PLAN_ROLE_ARN), runs terraform plan, and surfaces the diff on the PR.

note

The plan job must run scripts/download-runner-lambdas.sh before terraform init. The runner-fleet module's Lambda bundles are not shipped in the Terraform Registry package and are gitignored, so without them plan fails with Call to function "filebase64sha256" failed — an error that names neither the file nor the cause.

cd-staging.yml — deploy to staging

Trigger: push to main, or manual dispatch with a ref input.

Runs on the minipc self-hosted runner, targets the staging environment, and deploys to https://dev.greekmanage.com. Playwright smoke tests run against the live site afterwards.

Staging supports pinning: dispatching with a ref other than main pins staging to it, so subsequent pushes to main will not overwrite it. Deploy main to unpin.

cd-production.yml — deploy to production

Trigger: manual dispatch only. There is no automatic path to production.

Targets the production environment and takes an optional image_tag input (defaults to the commit SHA). Authenticates to AWS via OIDC (AWS_DEPLOY_ROLE_ARN) — no long-lived AWS keys.

• deploy-backend — builds and pushes the backend image to ECR, runs migrations as a one-off ECS task, then updates the ECS service. If the tag already exists in ECR it is reused rather than rebuilt: the repository is IMMUTABLE, so until v0.68.4 any failure after the push made that commit permanently un-retryable — every re-run died at the push, before reaching the step that had actually failed.
• deploy-frontend — builds the frontend, aws s3 sync --delete to the origin bucket, then a CloudFront invalidation on /*.

docs-site.yml — documentation site

Trigger: push to main or PR touching docs-site/** or the workflow itself, plus manual dispatch.

Builds the Docusaurus site, audits its dependencies for HIGH/CRITICAL CVEs, and regenerates the API reference from the OpenAPI schema. On push to main it deploys to GitHub Pages.

:::note Two audits of the same tree, on purpose ci-v2.yml's security job audits docs-site/ on every PR — this workflow is path-filtered to docs-site/**, so its own audit only runs on PRs that already touch the docs site, which is how a HIGH advisory could sit undetected there.

The copy here is still the only thing that can stop a publish. deploy needs: build and nothing else; the two workflows are not coupled, so a red security job in ci-v2.yml does not block a Pages deploy. The CI audit gates a merge, this one gates the artifact. :::

Required checks

Required Checks is the only required status check on main — and it is an aggregator, not a job that tests anything itself.

Check	Blocks merge?	How
Required Checks (aggregator)	✅ Required context	The one context branch protection names
Fast checks (path filter + Gitleaks + doc consistency)	✅	via the aggregator
Backend tests	✅	via the aggregator
Backend Docker build	✅	via the aggregator
Frontend tests / build / Knip	✅	via the aggregator
Android build	✅	via the aggregator (push-to-main only; skipped on PRs)
Bandit / Semgrep / pip-audit / npm audit / Trivy	✅	via the aggregator
IaC Scan (Trivy config + Checkov)	✅	via the aggregator
E2E	✅	via the aggregator
Frontend Coverage Report	❌	continue-on-error by design — deliberately not a dependency
Terraform Plan	❌	separate workflow, informational diff
Nightly full ZAP	❌	async — failures open an issue
Mobile Release Drift	❌	weekly — breaches open an issue; red means it could not measure

Why an aggregator rather than eight required contexts

Requiring those contexts directly does not work, and the failure is silent and total: the path-filtered jobs are conditional, and a job skipped by a job-level if: posts no status at all. GitHub then holds the PR at "Expected — waiting for status to be reported" indefinitely. Requiring backend would make every docs-only PR permanently unmergeable; requiring android-build, which never runs on pull_request, would deadlock every PR without exception.

So the required job has to be one that always reports, and one that can read needs.*.result to tell "skipped because it was irrelevant" apart from "failed":

required-checks:
  name: Required Checks
  if: always()          # must post a status even when a dependency failed
  needs: [changes, backend, backend-docker-build, frontend,
          android-build, security, iac-scan, e2e]
  runs-on: ubuntu-latest

The result mapping is: success → pass, skipped → pass, failure → fail, cancelled → fail. A job that never reports at all (a stalled fleet) leaves the aggregator itself queued, so the PR is not mergeable.

:::warning Adding a new gate job means adding it to needs: Otherwise it silently stops blocking merges. A step inside the aggregator diffs the job list in ci-v2.yml against its own needs: and fails if one was forgotten — add the job to needs:, or to that step's EXEMPT list with a reason. :::

:::note Required contexts are matched by display name name: Required Checks, not the job id required-checks. Renaming the job breaks branch protection and blocks every PR until the protection rule is updated in the same change — the same trap that the changes → Fast checks merge hit. :::

Caching

Cache	Keyed by
pip (cache: pip)	backend/requirements.txt
npm (cache: npm)	frontend/package-lock.json
Playwright browsers	package-lock.json, plus a selfhosted discriminator

:::warning Cache keys must distinguish runner types actions/cache archives paths relative to the workspace, using ../ traversal for anything outside it — and runner.os is Linux on both GitHub-hosted and self-hosted runners.

That means a cache saved by a GitHub-hosted runner is a key match on the fleet, but replays the directory depth it was created with:

hosted   ~ = /home/runner    workspace /home/runner/work/repo/repo
         ../../../ → /home/runner            ✅
fleet    ~ = /home/ubuntu    workspace /opt/actions-runner/_work/greekmanage/greekmanage
         ../../../ → /opt/actions-runner     ❌

The Playwright browsers unpacked to /opt/actions-runner/home/runner/.cache/... while Playwright looked in /home/ubuntu/.cache/..., and every test failed — after the step reported Cache restored successfully and cache-hit == true.

If you add a cache for a path outside the workspace, put a discriminator in the key, and never gate an install step on cache-hit alone. :::

Secrets

Secret	Used by
AWS_DEPLOY_ROLE_ARN	cd-production — OIDC role assumption
AWS_PLAN_ROLE_ARN	terraform-plan — read-only OIDC role
E2E_ADMIN_EMAIL / E2E_ADMIN_PASSWORD	E2E fixtures
E2E_MEMBER_EMAIL / E2E_MEMBER_PASSWORD	E2E fixtures
GITHUB_TOKEN	Auto-provided; issue creation in the nightly scan

Deployment uses OIDC role assumption, not stored AWS keys. The backend's runtime secrets live in SSM Parameter Store encrypted with a customer-managed KMS key, not in GitHub.

The ENCRYPTION_KEY used by the backend and E2E jobs is generated per run rather than read from a secret, so Dependabot and fork PRs — which cannot access repository secrets — still run. It only ever encrypts disposable test data.

Local CI parity

Full security suite before opening a PR:

npm run security:full

Bandit, Semgrep, pip-audit, npm audit, Trivy, Gitleaks, and a credentialed ZAP full active scan (15–45 min). It builds, seeds, scans and destroys its own ephemeral local stack from your working tree, so it scans the branch you are about to open a PR for. Reports land in security-reports/.

Exit codes are three-way, matching scripts/iac-scan.sh: 0 clean, 1 HIGH/CRITICAL findings, 2 inconclusive — a scan that did not run, whose coverage is therefore unknown. Treat 2 as a tooling failure to fix, never as something to suppress.

Read the banner on the last lines rather than only the status: piping the run (… 2>&1 | tail) exits with tail's status, so the in-band PASSED / FAILED / INCONCLUSIVE verdict is the reliable signal.

The compose project is derived from the checkout path, so scans from separate worktrees coexist; a second run in the same checkout refuses instead of destroying the first one's stack mid-scan.

Backend tests (run inside Docker):

echo "yes" | docker compose exec -T backend python manage.py test

E2E:

npx playwright test

mobile-drift.yml — released-vs-main drift

Trigger: cron 0 9 * * 1 (Mondays 09:00 UTC), plus manual dispatch.

Compares main against the last released mobile build recorded in frontend/mobile-release.json and opens (or updates) one mobile-drift issue when the gap crosses a threshold — 10 commits touching frontend/src/**, or 21 days with any drift, or a platform with nothing released at all.

The workflow exits 0 on breach and carries the signal in the issue. Red is reserved for "could not measure": a missing, unparseable or stale marker fails the job, because an undeterminable shipped version is a failure and never a pass.

Decision logic lives in scripts/ci/mobile-drift.js with node:test cases; the workflow only calls scripts/ci/mobile-drift-run.js. See Uploading is not releasing for why published and not uploadedAt.

Adding a new workflow

1. Create .github/workflows/<name>.yml.
2. Use path filtering to avoid running on unrelated changes — and make sure the paths you list actually exist. All four ci-v2.yml filters once referenced .github/workflows/ci.yml after the file was renamed to ci-v2.yml, so editing CI triggered none of the jobs it configures. An unmatched path fails silently; nothing warns you.
3. Decide where it runs. Default to ubuntu-latest; use the fleet for anything heavy, and re-read Where jobs run first.
4. Do not build multi-line output inside a run: | block. A continuation line at column 0 terminates the YAML block scalar and makes the whole file unparseable. mobile-drift.yml shipped that way and produced 33 runs, 33 startup_failures, 0 jobs executed and 0 issues opened before anyone noticed — because a startup failure posts no status check, blocks nothing, is not a required context, and reports its name as the file path. Put logic in scripts/ci/*.js with node:test cases instead; scripts/ci/workflows-parse.test.js (run by Fast checks) now fails on that shape.
5. Give it what the app needs to boot. security-nightly.yml ran without ENCRYPTION_KEY and died at the seed step, so its scan covered nothing. Copy ci-v2.yml's Generate ephemeral encryption key step for any job that writes to the database.
6. Document it here.

When CI is red

Failure	Common cause	Fix
Backend tests fail	Migration missing from the PR	Add the migration
Frontend build fails	TypeScript error	npm run typecheck locally
Knip fails	Unused export or orphan file	Remove it, or justify in the PR
Android build fails	Capacitor sync broken	npx cap sync android locally
Semgrep finding	New code matches a rule	Fix it, or # nosemgrep: <rule-id> # reason
pip-audit / npm audit	New CVE in a dependency	Update, or pin with justification
ZAP HIGH	Endpoint without auth, or missing CSRF	Audit the endpoint
check-docs.sh fails	CLAUDE.md / README / CHANGELOG disagree	./scripts/bump-version.sh
Job never starts, no error	Fleet is not launching runners	Check EC2 console output and the scale-up Lambda logs — not the Actions UI

Related

• Testing
• Deployment
• Contributing