Observability

GreekManage's observability story today is container logs + audit logs + an in-app platform status dashboard. There's no APM, no distributed tracing, no external metrics dashboard yet. This page documents what exists and what's planned.

Current state

Concern	Tool	Where
Application logs	Container stdout (Django + Celery)	kubectl logs <pod>
Database logs	Postgres stdout	kubectl logs postgres-...
Audit log	DB table + S3 archive	AuditLog model + S3 daily archive
Platform health/status	In-app dashboard, 5-min snapshots	/platform/status, apps.platform.health
Error tracking	None	–
Metrics	None	–
Tracing	None	–
Uptime	Internal only (platform status dashboard) — no external monitor wired up	–
Security scans	OWASP ZAP	E2E + nightly

Application logging

Django + Celery write to stdout in JSON format (configurable via LOGGING in settings.py). In production:

kubectl logs -n greekmanage deploy/backend --tail=200 -f
kubectl logs -n greekmanage deploy/celery --tail=200 -f

For aggregated viewing across pods, use your cluster's log aggregator (Loki, Cloudwatch Logs, GCP Logging — depends on your platform).

Log levels

Module	Level	Why
Django	INFO	Request lifecycle, ORM warnings
Celery	INFO	Task start / complete
apps.common.middleware.AuditMiddleware	INFO	Audit log writes
apps.ai_services	INFO	LLM calls (tokens, latency)
apps.authentication.encryption	WARN	Key rotation events
Third-party libs (urllib3, etc.)	WARN	Reduce noise

Override per environment via LOG_LEVEL env var.

Audit log

The single source of truth for "who did what when":

• Every sensitive write logged automatically by AuditMiddleware
• Append-only at the DB level (app-tier user has no UPDATE/DELETE on the table)
• Daily archive to S3 (gzipped JSONL, partitioned by org / year / month / day)
• Per-org retention (default 180 days in DB; longer in S3)

→ Audit & encryption

Health checks

There are two probe endpoints, and the split is load-bearing. Both are public, unauthenticated and probe-only — neither leaks integration detail.

/api/livez/ — liveness

GET /api/livez/
200 OK
{ "status": "alive" }

Answers one question: is this process still serving requests? It touches no database, no Redis and no disk. This is what the ALB target group probes.

/api/health/ — readiness

GET /api/health/
200 OK
{ "status": "ok", "database": true, "redis": true, "build": "d040a3304987" }

Returns 503 ("status": "error") if the database or Redis check fails. Consumed by the Route 53 health check behind the greekmanage-backend-dependencies composite alarm, by the login page's "test connection" control, and by /platform/status.

build — the deploy-verification marker

build is the short git SHA of the image serving the request, and it is the only way from outside the app to tell which build an environment is running. It is emitted on the 503 response as well as the 200: identifying the build of a server that is misbehaving is the case it exists for.

The value is baked in at image-build time by the GIT_SHA Docker build ARG (backend/Dockerfile) — a running container has no .git directory to read. An image built without that arg reports "unknown", which is what a local docker build and the CI-native backend both report.

:::note Why the SHA and not the release version This endpoint is unauthenticated and reachable from the internet on both environments. A release number maps to a public CHANGELOG entry and from there to whatever is publicly known about that release; a SHA is opaque to anyone without the repository. It is also the more precise identifier — docs-only PRs ship without a version bump, and staging can be pinned to an unmerged branch, so several distinct deploys routinely share one version string.

The release version is published in the authenticated UI instead, as the build stamp in the sidebar footer (components/layout/build-stamp.tsx, reading __APP_VERSION__). The frontend needs its own marker because in production the two deploy by separate mechanisms — ECS rollout versus aws s3 sync to S3/CloudFront — so one can advance without the other.

backend/greekmanage/build_info.py holds the normalisation and the full rationale. :::

Both deploy paths assert this value rather than merely checking for a 200: scripts/deploy-staging*.sh fail the deploy when the reported build is not the tag they just built, and cd-production.yml's smoke test compares it against github.sha. A responsiveness check cannot detect a wrong-artifact deploy — a stale container answers /api/health/ exactly like a fresh one, which is how a stale :latest image served staging undetected under a green CD Staging run.

Why they must stay separate

On ECS there is no separate liveness concept to decouple from. The ALB target group is the only health signal, and because the service is registered with the load balancer, failing it does not merely drain a target — it gets the task replaced. Pointed at the dependency-gated /api/health/, a brief Postgres or Redis outage failed the check on every target simultaneously and ECS killed the whole service, turning a recoverable blip into a cold-start outage.

A task therefore stays healthy while its database is unreachable. That is intended: those requests return 5xx, which the alb-target-5xx alarm covers, and restarting the task would not have fixed them either way.

:::warning Changing the probed path touches three places, all silent A probe path has to be admitted by HealthCheckHostMiddleware (the ALB sends the task's private IP as Host, which a real ALLOWED_HOSTS rejects with 400), by SECURE_REDIRECT_EXEMPT (the ALB→task hop is plain HTTP with no X-Forwarded-Proto, so SECURE_SSL_REDIRECT answers 301), and by ModuleAccessMiddleware.EXEMPT_PREFIXES. None of them fails loudly — the check just stops returning 200 and every target drains.

All three derive from a single tuple in backend/greekmanage/probes.py. Add the path there, not to three literals. :::

:::warning Deploy ordering The ALB health-check path is Terraform; the endpoint is application code. Deploy the image first, then apply — cd-production.yml, then terraform apply. Reversed, the ALB probes a route the running image does not have and drains every target. :::

Platform status dashboard

Platform admins get a consolidated health view at /platform/status, backed by GET /api/platform/status/ (IsPlatformAdmin-gated). Unlike the bare /api/health/ probe, this aggregates:

• Core infrastructure (critical — pages platform admins in-app + by email on a healthy→down transition): database, Redis (cache + Celery broker), Celery workers, object storage/backup target. Snapshotted every 5 minutes by the snapshot-platform-health Celery Beat task into PlatformHealthSnapshot, giving a rolling 24h uptime % per component.
• Integrations (informational, no alerting): email delivery, AI providers, payment processors, SSO, plus compact tiles linking out to the existing push-delivery-health and Stripe-webhook-event pages.
• Scheduled jobs: last-run outcome for every task in CELERY_BEAT_SCHEDULE, tracked via task_prerun/task_postrun/task_failure Celery signal handlers (greekmanage/celery.py) into PlatformBeatJobRun — a lightweight alternative to adding django-celery-beat/-results.

See apps/platform/health/ (checks.py, aggregator.py, alerts.py, tasks.py) for the implementation.

For the admin-facing walkthrough, see Platform Health & Status Dashboard.

What's missing (priority order)

1. Error tracking — Sentry (recommended)

Why first: production errors are silently lost in container logs unless you tail them. Sentry de-dupes, groups by stack, alerts on first occurrence.

Setup:

pip install sentry-sdk

# settings.py
import sentry_sdk
from sentry_sdk.integrations.django import DjangoIntegration
from sentry_sdk.integrations.celery import CeleryIntegration

if SENTRY_DSN := env("SENTRY_DSN", default=None):
    sentry_sdk.init(
        dsn=SENTRY_DSN,
        integrations=[DjangoIntegration(), CeleryIntegration()],
        traces_sample_rate=0.1,
        send_default_pii=False,  # don't ship PII
        environment=env("DJANGO_ENV", default="development"),
        release=env("RELEASE_VERSION", default="dev"),
    )

Frontend equivalent for browser errors:

import * as Sentry from "@sentry/react";

Sentry.init({
  dsn: import.meta.env.VITE_SENTRY_DSN,
  integrations: [Sentry.browserTracingIntegration()],
  tracesSampleRate: 0.1,
  environment: import.meta.env.MODE,
});

2. Metrics — Prometheus + Grafana

Why second: capacity planning, performance regressions, SLO tracking.

Setup:

pip install django-prometheus celery-prometheus

# settings.py
INSTALLED_APPS += ["django_prometheus"]
MIDDLEWARE = [
    "django_prometheus.middleware.PrometheusBeforeMiddleware",
    *MIDDLEWARE,
    "django_prometheus.middleware.PrometheusAfterMiddleware",
]

Endpoint: /metrics (restrict to cluster-internal scrape).

Useful initial metrics:

• django_http_requests_total{method, view, status} — request rate, error rate
• django_db_query_duration_seconds — slow queries
• celery_task_runtime_seconds{task} — task latency
• process_resident_memory_bytes — memory leaks

Grafana dashboards: import community dashboards for Django + Celery, customize per app.

3. Distributed tracing — OpenTelemetry

Why third: trace a single request across backend → DB → Celery → external API.

Setup:

pip install opentelemetry-distro opentelemetry-instrumentation-django \
            opentelemetry-instrumentation-celery opentelemetry-instrumentation-psycopg2
opentelemetry-bootstrap --action=install

# settings.py — auto-instrumentation via env
# OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4317
# OTEL_SERVICE_NAME=greekmanage-backend

Backends: Tempo, Jaeger, Honeycomb, Datadog (your choice).

The edge is a monitoring blind spot

greekmanage-waf-common-rule-set-blocking (v0.73.73) alarms when the WAF Core Rule Set blocks 5 requests in 15 minutes, publishing to greekmanage-ops-alerts.

It exists because of a failure mode nothing else in this stack can see. The managed rule SizeRestrictions_BODY blocks any request body over 8 KB, and it was armed against production from 2026-07-25 to 2026-08-24. Every file upload in the app failed with a CloudFront 403 for a month. Nothing reported it:

• The backend is healthy throughout. The request is rejected at CloudFront and never reaches Django, so there is no 5xx, no django.request warning, no audit row, and no ALB metric. Every alarm on this page stayed green.
• CI is green throughout. E2E runs against a local compose stack with neither CloudFront nor WAF in front of it. No test in the repo sends a request through the layer that actually rejects them, so no test can fail.

Anything that terminates in front of the ALB — WAF, CloudFront, the proxy in front of staging — has this property: it is invisible to both the application's telemetry and its test suite. The only signals are the WAF logs (aws-waf-logs-greekmanage) and metrics on the edge itself, which is why the alarm keys on AWS/WAFV2 BlockedRequests rather than anything the app emits.

The threshold is deliberately low. The outage was one admin and eight blocked requests; a threshold tuned for mass events would not have fired. The Amazon IP Reputation list is deliberately not alarmed — it blocks scanners continuously and correctly, and would page nightly.

4. Uptime monitoring — done, via Route 53

An external GET against /api/health/ every 30s, failing after 3 consecutive errors, feeding the greekmanage-backend-dependencies composite alarm → greekmanage-ops-alerts SNS topic. See terraform/alarms.tf.

This is the only active check in the stack. Every other alarm is passive — it measures traffic users generate, so alb-target-5xx needs ten real requests to fail before it fires, and a dependency outage during a quiet period produced no signal at all. The RDS alarms do not cover it either: all four (CPU, free storage, connections, and FreeableMemory added in v0.73.0) use treat_missing_data = "notBreaching", so a stopped or unreachable instance publishes no metrics and stays silent.

greekmanage-rds-freeable-memory is the newest and the one most likely to earn its keep on db.t4g.small: RDS answers memory exhaustion by killing and restarting the engine, which reaches the application as a burst of connection errors and reads as a network fault rather than a memory one. 🔴 Its 256 MiB threshold is ~12.5% of that instance class and must be raised alongside any instance-class bump — the same boundary at which #689 turns Performance Insights on.

Notification is gated by the composite's actions_suppressor, pointed at an alarm on the ALB's HealthyHostCount. That reads live state rather than duplicating any schedule, and it narrows the alarm to the failure it is actually for: the app is serving but its dependencies are not reachable. "Nothing is serving" is a different failure with its own alarms (alb-unhealthy-hosts, alb-elb-5xx), and without the suppressor a total outage would page twice with two different stories.

The suppressor alarm is still named greekmanage-backend-asleep. Since v0.73.0 (#882) production is always-on and there is no sleep window, so the name is historical — it fires only on a genuine outage now. Renaming a CloudWatch alarm is a destroy-and-create, the composite references it by name, and every runbook and past incident note points at the old name.

:::warning Do not express this as alarm_rule = A AND B The first version did, and it emailed on every sleep and every wake — four transitions a weekday. The two signals fail and recover in opposite order: Route 53 trips in ~2.5 min, while HealthyHostCount clears slowly because a deregistered target stops publishing the metric and CloudWatch must conclude the data is missing. On sleep the dependency alarm led by four minutes; on wake the gate led by one second. No evaluation_periods value fixes both ends.

actions_suppressor does, because it is two-sided: wait_period holds notification while the suppressor catches up, and extension_period keeps suppressing after it clears. :::

wait_period is 60s, down from 300s (v0.73.0, closes #833). The 300s existed solely to absorb a sleep transition, where the dependency alarm led the suppressor by a measured 250s. Always-on leaves no transitions, so keeping 300s would buy no safety and only delay every real dependency page by five minutes — on the stack's only active check. extension_period stays at 300s because it covers recovery, not sleep: after a real outage a target can serve before Route 53 has re-confirmed the endpoint, and that race is unchanged.

:::danger Restoring the wake/sleep schedule means restoring wait_period too At 60s a sleep window ships a false dependency page on the very first transition — the measured gap is four times that value. Change both in the same commit. :::

A third-party monitor (Better Uptime / Pingdom / Healthchecks.io) is still worth adding if you want alerting that survives an AWS-wide event.

5. Log aggregation — Loki / Grafana Cloud Logs

Central place to search across all pods. Set up at the cluster level:

• Promtail → Loki (self-hosted)
• Grafana Cloud Logs (managed)
• AWS CloudWatch Logs (if on AWS)

Tag logs by pod + app for filtering.

6. Frontend RUM — Web Vitals

Capture Core Web Vitals (LCP, FID, CLS) via the web-vitals package and ship to Sentry / Datadog / your analytics:

import { onCLS, onFID, onLCP, onINP, onTTFB } from "web-vitals";

const send = (metric) => {
  navigator.sendBeacon("/api/web-vitals/", JSON.stringify(metric));
};
onCLS(send); onFID(send); onLCP(send); onINP(send); onTTFB(send);

Cost considerations

Tool	Cost (approx)	Notes
Sentry	Free up to 5K events / month, then $26+/mo	Most affordable error tracking
Better Uptime	Free for 10 monitors	Simplest uptime tool
Healthchecks.io	Free for 20 checks	Cron / job monitoring
Grafana Cloud	Free tier 10K series, 50GB logs	Easiest hosted Prometheus + Loki
Datadog	$15/host/mo + extras	Most expensive but most integrated

For a typical org running GreekManage in production, Sentry + Better Uptime + self-hosted Prometheus is a reasonable starting baseline (~$30/mo).

Manual debug techniques (until tooling lands)

Tail logs across pods

kubectl logs -n greekmanage -l app=backend --tail=200 -f --max-log-requests=10

Inspect a slow query

Use EXPLAIN ANALYZE in psql:

kubectl exec -it -n greekmanage postgres-... -- psql -U greekmanage greekmanage

EXPLAIN ANALYZE
SELECT * FROM members_memberprofile mp
JOIN organizations_membership m ON mp.membership_id = m.id
WHERE m.chapter_id = 'uuid-here';

Check Celery queue depth

kubectl exec -it -n greekmanage redis-... -- redis-cli LLEN celery

A growing queue means workers are saturated → scale up celery deployment replicas.

Find a request in logs by request ID

Backend includes X-Request-Id header in responses. Grep for it across logs:

kubectl logs -n greekmanage -l app=backend | grep "abc123-..."

(Add request_id to every log line via Django middleware — see apps/common/middleware.py.)

Roadmap (rough priority)

1. Sentry integration — biggest immediate value
2. Uptime monitoring — cheap, fast
3. Prometheus + Grafana — for capacity + perf
4. Log aggregation — when scale demands it
5. Distributed tracing — when LLM / external API debugging gets hard
6. Frontend RUM — when UX perf becomes a focus

Tickets to track each are in the issue tracker (observability label).

Related

• Deployment
• CI / CD pipeline
• Audit & encryption