Skip to main content

KPI snapshot retention

Every night, GreekManage writes one KPI snapshot row per chapter: active members, total members, alumni members, compliance %, dues collection %, and the rest of the analytics KPI set. Those rows are what every trend line, cohort curve and longitudinal chart in Analytics is drawn from.

This page's setting controls how long they are kept.

What you see​

Open Platform → Organizations, click an organization, and find the KPI Snapshot Retention card. It has one control: a Retention Period (days) number input, with the equivalent in years shown underneath it.

The default is 10950 days — 30 years.

There is no maximum, and that is deliberate​

The audit log retention setting on the same page caps at 180 days. This one has no cap at all. That asymmetry is intentional, and the two settings should not be reconciled:

Audit logsKPI snapshots
Default180 days10,950 days (30 years)
Maximum180 days, enforcednone
ContentsPer-request records naming users and IP addressesAggregate counts and percentages, naming nobody
VolumeHigh — potentially every requestOne row per chapter per night
Why the policyHigh-volume, PII-bearing, legally bounded — get it off the databaseTiny, aggregate, PII-free, and the long tail is the entire value

A KPI snapshot is roughly 660 bytes. A 5,000-chapter platform accumulates about 14 GB over a decade — on the order of $19/year of RDS storage.

Why shortening this is not reversible​

A snapshot is a nightly observation of a roster that changes. It is not derived from anything still on the system, so it cannot be recomputed after the fact. If you shorten an org's window and the pruning job runs, that slice of the chapter's membership history is gone — there is no archive, and no undo.

This is the failure mode to be careful about, because it is silent and slow: nothing breaks at the time, and months later a chart simply starts later than it used to.

Before shortening a window, note that the weekly pruning job logs exactly what it removes — see below.

What the pruning job does​

apps.analytics.tasks.prune_kpi_snapshots runs weekly (Sundays, 06:00 UTC). For each organization it reads that org's retention setting — falling back to the 30-year default when the org has no config row — and deletes snapshots with a captured_at strictly older than the cutoff. A snapshot dated exactly on the cutoff is kept.

Every deletion is logged. For each org that loses rows, the job records the retention window in force, the cutoff date, the number of rows, and the oldest and newest dates affected. Orgs that lose nothing log nothing, so the weekly entry stays readable. That log is what makes "why does this chapter's chart start in 2029?" an answerable question rather than a shrug.

The task also supports a dry run, which logs exactly the same detail and deletes nothing:

from apps.analytics.tasks import prune_kpi_snapshots
prune_kpi_snapshots(dry_run=True)   # returns the number of rows it WOULD delete

Run that before shortening a window if you want to see the blast radius first.

Very large windows​

There is no maximum, but there is a physical limit: a window beyond roughly 8,200 years puts the cutoff before the earliest date the system can represent. The job detects that, logs it, and prunes nothing for that org — which is the correct outcome, since no snapshot can be older than that. One org set that way does not affect any other org's pruning.

History note​

Before v0.73.48, retention was not a setting at all. It was a retention_days column on each individual snapshot row, defaulting to 730 days (2 years), with no interface anywhere. The pruning job read it off OrgKPISnapshot.objects.filter(organization_id=...).first() — and because snapshots are ordered newest-first, that meant the policy governing an org's entire history was whatever value last night's row happened to carry. The column has been dropped; the org-level setting is the only source of truth.