Skip to content

storage-forecast

Warns weeks before any place we write to fills up, instead of the writer failing on the day it is full.

Input → Movement → Output

Daily at 08:00 (cron, cron-run storage-forecast) it samples used space on /, disk1-3, parity, the rescue stick and the Cloudflare R2 account. It fits the last 14 days of samples and pages once per target per week when a target is over its limit or will reach it within 30 days. Samples live in ~/.local/state/storage-forecast/samples.jsonl.

Why this shape

Three "storage quietly filled" outages in one week (2026-10-08..10: R2 backup stop line, rescue stick, erp-backup never pruning). Every writer had a guard, but each guard only fired once the place was already full. A trend check is independent of the writers, so a bug in one writer's retention cannot hide the growth.

Steps

  1. Limits: disks 90 %, / 85 %, parity 98 % (sized to the largest data disk by design), rescue stick 90 %, R2 8.5 GB (offsite-backup stops at 9 GB). ~/.local/bin/storage-forecast DISKS, R2_LIMIT_GB.
  2. R2 unreadable is reported on stderr, never treated as room to spare.
  3. --check prints the table and records nothing; --test runs the forecast self-test.

If you change this

  • Hits: the one weekly page per target (paged.json); cron-watch limit is computed from the crontab.
  • Does not hit: the writers' own guards (offsite-backup stop line, rescue-sync prune-to-fit), which still apply.

Surfaces

ntfy topic plex-hgviaey. Related: rescue-build, alerting.

Opens a Kanboard task (2026-10-25)

Each target in WARN also gets one open task in Administration (ensure_task, ids in ~/.local/state/storage-forecast/tasks.json, re-created only once the old one is closed). The R2 ping of 10-18 alone was ignored for a week until offsite-backup hit its stop line. First tasks: #335 R2, #336 root /, #337 rescue stick.

Automatic cleanup first, then a task (2026-10-25, owner-approved)

Order for a target in WARN: run its allow-listed cleanup (reclaim_rules), measure again, and only if it is still in trouble page + open/keep one Kanboard task. Root / rules: docker build cache idle >7 d (docker builder prune -af --filter until=168h), docker images no container uses AND named by no crontab/script/compose file AND untagged or pulled >30 d (docker rmi, no -f), npm cache clean, uv cache prune, /tmp/claude-1000/*/* idle >7 d, ~/kicad-fork*/build idle >30 d. Never precious/, Trash, repos.

  • Report-only until ~/.config/storage-forecast/reclaim-enabled exists: it prints/records what it would free and the task carries that. Set up 2026-10-25; switch on after a week of sane reports.
  • Attribution: every sample stores g = sizes of ~/ , ~/.local/share/, /tmp, /var/log, docker parts; tasks and pages say "grew most in 7 days".
  • R2 is now the object listing (exact), not /usage (lags hours). A >5 % drop restarts the trend (a cleanup's old slope means nothing).
  • Rescue stick is SELF_MANAGED: rescue-sync prunes to fit and warns itself; only "over limit now" counts (#337 was a false alarm).
  • Outputs: reclaim.jsonl, summary.json (read by the Monday job report, job_failure.storage_line). Self-test --test covers enough→no task, short→one task, report-only→no deletes, failing rule→others still run.
  • First manual run 2026-10-25 freed 8.7 GB (442.7→434.0 GB). Honest ceiling: most "reclaimable" docker images are named by scripts/compose and stay.

Reclaim switched on (2026-11-05)

~/.config/storage-forecast/reclaim-enabled created (owner: "fix those issues" on the 11-05 root / page; reports had been sane since 10-25). Allow-list only frees ~0.5 GB/day-run on /. The 11-05 forecast (limit in ~7 days) is mostly project data outside the allow-list: ~/kicad-fork +8.5 GB (clone+build 10-29, build/ 4.5 GB), ~/eeroute/work 17 GB (pns-spike 7.9, native-editor 3.6), docker images 52 GB (14.6 reclaimable but script/compose-named). Moving those is an owner decision. docker image prune (dangling) freed 0 B. - 2026-11-05 (owner: "free all this"): ~/eeroute/work/pns-spike + native-editor (12 GB, gitignored run outputs, NOT on GitHub) copied to /mnt/storage/EmbeddedEra/Github/eeroute-work-archive-20261105/ (diff-verified; only stale api.sock sockets differ). ~/kicad-fork (8.5 GB) verified clean, 0 unpushed (GitHub EmbeddedEra/kicad-fork + NAS origin); deleting it means eeroute routing needs a re-clone + build of build/fr/qa/tools/pns/pns_route_nets first. Owner deleted them 11-05; / 404.9 -> 384.2 GB (81 %), forecast ok. Link ~/eeroute/work/archive-20261105 -> pool copy.