Server scheduled jobs storage audit and fix plan¶
Audited November 10, 2026, approximately 21:05–21:20 EST. The audit is complete for the schedule inventory below; the fixes are proposed. No production schedule, backup policy, container, or protected data was changed.
The first priority is the root disk: it is at 87% according to df, with about 64 GB available. The largest recent contributor is precious-sync: its destination grew 17.42 GB between November 3 and 10. Of its current contents, 12.21 GB is the routing archive moved off root on November 5, then copied back by the nightly job. Fix placement and copy policy before another cleanup. Cloudflare R2 is the next constrained destination, at 7.466 GB out of the 10 GB account budget.
Scope and proof¶
The inventory covers all 81 active angel crontab entries, 28 user timers, 25 system timers, /etc/crontab, all readable /etc/cron.d declarations, anacron and daily/weekly/monthly dispatchers, and the 3 root cron entries exported to the rescue stick at 01:30 today. All 80 containers were checked for configured stdout log limits. Backup scripts, destinations, retention code, job heartbeats and selected application-managed storage were inspected. The companion inventory lists each cron entry and timer individually.
Direct access to root's current crontab and root-only Docker storage was denied; the root inventory therefore uses today's rescue export plus logs and readable installed scripts. Application-internal schedules were checked selectively through their host data and configuration; this is not a complete inventory of every scheduler inside every application. Unreadable files make directory sizes lower bounds where applicable. Root privileges are needed to close those specific gaps, not to discover the main findings.
Professional checks used here: follow the data through every backup layer; use live scripts, filesystem allocation, exact R2 object listings and seven-day samples; distinguish independent recovery copies from redundant generated data; reproduce failure behavior in disposable fixtures. Completion of each proposed fix is defined by an external measurement or recovery check below. Only the primary Codex agent ran. Helpers were not launched because the available native helper route did not expose enforceable read-only tool restrictions required by SOP-025.
Private evidence is in ~/.local/state/server-audits/cron-space-audit-2026-11-10/. Raw commands, service properties and logs are kept out of the published notebook.
Measured storage¶
All GB values below are decimal GB; df -h uses binary units. Shared backup layers are not additive savings.
| Destination or writer | Measured allocation | Interpretation |
|---|---|---|
| Root filesystem | about 412 GB used of 502 GB; 64 GB available | 87% in df; about 25.57 GB of free blocks are reserved |
| Pool | about 4.8 TiB used, 6.0 TiB available | Capacity is adequate; reduce unnecessary writes and snapshots |
| Precious copy on root | 147.35 GB | 85.65 GB Immich, 61.30 GB EmbeddedEra; never auto-delete |
| Engineering workspace | 20.55 GB | Repos 9.90 GB, cache 2.85 GB, 3D models 2.47 GB, test output 3.51 GB |
| Active rsnapshot tree on disk1 | 214.56 GB | One traversal across all hardlinked intervals; not a reclaim estimate |
| Stale rsnapshot trees on disk2 and disk3 | 109.23 GB combined | 71.27 GB + 37.96 GB; preserve any unique history before removal |
| Local restic history | 143.35 GB | Versioned recovery copy on the pool; preserve it |
| Docker configuration snapshots | 21.42 GB across seven nights | Latest 3.17 GB; full independent copies each night |
| Local encrypted ERP and store sets | 9.85 GB | ERP 9.46 GB, store 0.40 GB; also included by broader history jobs |
| Rescue USB | 11.72 GB used | Eight approximately 1.3 GB archives plus 0.67 GB host data; space pruning works |
| Pod state | 4.74 GB | Designs 2.74 GB, approval snapshots 1.88 GB; business records, not generic cache |
| User cache | 7.34 GB at audit | Playwright 2.30 GB, Codex runtime 1.81 GB, uv 1.32 GB; much is active |
| Docker | 52.5 GB images, 5.19 GB container layers, 4.14 GB volumes | CLI reclaimable figures are not permission to prune |
| Journals and host logs | about 1.05 GB journal; 2.99 GB accessible /var/log | SnapRAID logs include a 0.215 GB old manual run |
| User .log files | 79.5 MB across 243 files | Not the principal storage problem; cron-run directory is only 6.69 MB |
Seven-day changes to November 10 at 08:00: precious +17.42 GB, engineering +4.95 GB, user cache +3.25 GB, /tmp +1.10 GB, ~/.local +1.10 GB. Parent and child measurements overlap and must not be summed. These are measured changes, not a promise that the rate continues.
R2 was measured again through a paginated object listing during this audit:
| Account component | GB | Objects or sets |
|---|---|---|
| Encrypted ERP sets | 4.680 | 225 objects, 19 sets |
| Offsite restic repository | 2.276 | 163 objects |
| Encrypted store sets | 0.311 | 34 objects, 17 sets |
| Erascope application data | 0.194 | 8 objects |
| Other objects | about 0.005 | Remaining account data |
| Total | 7.466 | Account-wide, not just the backup bucket |
The 08:00 forecast estimated seven days to its 8.5 GB R2 warning line. That is a trend estimate affected by weekly pruning, not seven days to a guaranteed outage.
Findings and fixes¶
1 Stop the archive copy loop¶
Priority 1. precious-sync.sh copies EmbeddedEra broadly, excluding backups/. The 12.21 GB Github/eeroute-work-archive-20261105 directory is inside that scope, so the space released on root was consumed again. Both pool and precious copies were measured. This explains most, not all, of the 17.42 GB weekly increase; there is no historical per-subdirectory sample to attribute the remainder exactly.
Plan: distinguish source and business records from reproducible routing/build output. Relocate the archive outside the broad irreplaceable-data source, or introduce a precise approved exclusion consistently in copy, verification, byte counts and history policies. First establish which archived outputs can be regenerated; gitignored does not mean disposable. Preserve any unique evidence. Also classify the broad EmbeddedEra copy by purpose: precious contains 16.29 GB under Programs and 13.59 GB under Yara, including application-profile paths observed during snapshot traversal. These totals are not a cleanup allowance; distinguish installers, downloaded resources and caches from personal/business originals before narrowing future copy scope. Only after checking alternative copies and the restore path should the owner authorize removal of the existing precious duplicate. Moving it elsewhere on the same root disk would free nothing; moving precious to the same pool would remove device independence.
The existing precious headroom guard also credits the size of the entire destination against the current source size. Old-only files and unrelated precious archives can make that credit optimistic; replace it with an estimate of actual new transfer bytes plus staging headroom.
Proof: compare file manifests/hashes, verify a restore, run the proposed sync in dry-run mode, then verify that the next real nightly copy does not recreate the archive. Up to 12.21 GB root recovery is a conditional opportunity, not authorized deletion.
2 Make the storage forecast use available capacity¶
Priority 1. Live statvfs measured about 501.81 GB total, 412.13 GB used, 64.11 GB available to the user and 25.57 GB reserved free. storage-forecast compares used with total, yielding 82.13%; df compares used with used plus user-available space, yielding 86.54%, rounded to 87%. Consequently its nominal root 85% limit is not the same limit the owner sees in df.
Plan: use one documented usable-capacity percentage, show available GB, and forecast exhaustion from available bytes with an explicit reserve for backup staging. Preserve the filesystem's reserved blocks. Report growth after cleanup as well as before it, and retain the existing one-task-per-target behavior. Add a modest daily attribution for the largest scheduled destinations so future growth has a producer and subdirectory, rather than only a home-directory total.
Proof: a fixture with reserved blocks must reproduce the current discrepancy and pass after the change; live output must agree with df within rounding. No root write is necessary for this script change.
3 Publish only successful Docker snapshots¶
Priority 1. /usr/local/bin/docker-backup.sh creates the final timestamp directory first, treats rsync errors as warnings, prints Backup Completed and then prunes old snapshots. A disposable test of the installed script, with rsync forced to exit 23 and all output redirected to temporary fixture directories, returned exit 0 and left a final snapshot directory. Rescue subsequently selects a directory by newest name and age, not by an explicit success record.
Plan: take a job lock, write into a staging directory, fail on missing required inputs or copy errors, verify required files, then atomically promote and write a completion manifest. Rescue should select only a completed snapshot. Preserve the last verified good snapshot on failure; prune after successful promotion. Add free-space admission before making another full copy. Report failures through the existing root failure path without duplicate phone alerts.
Proof: inject rsync failure and ENOSPC in fixtures; neither may create a selectable snapshot or prune the last good one. A normal fixture must promote successfully. Then restore a real configuration set without restarting production services.
4 Retire or repair the obsolete USB backup¶
Priority 1. Today's rescue export includes root's daily 02:00 torrentvpn-backup.sh job. Its latest log says USB not mounted at /media/angel/ESD-USB, and no mount exists there. The job is absent from the three open incidents printed by job-incidents. If that path becomes mounted again, the script stops qBittorrent, archives the entire stack, writes the latest link before checking archive validity, and keeps three archives only after writing the new one. It overlaps the working Docker snapshot and rescue jobs.
Plan: compare recovery coverage with the working 01:00/01:30 chain. Retire the obsolete entry if that coverage replaces it. Otherwise give it mount identity checks, a byte budget, a lock, tested failure cleanup, a success manifest and monitoring; coordinate any qBittorrent stop/start with its VPN interlock. Do not run it merely to test this finding.
Proof: read back root's actual crontab after the scoped root change. Confirm the intended rescue archive is current and restores. No backup data needs to be deleted to retire a duplicate schedule. Immediate savings are negligible because the job currently exits before writing.
5 Bound every R2 writer before upload¶
Priority 1. offsite-backup checks account usage at 7/9 GB, but ERP/store upload their independent encrypted sets without the same account admission guard. ERP alone accounts for 4.68 GB. Checking current usage before one uploader does not reserve capacity for its payload or another uploader. Offsite retention logs failures and can still report backup ok, hiding a cleanup failure until later growth.
Plan: share one account-wide budget policy and lock across backup writers. Measure the prospective payload or use a conservative upper bound for restic. Reserve room for the new set and prune working space. Surface retention failure as degraded/failing health; preserve the required minimum good recovery sets. Do not prune unrelated Erascope or application objects. Keep pod art excluded from R2.
Proof: fixtures for account near limit, concurrent writers, failed listing, failed prune and oversized payload must deny unsafe uploads without deleting the last good copy. Re-list all buckets after a real run and verify ERP, store and offsite restore checks.
6 Stop rearchiving rotating encrypted sets indefinitely¶
Priority 2. Local ERP/store sets total 9.85 GB. history-backup includes the whole EmbeddedEra tree, and rsnapshot includes it too, so these rotating sets have additional histories. Age encryption produces fresh ciphertext for each dump; one cannot assume content deduplication removes that cost. The offsite restic source also overlaps some home data protected by the separately encrypted ERP set, including secrets and assistant state.
Plan: draw a coverage matrix for each database, media directory, secret bundle and business tree: live copy, consistent export, independent local copy, versioned copy, offsite copy, verified restore. Choose one authoritative version history per data class. Prefer consistent database snapshots over copying live databases. Then propose exact exclusions for rotating encrypted backup directories in broader histories, with explicit recovery windows. Existing historical packs should expire through approved retention, not ad hoc deletion.
Proof: restore the newest and an older required ERP/store set with the paper-key recovery procedure before exclusions; compare restic bytes added over seven nights afterward. Savings are not estimated from overlapping logical sizes. The current 02:50 history run also precedes the 03:15 ERP dump, so its newest ERP set is normally from the previous day; preserve that fact in the recovery design.
7 Consolidate rsnapshot after measuring unique history¶
Priority 2. Live retention is 24 hourly, 30 daily, 12 weekly, 24 monthly and 5 yearly. Restic provides overlapping version history; rsnapshot also copies broader Immich paths than the selective restic backup. The existing map card's claim that cutting hourly snapshots reclaims most of the tree is not a safe saving estimate: hardlinks share file contents, and old-only versions may reside in other intervals. Disk1 currently uses about 5.30 million inodes across the filesystem, not all attributable to rsnapshot.
Plan: first determine the unique oldest versions, application-generated directories actually captured and actual allocated bytes across the whole tree in one du traversal. Keep any history predating restic until deliberately migrated. Reduce hourly checkpoints only after choosing a recovery objective; verify generated directories before exclusions. Live hourly.0 excludes thumbs and postgres correctly, but includes 32.09 GB of encoded-video. Excluding that whole tree would lose the 13 motion-photo originals; use the same original-path query as precious/restic before excluding reproducible transcodes. The stale trees on disk2/disk3 measured 71.27 GB and 37.96 GB, respectively, for a 109.23 GB pool opportunity. Verify their newest/oldest recovery dates and unique files independently before retiring them. Avoid summing separate du runs over hardlinked snapshots.
Proof: old-file restore from the oldest required date; unique-byte accounting; next rsnapshot succeeds; filesystem allocation/inodes improve after approved cleanup. A single full-tree traversal completed successfully at 214.56 GB. That is actual allocated storage across the hardlinked intervals, not bytes that can all be removed.
8 Shrink the configuration payload and engineering outputs¶
Priority 2. Docker makes seven full copies, totalling 21.42 GB. The latest includes 0.337 GB Whisper models, 0.151 GB Recyclarr resources and 0.271 GB old pre-update directories. These candidates alone total about 0.759 GB per run, or 5.31 GB across seven full copies, before accounting for growth, compression or shared files. They are candidates for classification, not automatic exclusions.
Plan: document the reinstall/restore method for downloaded models/resources and the expiry of rollback folders. Keep required configuration, databases, credentials and unique uploads. Consider hardlink or restic snapshots for unchanged configuration while keeping a standalone rescue archive and validated restore process. Use consistent exports for live SQLite databases. Set a measured maximum payload and free-space floor.
Engineering occupies 20.55 GB and grew 4.95 GB in a week. Its kicad-fork checkout alone is 6.46 GB; test output is 3.51 GB. engineering-deploy already cleans its temporary staging directory and skips an unchanged upload, but the expensive build occurs before that hash comparison. Its cache root has no general age/size budget. Move appropriate generated output to a dedicated pool location outside the precious source tree; preserve dirty repos, worktrees and unique test evidence. Check inputs before expensive rebuilding when correctness permits.
Proof: fresh configuration restore including model redownload; successful rescue validation; engineering builds and serves identical required assets after relocation; next precious run does not copy generated output back. Pool snapshots offer at most the classified payload saving; engineering relocation offers at most the verified moved bytes. Do not add these estimates to overlapping archive savings.
9 Repair ERP status and temporary cleanup behavior¶
Priority 2. r2_prune uses for stamp in ... without a local stamp variable. Bash resolves it to the caller's local stamp, overwriting the date later used in status/log output. The November 10 status claims an October 27 backup although a November 10 local set exists. A fixture of the extracted function with mocked listing/deletion reproduced the timestamp overwrite. This is a reporting defect, not evidence that the uploaded November 10 set is absent.
Plan: localize retention loop variables and report the immutable created-set identifier. Add a complete manifest and expected-file validation so missing dumps cannot look like a fully successful backup merely because something encrypted. Audit RETURN/EXIT cleanup traps across backup and restore modes; preserve failed restores only in an explicitly bounded diagnostic area. /var/tmp is only about 0.34 GB now, with one old ERP verify directory, so this is prevention rather than a large immediate saving.
Proof: retention fixture must leave the caller stamp unchanged; injected missing required dump must fail; success and failure cleanup fixtures must stay within the job's own directories; a real local/R2 manifest must match the heartbeat date.
10 Use file rotation for logs and check the uncapped container¶
Priority 2. cron-run keeps the last 200 KB before the next run, which does not bound one runaway run or a job that never runs again. Some jobs also append to a second log. The monthly retention sweep processes only parseable leading timestamps and leaves undated output intact; it reads and rewrites active files without coordinating with appenders. Its November 2 run freed zero log bytes. This is not a reliable byte cap.
Plan: rotate active logs daily with a size trigger, compression, bounded total storage and the approved 90-day maximum. Use app-supported reopen/locking or a deliberately chosen copytruncate policy; do not casually rewrite live files. Preserve useful failure details without duplicate copies. Include SnapRAID date-named logs and qbit legacy logs in explicit policies, preserving audit evidence as required. Keep security logs under their approved retention. For assistant SQLite cleanup, distinguish deleted row text from filesystem bytes; measure free pages before considering incremental/full vacuum.
79 of 80 inspected containers show explicit stdout rotation limits. mosquitto has json-file logging with an empty options map, so check its effective cap and log size under scoped root access; stage only that container's recreation with explicit limits if needed. Its application log volume is separate and also needs a measured policy. Current journal usage is modest and not a reason for blind vacuuming.
Proof: a chatty fixture remains within its byte budget; failure details survive rotation; logrotate dry-run covers the intended paths; effective container options are correct after any targeted recreation. Expected immediate savings are small, not tens of GB.
11 Preserve business artifacts and repair failure observability¶
Priority 3. Pod designs and immutable approval snapshots are 4.62 GB together. They are deliberately excluded from R2 and protected by pool history. Do not treat them as disposable cache or alter approved art. Other business-state ledgers and daily marketing metrics need their own row/object retention budgets rather than broad directory deletion. books-snapshot copies from root into precious on the same root device, so it offers accidental-deletion recovery but not another device; retain the pool/R2 copies in its recovery description.
Three existing open incidents were observed: engineering-deploy #299, therealcow-close-github #353 and an onboard-access retry. #299 relates to a scheduled build and should be handled with its owning engineering task. The other two are not established storage causes and were not modified. Cataloguing root backups should make the obsolete USB failure visible. Keep VPN, offbox heartbeat, booking and security checks at their required responsiveness; reducing every high-frequency check would not address the measured storage growth.
Proof: explicit data-class budgets and restore coverage; a simulated catalogued root backup failure creates one actionable incident; idle checks remain cheap and recovery/security deadlines remain met.
Order of work and acceptance¶
| Phase | Work | Required completion evidence |
|---|---|---|
| First implementation unit | Fix forecast denominator, immutable ERP stamp and backup failure publication; catalogue/retire obsolete USB cron through scoped root change | Fixtures fail against old behavior, pass against new behavior; live readback and truthful job status |
| Root capacity unit | Approve archive classification/exclusion; verify alternate copy; relocate generated engineering data and authorize any precious duplicate removal | Hash/restore evidence, measured free bytes, next nightly sync does not recreate the problem |
| Backup design unit | Coverage matrix, R2 budget admission, exact nested-backup exclusions and configuration payload reduction | Latest and historical restores; account-wide listing; seven days of bytes-added measurements |
| History and retention unit | rsnapshot migration/retention, log rotation, generated-cache budgets, scoped mosquitto log repair | Oldest required restore, allocated-byte/inode measurements, rotation/ENOSPC/concurrency checks |
| Follow-up at 7 and 30 days | Compare daily producer growth, root runway, R2 sawtooth and restore health | Root below the agreed usable-capacity limit with staging reserve; R2 below its warning line through a pruning cycle; zero missed required backups |
Roll back a script/policy unit to its versioned prior configuration if a restore or required backup fails. Do not delete retained history to make a test pass. Root changes need the existing scoped grant/human installation workflow. Deletion in precious requires specific owner authorization; this audit does not expand the standing cleanup allow-list. No schedule or future unattended work was created.
Verification limits and follow-up measurements¶
The two failure reproductions ran only against temporary fixture directories and mocked remote operations. Existing history logs show the November 8 local restic check and restored-photo drill passed. New full restore drills were not launched during an audit of a live server. The local full-photo history has no configured offbox replica; the business-only R2 backup is not a full offsite photo copy. Preserve that distinction when consolidating protection.
The full rsnapshot allocation scan completed successfully: 214,564,343,808 bytes on disk1. The stale-tree scan also completed: 71,271,329,792 bytes on disk2 and 37,958,660,096 on disk3. These are allocated-byte measurements with hardlinks counted once per traversal; they replace the old map estimates, and do not by themselves prove that any historical file is safe to delete. Direct root crontab readback, privileged Docker log/volume accounting, and exhaustive application-internal scheduler discovery remain explicit follow-up checks.
Companion: server-schedule-storage-inventory-2026-11-10. Owning server note: plex-server.