turns out nothing was backing up (and a downsize) #71

Merged

sarahlament merged 10 commits from tweak into main 2026-07-08 16:29:06 -05:00

Conversation 0 Commits 10 Files changed 12 +145 -17

sarahlament commented 2026-07-08 13:13:54 -05:00

Owner

Copy link

• borg was silently dead on two hosts, fixed both
  • athena: a knotc zone-dump preHook aborted every run - knotc isn't on the job $PATH, and zone-dump isn't even a real subcommand. dropped the hook, /var/lib/knot lmdb is crash-consistent so the flush bought nothing
  • minerva: never ran once - hetzner sub3's host key was never trusted, and /root/.ssh/known_hosts is tmpfs on an impermanent box so a manual accept wouldn't survive a reboot anyway
    • declared the storage-box key in programs.ssh.knownHosts, derived from constants.borg - survives the wipe, unblocks minerva, hardens athena/ishtar for reinstalls
  • both verified now (vmail/dkim/knot on athena, vaultwarden/forgejo/gollum on minerva); zero failure alerting anywhere -> follow-up
• prepped athena for an impermanent reinstall on a downsized nanode
  • diskoConfigurations/athena.nix: gpt + 1M bios_grub + ext4 /boot + btrfs @nix/@persist/@log + tmpfs /. MBR would've dropped the bios_grub part but disko deprecated the legacy table type; ext4 /boot keeps GRUB off btrfs
  • athena.roles.impermanent = true
  • persist dirs w/ explicit ownership (a plain string dir lands root:root and locks the service out): vmail (storage.owner/group), dkim (rspamd), /var/lib/knot (knot, the KASP keys or DNSSEC goes bogus on reboot)
  • linodeBase: dropped grub.device - disko's bios_grub already sets grub.devices, having both dup'd the disk in mirroredBoots
• cleanup:
  • /persist neededForBoot -> impermanence.nix (identical for every impermanent host), out of athena + minerva host configs
  • acme rides in via mailserver's imports (its only consumer) instead of athena's activeModules
  • journald SystemMaxUse=500M fleetwide - athena's had ballooned to 3.9G
• left for later: the reinstall itself - linode Direct Disk -> delete/resize/new disk -> nixos-anywhere .#athena -> seed /persist (age key + borg extract) -> verify dnssec/mail/hub

- borg was silently dead on two hosts, fixed both - `athena`: a `knotc zone-dump` `preHook` aborted every run - `knotc` isn't on the job `$PATH`, and `zone-dump` isn't even a real subcommand. dropped the hook, `/var/lib/knot` lmdb is crash-consistent so the flush bought nothing - `minerva`: never ran once - hetzner `sub3`'s host key was never trusted, and `/root/.ssh/known_hosts` is tmpfs on an impermanent box so a manual accept wouldn't survive a reboot anyway - declared the storage-box key in `programs.ssh.knownHosts`, derived from `constants.borg` - survives the wipe, unblocks minerva, hardens athena/ishtar for reinstalls - both verified now (`vmail`/`dkim`/`knot` on athena, `vaultwarden`/`forgejo`/`gollum` on minerva); zero failure alerting anywhere -> follow-up - prepped `athena` for an impermanent reinstall on a downsized nanode - `diskoConfigurations/athena.nix`: gpt + 1M `bios_grub` + ext4 `/boot` + btrfs `@nix`/`@persist`/`@log` + tmpfs `/`. MBR would've dropped the `bios_grub` part but disko deprecated the legacy `table` type; ext4 `/boot` keeps GRUB off btrfs - `athena.roles.impermanent = true` - persist dirs w/ explicit ownership (a plain string dir lands `root:root` and locks the service out): `vmail` (`storage.owner`/`group`), `dkim` (`rspamd`), `/var/lib/knot` (`knot`, the KASP keys or DNSSEC goes bogus on reboot) - `linodeBase`: dropped `grub.device` - disko's `bios_grub` already sets `grub.devices`, having both dup'd the disk in `mirroredBoots` - cleanup: - `/persist` `neededForBoot` -> `impermanence.nix` (identical for every impermanent host), out of athena + minerva host configs - `acme` rides in via `mailserver`'s `imports` (its only consumer) instead of athena's `activeModules` - `journald` `SystemMaxUse=500M` fleetwide - athena's had ballooned to `3.9G` - left for later: the reinstall itself - linode `Direct Disk` -> delete/resize/new disk -> `nixos-anywhere .#athena` -> seed `/persist` (age key + `borg extract`) -> verify dnssec/mail/hub

sarahlament added 6 commits 2026-07-08 13:13:54 -05:00

tweaks, part 1 8882774468

whoops 46bc2f7a22

borgbackup started silently failing 8accb5ef60

knot db is crash-resistant anyways 5a2cf4453f

trust hetzner storage box pub key 9fff952953

impermanence on athena cb0be9b1ed

sarahlament changed title from turns out nothing was backing up to turns out nothing was backing up (and a downsize) 2026-07-08 13:14:12 -05:00

sarahlament added 1 commit 2026-07-08 14:50:37 -05:00

acme persistence cc0ed4a68f

sarahlament added 1 commit 2026-07-08 15:45:37 -05:00

LE and HE and ACME race c3b4ec359c

sarahlament added 2 commits 2026-07-08 16:03:17 -05:00

LE and HE and ACME race the second a4f4435e9f

TLSA changed aa6ad8aaac

sarahlament merged commit b220280ded into main 2026-07-08 16:29:06 -05:00

sarahlament deleted branch tweak 2026-07-08 16:29:06 -05:00

sarahlament referenced this pull request from a commit 2026-07-08 16:29:07 -05:00

turns out nothing was backing up (and a downsize) (#71)

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

dropped

This is a dropped feature

  

feature

This is a new feature

  

fix

This fixes a problem

  

fleet

This is a fleet issue

  

lock

This is an update to the lock file

  

module

This is a new module

No labels

dropped

feature

fix

fleet

lock

module

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Reference

sarahlament/nix-configurations!71

No description provided.