Lessons — Knowledge Wiki

Lessons — Knowledge Wiki

What Broke

Git dubious ownership error on NFS mount
git init on /Volumes/knowledge failed with “detected dubious ownership” because the NFS mount is owned by a different UID than the Mac user. Fix:

git config --global --add safe.directory /Volumes/knowledge

Run this before any git operation on an NFS-mounted share.

Git initialized as master, not main
git push -u origin main failed because the branch was named master. Fix:

git branch -m master main

Do this immediately after git init before any commits.

macOS resource fork files committed to git
NFS shares on Synology get ._* files (AppleDouble resource forks) and a #recycle/ directory written to them by macOS. These got committed in the initial push. Fix: create .gitignore before the first commit:

._*
.DS_Store
#recycle/
@eaDir/

Then git rm -r --cached to remove already-tracked junk files.

Gitea repo must be created manually before first push
git push to a non-existent Gitea repo returns a 403 “Push to create is not enabled for users” error. You must create the repo in the Gitea UI first (unchecked “Initialize repository”), then push.

Shell comments in terminal instructions
Inline # comments in multi-line shell instructions are interpreted by zsh as commands and produce “command not found: #” errors. Harmless but confusing. When running Claude’s commands, strip the comment lines first or paste commands one at a time.

NFS mount kills mkdocs live-reload
mkdocs serve uses inotify to watch for file changes. NFS mounts on Linux do not reliably deliver inotify events, so the container never detects new or changed files. WATCHDOG_USE_POLLING=true was attempted but did not resolve it. The working fix: restart the knowledge stack in Komodo after every commit. The container does a full fresh read of the NFS mount on startup.

Explicit nav in mkdocs.yml doesn’t auto-discover subdirectories
Setting nav: - Projects: projects/ shows only the section landing page — it does not recursively discover sub-pages in nested directories. Fix: remove the nav: block entirely. MkDocs auto-discovers the full directory tree and builds the nav from the folder structure. Section names come from directory names (title-cased automatically).

2026-09-05 — mkdocs serve leaks a ~200 MB build dir into the container on every restart

The restart-after-every-publish workaround above (NFS gives no inotify) has a cost
nobody costed. mkdocs serve builds into a fresh /tmp/mkdocs_XXXXXXXX and deletes
it in a finally: that is only reached on SIGINT or a clean exit — and mkdocs runs
as PID 1 in both viewers (… && exec mkdocs serve), where the kernel discards
default-disposition signals. The SIGTERM from docker restart is ignored, SIGKILL
follows ten seconds later, and the cleanup provably never runs.

At 17–39 publishes a day since 2026-07-20 that came to 82 GB across the two
viewers: 634 leaked dirs / 51.2 GB in knowledge-test, 479 / 30.7 GB in knowledge.

What made it hard to see is where the bytes were: the container’s writable layer.
docker system prune cannot enter a running container’s layer, so every routine
cleanup reported nothing to reclaim while the vdisk kept filling. docker system df
did name it — 88.5 GB under Containers, not Images or Volumes — which is the
column to read first the next time a docker vdisk fills for no visible reason.

Fix: tmpfs: ["/tmp:exec,mode=1777,size=2g"] in both compose files. A tmpfs is
remounted empty on every container start, so the leak is bounded by construction
rather than by remembering to clean it. Full write-up:
Knowledge Wiki container → the /tmp leak.

Two things generalize:

• A workaround that restarts a container on every operation inherits everything
  that process fails to clean up on the way down. The restart was the right call
  for NFS; the leak was the bill for it, and it arrived two months later.
• unless-stopped on a hand-run container outlives repo-wide sweeps.
  knowledge-test‘s compose lives on the NAS share, not in homelab-containers, so
  commit bb56779 — “restart: always on all 15 remaining unless-stopped stacks” —
  never saw it. A sweep is only as complete as the set of files it can enumerate.

2026-09-05 — a “throwaway” preview kept a root-equivalent socket mounted for a month

knowledge-test, the staging viewer, was stood up as a disposable redesign preview and
hand-run from a compose.yml on the NAS share. It never got adopted into Komodo, and
two costs accrued quietly while it stayed load-bearing:

• knowledge-builder mounted /var/run/docker.sock purely so a stage publish could
  docker restart knowledge-test. That socket is root-equivalent on SpaceDock — code in
  that container could start a privileged container and mount the host filesystem. The
  compose comment even said “drop it if knowledge-test is ever adopted into Komodo”,
  which is the tell: the cost was known and written down, and that was mistaken for
  handling it.
• Repo-wide sweeps could not see it. bb56779 set restart: always on “all 15
  remaining unless-stopped stacks” and missed this one, so the staging viewer would not
  have come back after a clean reboot.

Adoption was small — version the compose, add a [[stack]] block, flip reload_kind
from "docker" to "komodo" — because the Profile already branched on that field and
the KB_KOMODO_* credentials were already present for prod. The whole privilege existed
to reach a code path that a two-line config change made redundant.

Sequencing that mattered: the socket came out in a second commit, after a real stage
publish was observed reloading the viewer through Komodo. Had the API path been wrong,
the old one would still have been there to fall back to.

What nearly produced a false alarm: Komodo’s /execute is asynchronous, and
_komodo_restart() only checks the HTTP status. The first verification checked the
container’s StartedAt five seconds after the publish, saw it unchanged, and read as a
failure — the restart actually landed ~25 s later. reloaded: true means the restart was
requested, not that the viewer is serving again. Confirm an async job against the
executor’s own record (Komodo’s update log) before believing a short poll.

Two things generalize:

• “Throwaway” is a property of intent, not of the system. Nothing re-examines a
  temporary thing once it acquires dependents. This one was named _testsite, described
  as a throwaway on its own wiki page, and had been the mandatory review gate for every
  publish for two months.
• A privilege granted for one narrow call is worth re-checking whenever the thing it
  reaches changes shape. The socket was justified when knowledge-test was outside
  Komodo; nothing prompted a re-read of that justification once it no longer needed to be.

What Worked Well

• Keeping the content repo (knowledge) separate from the infra repo (homelab-containers) means wiki content commits don’t pollute the container changelog and vice versa
• NFS writable from SpaceDock confirmed before deploying the container — test this early on any new NFS-backed container
• No nav: block in mkdocs.yml means no manual config updates when adding new projects or sections

The Add-Content Workflow

# 1. Write files to /Volumes/knowledge/docs/...
# 2. Commit and push
kcommit "added: description of what you added"
# 3. Restart the container so MkDocs re-reads the NFS mount
#    Komodo → Stacks → knowledge → Restart

What to Do Differently Next Time

• Run git config --global --add safe.directory <mount> and create .gitignore before git init on any NFS share
• Create the Gitea repo before running git remote add + git push
• Rename branch to main immediately after git init
• Do not use a nav: block — let MkDocs auto-discover