Operator runbook — Cloudron tokens¶

How the co-op's Cloudron credentials are organized, and how to rotate them without breaking the stack.

The two-token split¶

There are two separate Cloudron API tokens, deliberately decoupled so rotating one doesn't break the other:

Token	Purpose	Where it lives
Agent token	Inferencebot's own Cloudron access	The agent's private `.env` (gitignored)
Portal token	The member portal's Cloudron access (create users, set groups, invite links)	Cloudron's env store, under the portal app's `CLOUDRON_TOKEN`

The portal needs only a handful of Cloudron capabilities (user/group management, invite links). It is the most exposed app (public webhook + broker endpoints), so it must not hold the agent's token — a portal compromise shouldn't reach the agent's wider Cloudron access.

Rotating a token¶

Portal token¶

In my.inference.coop → Settings → API Tokens, create a "Read and Write" token.
Set it as the portal's CLOUDRON_TOKEN env var (custom apps have no env-var UI — use the Cloudron CLI or the API; see the next section).
Verify: the portal can reach GET /api/v1/users with the new token (HTTP 200).

Agent token¶

Create a fresh token in Settings → API Tokens.
Put it in the agent's .env as CLOUDRON_TOKEN.
Verify against /api/v1/cloudron/status.

The trap this split exists to prevent: the portal keeps its own copy of a Cloudron token. If the agent's token is rotated without also updating the portal copy, the portal's Cloudron calls start returning 401 — which breaks provisioning (new members can't be created) and the member dashboard (its usage/keys data is resolved through the portal's Cloudron calls).

Admin token vs webhook token (portal)¶

Two more, separate from the Cloudron tokens above:

Token	Purpose	Where it lives
`ADMIN_TOKEN`	Authenticates `/admin/*`; sent ONLY as an `X-Admin-Token` header	Portal env; the admin panel's own env has a separate copy; `~/.hermes/.env` on the agent host
`WEBHOOK_TOKEN`	Authenticates Open Collective's inbound webhook	Portal env, and inside the webhook URL configured on the collective

They were once one value, so anything that learned the webhook URL — OC admins, OC's logs, our access logs, a Referer — got portal admin. Split 2026-10-27; the cutover completed 2026-10-28 (ADMIN_TOKEN_LEGACY=0), verified on the wire: ADMIN_TOKEN → HTTP 200, WEBHOOK_TOKEN → 403, no header → 403.

WEBHOOK_TOKEN is not retired and cannot be. OC's generic webhooks cannot send headers, so a secret in the URL is the standard authentication. Its only power now is authenticating that one inbound route, which is the intended end state.
Rotating ADMIN_TOKEN touches THREE places — the portal env (shared by the app and the scheduler addon), the admin panel's separate copy, and the agent's ~/.hermes/.env. Miss one and that caller 403s silently.
WEBHOOK_TOKEN was rotated on 2026-10-27, as a precaution after a test handled the URL. Rotating it again means updating BOTH the portal env and the webhook URL configured on the collective, which needs Open Collective admin access — the same blocker as moving OC_PERSONAL_TOKEN (below) to a co-op-owned account.
Whether the collective's webhook URL matches the current token is still unconfirmed (as of 2026-10-28). The route compares the URL token against the env var, so a mismatch means OC's deliveries 401 and new-member provisioning waits for the nightly sweep — delayed, not lost.
To confirm it directly, compare by hash rather than by reading the value out: take the URL's last path segment in the collective's webhook settings and run printf %s '<token>' | sha256sum LOCALLY, then compare its first 12 hex characters against the same prefix of the portal's WEBHOOK_TOKEN. A hash prefix of a live secret does not belong in this file or in chat.
The monitor already covers the failure case. The portal webhook delivery watch cron job alerts on ANY failed delivery (401/404 = OC is posting to a URL the portal rejects) and stays silent on the happy path, which is why silence in its output means "nothing to report" and not "blind". Verified 2026-10-28 by posting a benign event with the current token: the portal answered 200 and the job reported it with --verbose. Run it with --verbose to see accepted deliveries; run it plainly to see only problems.
No caller may fall back from ADMIN_TOKEN to WEBHOOK_TOKEN. The legacy token 403s, so a fallback turns "unset" into a confusing mid-run failure instead of a clear abort.

Chat image: spaCy model for document parsing (EPUB/DOCX)¶

Open WebUI's document parser (unstructured) lazily downloads the spaCy model en_core_web_sm on first EPUB/DOCX parse — but the app directory is read-only under Cloudron, so that install always fails ("Errno 30 Read-only file system"). Fixed 2026-10-14 (issue #20) by pre-installing the model into persistent storage:

/app/data/spacy_model/ holds the unpacked wheel (SHA-verified en_core_web_sm-3.8.0-py3-none-any.whl from the pinned URL in unstructured/nlp/tokenize.py).
App env has PYTHONPATH=/app/data/spacy_model so import en_core_web_sm resolves and the lazy download never triggers.

Restore caveat: a backup restore from before 2026-10-14 will NOT contain the model dir — re-create it (download the pinned wheel, unzip into /app/data/spacy_model/, verify spacy.load("en_core_web_sm") works in the container). The PYTHONPATH env survives restores (it's in app config).

Scheduled tasks (scheduler addon)¶

The portal and gateway run their own housekeeping via the Cloudron scheduler addon, declared in each app's CloudronManifest.json under addons.scheduler:

portal: membership_sweep (nightly 3am, /app/code/sweep.sh → POST /admin/sweep with the ADMIN_TOKEN header — NOT WEBHOOK_TOKEN, which returns 403 on /admin/* since the 2026-10-28 cutover), rag_refresh (nightly 3:30am, /app/code/rag_refresh.py — pulls docs from the Gitea raw API and refreshes the chat knowledge base; needs OWUI_API_KEY + DOCS_KNOWLEDGE_ID in the app env), and metrics_snapshot (nightly 3:15am, /app/code/metrics_snapshot.py — writes the day's members / usage / energy / carbon row into the portal's metrics_daily table; --backfill rebuilds history and is idempotent).
gateway: audio_keepalive (every 10 minutes, /app/code/audio_keepalive.py — pings the Tinfoil audio models locally to avoid the ~24s TEE cold-start; reads the master key from /app/data/.master_key, which start.sh writes — the env var LITELLM_MASTER_KEY is only exported into the litellm process).

After any manifest change, verify the scheduler survived the deploy: the build log must show setupAddons: setting up addon scheduler with options {...} listing your task(s), and each script must succeed when run manually inside the container (cloudron exec --app <id> -- <command>). The scheduler has historically gone missing from the manifest during edits — a deploy without the setupAddons line means nightly reconciliation is NOT running.

Custom apps have no env-var UI¶

The member portal is a custom app (packaged by us), not an App Store app. Custom apps do not expose environment variables in the Cloudron dashboard. Env vars for custom apps are set via:

Cloudron CLI (on your workstation, not the server): cloudron env set --app portal.inference.coop CLOUDRON_TOKEN=...
Cloudron API: POST /apps/:appId/configure/env

`configure/env` is a full replace¶

POST /apps/:appId/configure/env with body {"env": {KEY: value, ...}} replaces the entire environment — it does not merge. Sending a partial map (e.g. a single test variable) wipes every other variable.

To change one variable, read the current env first (GET /api/v1/apps/:id → .env), merge in memory, and POST the full map back.
Never probe the endpoint with a throwaway body against a live app.
If the env is wiped, it is recoverable from a Cloudron app backup: GET /api/v1/apps/:appId/backups → newest entry → GET .../backups/:backupId/download (a .tar.gz containing config.json, whose .env field holds the original map).

Reference¶

Cloudron CLI: npm install -g cloudron (install on your workstation, not the server), then cloudron login my.inference.coop.
Cloudron API docs: docs.cloudron.io/api/.

Open Collective: the bot token needs the `email` scope (Sept 2026)¶

The portal identifies members by email, read via a personal OC token (OC_PERSONAL_TOKEN) — currently Nathan's own rather than a co-op-owned account's (open item F5); the fix is a co-op account plus a token minted there as collective.members.nodes[].account.email with the ... on Individual { email } inline fragment.

The email scope is load-bearing. The bot's personal token must be minted with the email scope. Without it, the GraphQL email field returns null for every member — including the bot's own me { email } — even though:

the bot still authenticates and reports isAdmin: true, and
the OC admin CSV export still shows emails (it's a separate, non-GraphQL path; its contributorEmail column is not a GraphQL field).

Symptoms when the scope is missing: fetch_members() filters out empty emails and returns [], so the webhook can't provision anyone by email and the nightly sweep goes into its empty-list fail-safe (skips, doesn't deactivate). A new member's webhook fires cleanly but no "Sent email to …" log line follows, and no Cloudron user is created. (This is easy to misread as a "guest with no email" problem — it is not; verify the scope first.)

Diagnosis: run me { email } with the bot token. null here while isAdmin: true → the token is missing the email scope (or the scope was revoked). Fix: re-add the email scope to the token on Open Collective (Settings → Developers → Personal Tokens). No token rotation, no re-granting of admin — just the scope.