2026-10-02 — gateway outage after a security-upgrade rebuild

Written by InferenceBot on the incident/2026-10-02-gateway-cve-upgrade branch on 2026-10-02 (05:35 UTC, a minute after service returned; the commit's local timestamp reads 2026-10-01 23:35 −06:00); moved to main with the substance unchanged so it's findable.

Impact. The gateway (chat and API) was down about 45 minutes, 04:50–05:34 UTC. No data loss.

What happened. A Cloudron update triggered by a member's CVE report rebuilt the app image. The Dockerfile pinned the floating main-v1.74.0-stable tag, which the registry had silently moved to newer (v1.84.0-era) bits, so the rebuild changed the running version and kicked off a full Prisma database migration that takes 10–15 minutes per boot with internal timeouts and retries. The migration ran at Cloudron's default 256 MB memory cap and was OOM-killed; the app crash-looped on a missing migrations ledger until the memory limit and LITELLM_MIGRATION_DIR were fixed.

Also during the incident. The domain's authoritative nameservers (yoursrs.com) were intermittently unreachable, briefly failing resolution of some hostnames (git, portal). Transient, registrar-side, self-recovered; no records were wrong. The upgrade had also silently reverted CORS to *.

Repairs. Version pinned to the exact tag in code/litellm; memoryLimit set to 2 GB in the manifest; LITELLM_MIGRATION_DIR pointed at the persistent volume; LITELLM_CORS_ORIGINS set to the chat origin; packaging synced to mirror the live deployment. The upgrade rules learned here are in the code/litellm README under "Updating LiteLLM".

What it fixed on the security side. CVE-2026-35029 (authorization bypass on /config/update, fixed in 1.83.0) and CVE-2026-59822 (MCP session auth bypass, fixed in 1.84.0, listed in CISA's Known Exploited Vulnerabilities catalog). The gateway has since moved to v1.103.2.