Sanitized copy. Hostnames are placeholders and credential-store and cluster-access details are omitted.
How the Hermes agent platform is deployed, what it talks to, and — the part no diagram
usually shows — which half of it Flux actually owns. The pod shell is declared in git.
The profiles, cron jobs, skills, webhook routes and memory wiring live on the PVC and are
invisible to a git pull.
gitops repo · Flux-reconciled.clusters/home-clst-01/hermes.yaml
Three Flux Kustomizations in a dependency chain, all prune: true and
wait: true. Splitting the network phase off means an HTTPRoute never
applies before the Service it points at exists.
%%{init: {'theme':'base','themeVariables':{'fontSize':'12px','lineColor':'#5a6675','textColor':'#242c36','primaryTextColor':'#171c24','primaryBorderColor':'#8b98a8','primaryColor':'#dfe5ec','clusterBkg':'#e3e8ef','clusterBorder':'#b7c0cc','edgeLabelBackground':'#eef1f5'}}}%%
flowchart LR
akv[("Azure Key Vault")]:::ext
ir["Docker Hub
nousresearch/hermes-agent"]:::ext
repo["gitops repo
home-clst-01/apps/hermes/"]:::git
ip["ImagePolicy hermes-agent
semver v2026.x.y"]:::git
ir --> ip
ip -- "rewrites $imagepolicy marker
commits back" --> repo
akv -- "ClusterSecretStore" --> eso
subgraph flux["Flux apply order"]
direction LR
k0["external-secrets"]:::git
k1["hermes-prerequisites
ns + ~26 ExternalSecrets"]:::git
k2["hermes
deploy-app/"]:::git
k3["hermes-network
deploy-network/"]:::git
k0 -- dependsOn --> k1 -- dependsOn --> k2 -- dependsOn --> k3
end
repo --> k1
eso["ExternalSecrets → k8s Secrets"]:::git --> k1
k2 --> out1["StatefulSet hermes (1 replica)
Deployment camofox-browser
PVC hermes-data · SA hermes"]:::git
k3 --> out2["3 HTTPRoutes → hermes-gateway"]:::git
classDef git fill:#d9eaf3,stroke:#1f6684,color:#0f2c39;
classDef pvc fill:#f6e6cb,stroke:#8f5510,color:#402c08;
classDef ext fill:#e3dff1,stroke:#574a80,color:#2b2545;
5m. The image tag is
rewritten into git by Flux image-automation, so the running version is always readable
from the StatefulSet manifest. Four-part hotfix tags are skipped (not valid semver).
| Field | Value | Why it's set that way |
|---|---|---|
image | nousresearch/hermes-agent:v2026.8.3 | Flux-automated, 3-part semver only |
args | gateway run | Serves all platforms for every profile |
| requests | 3Gi / 500m | Burstable far above request = first evicted; request tracks real usage (~750Mi idle) |
| limits | 8Gi / 2 CPU | One cgroup holds the gateway and local ACP runs — an OOM takes every platform down |
fsGroup | 1000, OnRootMismatch | Skips a ~113s recursive chown of ~430k inodes on every start |
HERMES_HOME_MODE | 2770 | Hermes chmods HOME to 0700 each boot, which would defeat OnRootMismatch. Setgid also fixes root-owned files left by kubectl exec |
startupProbe | 60 × 10s on :9119 | Cold boot exceeds the liveness budget; without it a slow start gets killed and restarted |
| PVC | hermes-data 20Gi rook-ceph-block | Custom finalizer hermes.io/data-protection blocks accidental delete |
ns hermes
One pod, three listeners, three hostnames. The path split on
hermes.example.net matters: /webhooks goes to a different
port than everything else, and that port is authenticated by per-route HMAC rather than
by Entra.
%%{init: {'theme':'base','themeVariables':{'fontSize':'12px','lineColor':'#5a6675','textColor':'#242c36','primaryTextColor':'#171c24','primaryBorderColor':'#8b98a8','primaryColor':'#dfe5ec','clusterBkg':'#e3e8ef','clusterBorder':'#b7c0cc','edgeLabelBackground':'#eef1f5'}}}%%
flowchart TB
browser["Browser"]:::ext
tui["TUI / desktop app"]:::ext
fj["Forgejo (que-clst-01)
repo webhook"]:::ext
gw["Gateway hermes-gateway
envoy-gateway-system"]:::git
browser --> gw
tui --> gw
fj -- "action_run_failure" --> gw
gw -- "hermes.example.net /" --> p9119
gw -- "hermes-gui.example.net" --> p9119
gw -- "hermes-tui.example.net" --> p8642
gw -- "hermes.example.net /webhooks" --> p8644
subgraph pod["hermes-0"]
direction TB
p9119["dashboard :9119
Entra OIDC + offline_access"]:::git
p8642["api_server :8642"]:::git
p8644["webhook :8644
per-route HMAC"]:::pvc
end
pod --> pvc[("hermes-data 20Gi
config · profiles · sessions
skills · cron · memories")]:::pvc
pod --> shm["/dev/shm emptyDir 1Gi
Chromium shared memory"]:::git
pod --> kc["kubeconfigs Secret (ro)
home · que · ash"]:::git
pod --> sshk["ssh key Secret (ro)
+ /etc/ssh/ssh_config.d"]:::git
pod -- "CAMOFOX_URL" --> cam["camofox-browser :9377
headed anti-bot browsing"]:::git
classDef git fill:#d9eaf3,stroke:#1f6684,color:#0f2c39;
classDef pvc fill:#f6e6cb,stroke:#8f5510,color:#402c08;
classDef ext fill:#e3dff1,stroke:#574a80,color:#2b2545;
platforms.webhook.enabled: true in the PVC config; the single live route,
forgejo-ci, was created with hermes webhook subscribe and lives
in /opt/data/webhook_subscriptions.json with its own HMAC secret. Deleting
the PVC deletes the route; redeploying from git does not restore it.
Never enable a Hermes platform through StatefulSet env. WEBHOOK_ENABLED and
the mere presence of API_SERVER_KEY force-enable that platform in
every profile gateway, overriding an explicit enabled: false; they
then race for the port and boot order picks the winner. That's why
hermes-api-server-key is bound only to
HERMES_DASHBOARD_SESSION_TOKEN and is deliberately absent from
envFrom.
ns mcp
Thirteen MCP servers reach Hermes through one aggregator —
agentgateway v1.4.1 on
:9090, one route per backend at /<name>/mcp. It
health-checks each backend and re-lists that backend's tools on reconnect, so a crashed
MCP server self-heals without touching the other twelve.
%%{init: {'theme':'base','themeVariables':{'fontSize':'12px','lineColor':'#5a6675','textColor':'#242c36','primaryTextColor':'#171c24','primaryBorderColor':'#8b98a8','primaryColor':'#dfe5ec','clusterBkg':'#e3e8ef','clusterBorder':'#b7c0cc','edgeLabelBackground':'#eef1f5'}}}%%
flowchart LR
hermes["hermes-0"]:::git
cc["Claude Code
(operator workstation)"]:::ext
cc -- "mcp.example.net
via internal-gateway-01" --> agw
hermes -- "in-cluster :9090" --> agw
agw["mcp-gateway :9090
agentgateway v1.4.1
listener apiKey · strict"]:::git
subgraph incl["in ns mcp"]
direction TB
b1["mcp-grafana :8000"]:::git
b2["mcp-k8s-home / que / ash :8080"]:::git
b3["playwright-mcp :8931"]:::git
b4["mcp-panos :8000"]:::git
b5["mcp-cisco-9300 :3030"]:::git
b6["mcp-mikrotik :3000"]:::git
b7["mcp-unifi-network :3000"]:::git
ov["openviking :1933
v0.4.13 · PVC 20Gi"]:::git
end
ha["home-assistant :8080
(ns home-assistant)"]:::git
cf["mcp.cloudflare.com
docs.mcp.cloudflare.com"]:::ext
agw --> b1 & b2 & b3 & b4 & b5 & b6 & b7
agw --> ha
agw --> cf
agw -- "/openviking/mcp
for cron only" --> ov
hermes -- "REST :1933 direct
memory provider
netpol openviking-from-hermes" --> ov
ovui["openviking.example.net
Web Studio + REST admin"]:::ext
ovui --> o2p["oauth2-proxy :4180
Entra roles"]:::git --> ov
classDef git fill:#d9eaf3,stroke:#1f6684,color:#0f2c39;
classDef pvc fill:#f6e6cb,stroke:#8f5510,color:#402c08;
classDef ext fill:#e3dff1,stroke:#574a80,color:#2b2545;
:1933 directly — agentgateway cannot front it, because the
provider is not an MCP client and the listener's apiKey policy owns the
Authorization header the provider needs for its own key. The
/openviking/mcp route exists anyway because agent-backed cron skips memory
providers entirely, and the default profile runs 15 cron jobs.
Authorization: <token> — one full-access, one for Hermes — covering
both the in-cluster path and the Envoy path with a single policy.default/default. Root keys are refused on every tenant-scoped REST data API
and work only on the MCP surface, which is exactly what hid this for months.OPENVIKING_AGENT=hermes lands Hermes's
writes under peers/hermes/memories/; Claude Code writes the user root.hermes → :1933 NetworkPolicy hole also reaches the REST admin
API, bypassing the Entra oauth2-proxy. The USER key is the only boundary there./opt/data — not in git
Five agent scopes: the root/default profile at /opt/data/config.yaml, plus
four named profiles under /opt/data/profiles/. Each has its own independent
config.yaml that never re-syncs from the git template once created — editing
the ConfigMap in the repo does nothing to a live profile.
%%{init: {'theme':'base','themeVariables':{'fontSize':'12px','lineColor':'#5a6675','textColor':'#242c36','primaryTextColor':'#171c24','primaryBorderColor':'#8b98a8','primaryColor':'#dfe5ec','clusterBkg':'#e3e8ef','clusterBorder':'#b7c0cc','edgeLabelBackground':'#eef1f5'}}}%%
flowchart TB
subgraph profiles["Profiles on the PVC"]
direction TB
def["default
telegram · discord · webhook · api_server
15 cron jobs"]:::pvc
gen["general
chat scope"]:::pvc
kan["kanban-orchestrator
own gateway process
1 cron: alert-watch /20m"]:::pvc
wl["claude-code-worker-local
terminal.backend: local"]:::pvc
wr["claude-code-worker-remote
terminal.backend: ssh"]:::pvc
end
ds["DeepSeek API
deepseek-v4-flash"]:::ext
def --> ds
gen --> ds
kan --> ds
wl -- "copilot-acp" --> acpl["claude-agent-acp
in-pod (npm-global)"]:::pvc
wr -- "copilot-acp" --> sh["/opt/data/scripts/
claude-acp-remote.sh"]:::pvc
sh -- "ssh hermes@ · stdio JSON-RPC" --> dev["build-host-01
claude-agent-acp"]:::ext
acpl --> sub["Claude subscription
CLAUDE_CODE_OAUTH_TOKEN"]:::ext
dev --> sub
kan -- "dispatch every 60s
default assignee" --> wr
fb["Fallbacks, in order
1 modelrelay/nemotron-3-super-120b
2 openrouter/deepseek-v4-flash
3 kilocode/claude-haiku-4.5"]:::git
ds -. "on failure" .-> fb
fb --> mr["modelrelay :7352
ns modelrelay"]:::git
classDef git fill:#d9eaf3,stroke:#1f6684,color:#0f2c39;
classDef pvc fill:#f6e6cb,stroke:#8f5510,color:#402c08;
classDef ext fill:#e3dff1,stroke:#574a80,color:#2b2545;
model.default: copilot-acp alongside the provider — without it the dashboard
pairs provider: copilot-acp with the ambient DeepSeek model and produces
hung, zero-token replies while the CLI -p path still works. The remote worker
pipes ACP stdio over SSH to build-host-01; heavy builds belong there, not in
the pod's 8Gi cgroup.
| Profile | Provider / model | Inbound | MCP | Memory |
|---|---|---|---|---|
default | deepseek · deepseek-v4-flash | telegram, discord, webhook, api_server, dashboard | 13 (incl. openviking) | openviking |
general | deepseek · deepseek-v4-flash | CLI only | 12 | openviking |
kanban-orchestrator | deepseek · deepseek-v4-flash | none (cron) | 12 | openviking |
claude-code-worker-local | copilot-acp · in-pod | none (CLI subprocess) | 12 | openviking |
claude-code-worker-remote | copilot-acp · ssh build-host-01 | none (CLI subprocess) | 12 | openviking |
The named profiles get 12 MCP servers; only default also gets
openviking as an MCP server, because its cron jobs bypass the memory provider
and need the tools instead.
gateway.multiplex_profiles is false and should stay that way.
It only decides whether a secondary profile gets its own inbound channel; it does
not affect running one. Turning it on installs a fail-closed per-profile secret scope, so
every pod-wide envFrom secret resolves to an empty string — silently. It was
on for two days in July and killed Telegram and Discord inbound while outbound kept
working.
cron · webhook · kanban
Three intake paths, none of them declared in the repo. Cron jobs live in
<profile>/cron/jobs.json, webhook routes in
webhook_subscriptions.json, and the kanban dispatcher runs inside the gateway
process itself.
%%{init: {'theme':'base','themeVariables':{'fontSize':'12px','lineColor':'#5a6675','textColor':'#242c36','primaryTextColor':'#171c24','primaryBorderColor':'#8b98a8','primaryColor':'#dfe5ec','clusterBkg':'#e3e8ef','clusterBorder':'#b7c0cc','edgeLabelBackground':'#eef1f5'}}}%%
flowchart LR
fj["Forgejo CI failure"]:::ext -- "HMAC POST /webhooks/forgejo-ci" --> wh["webhook platform :8644"]:::pvc
wh -- "toolset hermes-cli
approvals: off" --> agent["agent session"]:::pvc
agent -- "opens hermes/fix-* PR
merges only if green" --> fjr["Forgejo repo"]:::ext
agent -. "cannot fix" .-> po["Pushover"]:::ext
cron["default profile
15 cron jobs"]:::pvc --> agent
aw["kanban-orchestrator
alert-watch every 20m"]:::pvc -- "pulls firing alerts
via Grafana AM proxy" --> graf["Grafana"]:::ext
aw -- "one kanban task per
new alert fingerprint" --> board["kanban board"]:::pvc
board -- "dispatch_in_gateway
every 60s" --> wrk["claude-code-worker-remote"]:::pvc
sync["skills-repo-sync every 10m"]:::pvc <--> ops["hermes-ops repo
self-hosted Forgejo · hermes-ops"]:::ext
classDef git fill:#d9eaf3,stroke:#1f6684,color:#0f2c39;
classDef pvc fill:#f6e6cb,stroke:#8f5510,color:#402c08;
classDef ext fill:#e3dff1,stroke:#574a80,color:#2b2545;
/webhooks
path was torn down in July: it cost an HMAC rotation on every prompt edit plus a relay app
on a second cluster, for an outcome a 20-minute cron job gets with infrastructure Hermes
already had.
hermes-ops is not a workload. It's a Forgejo repo holding the imperative half
of Hermes's config — skills and cron job definitions — cloned to
/opt/data/hermes-ops on the PVC. The skills-repo-sync cron job runs
sync_skills.py every 10 minutes: git pull --rebase (abort and
Pushover on conflict), then live→repo commit and push behind a sensitive-file guard, then
repo→live copy. Newer mtime wins. It's the closest thing the PVC half has to version control.
the boundary
The single most expensive misunderstanding about this deployment is assuming a
git pull shows you the whole thing.
| Concern | Home | Changed by |
|---|---|---|
| StatefulSet, Service, PVC, routes | gitops repo | commit + Flux |
| Secrets | a cloud secret store | AKV, surfaced by ESO every 5m |
| Image tag | gitops repo | Flux image-automation commits it |
Root config.yaml template | gitops ConfigMap | renders to the default profile only |
| Named profile configs | PVC | hermes -p <name> config set |
| Cron jobs | PVC (+ mirror in hermes-ops) | hermes cron create |
| Skills | PVC (+ mirror in hermes-ops) | edited either side, mtime-newer wins |
| Webhook routes & HMAC secrets | PVC | hermes webhook subscribe |
| OAuth credentials | PVC .claude/.credentials.json | login flow |
memory.provider activation | PVC, per profile | hermes config set — env alone activates nothing |
kubectl exec lands you as root while the gateway runs as uid 1000. Every
direct PVC edit needs chown -R hermes:hermes afterwards, or the next write
fails with permission denied. And a freshly created profile only gets its gateway
registered at container boot — a new profile needs a pod restart (or a manual
s6-svc -u) before it will serve or run cron.