Server Administration

This document covers Yuzu server deployment, configuration, and ongoing administration. It is intended for operators who install, configure, and maintain the Yuzu server.

Table of Contents

1. Server CLI Flags
2. Configuration Files
3. First-Run Setup
4. Settings Page
5. TLS Configuration
6. User Management
7. Agent Enrollment
8. OTA Agent Updates
9. RBAC Management
10. Tag Compliance
11. OIDC SSO Configuration
12. SAML 2.0 SP Configuration
13. Directory Sync (AD/Entra ID)
14. Data Storage and Encryption
15. PostgreSQL Substrate
16. NVD CVE sync
17. Retention Settings
18. Settings API Reference
19. Deployment
20. Windows Service Installation
21. Planned Features

Server CLI Flags

The Yuzu server binary accepts the following command-line flags. All flags are optional; defaults are shown in the table.

Sizing the Guardian reconcile budget

Read this first. The admission budget removes the heartbeat reconcile’s O(fleet) cost (the scope check is now evaluated for the one agent, not built from a pass over the whole fleet) and makes it shed-safe: under load a replica declines excess reconciles instead of falling over. It does not by itself make the server handle 500,000 agents. The default values of the three flags are provisional starting points, not derived from fleet measurement, and every per-push cost below is an estimate unless it says otherwise. The policy-bump fan-out that the REST and dashboard routes run (as distinct from this heartbeat path) is not budgeted and runs synchronously. Per-heartbeat Postgres writes, the gateway forwarding of each push (a thread per forwarded push), audit volume and a cluster-wide (rather than per-replica) budget are separate follow-ups.

After a policy-generation bump or a fleet-wide restart, every agent that reports a generation behind the server’s is re-pushed from its heartbeat (see Guaranteed State). The three --guardian-reconcile-* flags bound how fast one replica does that.

• What you need. Pushes per second fleet-wide = agents / convergence seconds. For 500,000 agents to converge within 5 minutes (300 s) that is about 1,700 pushes per second across the fleet, so about 1,700 / replicas per replica.
• What a replica admits. min(--guardian-reconcile-max-per-s, --guardian-reconcile-budget-ms-per-s / ms per push). An admitted push is charged its wall time minus ONLY the time it spent blocked waiting on another thread’s in-flight rule-set load, and at least 1 ms. Everything else is charged: the fresh policy-generation read, the per-agent scope reads, and the push’s own rule-set load when it is the thread that runs it. Read the elapsed cost from the yuzu_server_guardian_reconcile_duration_seconds histogram (see Metrics), not from a guess; a push that waited on another thread’s load reads higher there than the cost the budget charges. Postgres load from reconciles is the budget times the number of replicas. The in-flight cap of 256 is fixed, and soft by the number of heartbeat threads that are between the admission decision and the start of the push (the count rises when the push begins, not when it is admitted), so briefly more than 256 can be in flight; the default Postgres pool is small (--postgres-pool-size, default 16; the Guardian store reads wait at most 2 s for a lease), so about 256 concurrent pushes can queue on 16 leases; that is why the wall-time debit and the degraded counter together are the pool-pressure signal. A reconcile withheld because a rule or policy change landed while it was being prepared is deliberately not counted as degraded; it has its own counter, yuzu_server_guardian_reconcile_invalidated_total, which is the edit-churn signal (see “Delay after a bump”). To see saturation directly, watch the per-replica gauges yuzu_server_guardian_reconcile_budget_balance_ms (negative is debt: the replica is shedding until it is repaid), yuzu_server_guardian_reconcile_in_flight (against the cap of 256) and yuzu_server_guardian_reconcile_claims, all 15 s snapshots published on the serial health_recompute thread (a peak between samples is missed, including for in_flight; the metrics reference gives example queries that tell which bound is shedding).
• Estimated cost. An admitted push is estimated at 3 to 8 ms, dominated by Postgres round trips and the synchronous audit INSERT. This has not been measured end to end, and it assumes a policy with few distinct scope expressions: scope membership is evaluated once per DISTINCT scope expression among the rules that apply to the agent, and each does up to 2 point reads (the agent’s tag map and its property map). A policy with many distinct scoped rules therefore costs more per push and raises lease pressure on the Postgres pool, which can surface as spurious scope store degraded aborts (result="degraded"). At that cost the default 500 ms/s budget admits roughly 60 to 170 pushes per second per replica.
• How long convergence takes. Roughly agents / (replicas x admitted rate per replica). Example, assuming 1 ms per push (an assumption for illustration, not a measurement): at the defaults one replica admits min(1000, 500 / 1) = 500 pushes per second, so 3 replicas admit about 1,500 per second and 500,000 agents converge in about 330 s (5.5 minutes), assuming the push cost stays 1 ms. At the estimated 3 to 8 ms, converging 500,000 agents in 5 minutes (about 1,700 pushes per second) needs roughly 10 to 28 replicas at the default budget, or a larger budget (raise it only while the histogram stays flat and Postgres has headroom). On a single replica at 5 ms (100 pushes per second), 500,000 agents take about 83 minutes. Size for N-1 replicas, because the survivors absorb a failed replica’s agents without their budgets changing.
• The count ceiling is a one-second tumbling window. A phase-aligned burst (for example one gateway batch carrying many agents) admits at most --guardian-reconcile-max-per-s reconciles in that window; the rest are shed and wait for their next heartbeat. Effective convergence for aligned fleets can therefore be slower than budget / ms per push suggests.
• Fixed limits that are not flags. A replica sheds while 256 admitted pushes are in flight; this in-flight cap also limits how many threads can be inside the fresh policy-generation read at once. The cap is soft: a push counts as in flight from the moment it begins, not from the admission decision, so the overshoot is at most the number of heartbeat threads that sit between those two points at the same instant. It is not a hard ceiling of 256. Each admitted push is charged at least 1 ms. The rule set is loaded once per policy generation per replica and reloaded after at most 60 s. A failed load is remembered for 1 s, so a failing store is probed about once a second rather than by every admitted push. If an expired (60 s or older) rule set cannot be reloaded, the reconcile aborts (result="degraded", fail closed) rather than serving the stale rule set. A thread waiting on another thread’s rule-set load, or on the shared cached generation read, gives up after 5 s; the fresh policy-generation read an admitted push makes is NOT covered by that 5 s bound, only by the Postgres lease and statement timeouts. The budget balance never falls below -30 x --guardian-reconcile-budget-ms-per-s milliseconds (-15000 ms at the default; a debt of at most 30 seconds of budget), so after a long slow episode (a slow Postgres, say) the replica sheds until the bucket refills, up to about 30 s.
• Consequences of the design (not measurements). While a rule-set load keeps failing slowly, the negative cache turns the next 1 s of admitted pushes into fast aborts (one ABORTED audit row each, still at most one per agent per 25 s). The loading thread’s own load time is charged to the budget, so a load of L seconds can shed the replica for about 2L seconds at the default budget (capped by the 30 s debt floor), and this recurs each time the bundle expires (60 s) while agents are behind.
• Delay after a bump. Each replica reads the policy generation through a cache of about 2 s (1 s after a failed read). The cache is refreshed by the first heartbeat that finds the entry expired, never by a timer, so a heartbeat that arrives while another thread is refreshing an expired entry is served the old value, which is as old as the 2 s plus however long the cache sat idle plus the fetch time. Exactly these operations drop that cache and the held rule set at once on the replica that performs them: a rule create, update or delete (the rule store calls a post-mutation hook after the edit committed and after the generation bump was attempted, whether or not the bump succeeded), any Baseline update (a deploy rewrites the deployed snapshot, but a rename or a draft edit counts too, because the store does not look at the old row), any Baseline delete, or a Baseline created already deployed (the Baseline store calls its own hook after the commit, before the caller’s bump and fan-out; a draft create does not fire it, and the Guardian routes only create drafts), a policy-generation raise that increased the stored value (the rule store calls the same hook after the raise committed; a raise that changed nothing does not fire it), and every Guardian push the replica fans out (a Baseline deploy, the delete of a deployed Baseline, an explicit REST or MCP push) once its scope has parsed; a push with an invalid scope is refused first and drops nothing. What that guarantees, and what it does not. Covered: a reconcile that is already running on that replica when one of these lands is withheld, not sent: it sends nothing, increments yuzu_server_guardian_reconcile_invalidated_total (it is not counted in the result="degraded" metric series, although its audit row does carry result=degraded), writes an ABORTED audit row (rule bundle invalidated during load) and keeps its 25 s claim, so the agent is admitted again once the claim is old enough (up to about 25 s, in practice the next heartbeat after that, about 30 s at the default interval) and is then reconciled from a fresh load. This also covers an edit whose generation bump failed, which would otherwise have let the old rule set go out under the unchanged generation. The check is conservative: any invalidation that overlaps a reconcile aborts it, including one whose bump succeeded. The aborts from one change are bounded by the reconciles in flight on the replica that took it at that instant: roughly the admitted rate times the mean reconcile duration (Little’s law). The charged reconcile time is capped at the --guardian-reconcile-budget-ms-per-s value divided by 1000 per second, which at the default 500 is about 0.5, for example 100 per second at 5 ms; but the in-flight count can exceed that by the waiters on a bundle reload, which is the common case right after an invalidation (their wait is excluded from the charge, which is floored at 1 ms), and a burst (a first-second burst reaching about 1.5) or a pool stall exceeds it too, up to the in-flight cap of 256. Read the measured yuzu_server_guardian_reconcile_in_flight gauge instead of the 0.5 figure; concurrency is bounded by the admitted rate times the reconcile duration, and by the in-flight cap of 256 (the default --guardian-reconcile-max-per-s of 1000 is above that cap, so the per-second ceiling alone does not bound it). The bound is per instant, and a Baseline deploy has two instants, the hook after the commit and then the push fan-out after the bump (whose own lease can take up to 4 s), so a deploy can withhold up to about twice that number. The audit store gets at most one row per agent per 25 s. A sustained stream of rule edits or pushes delays convergence: each one aborts the reconciles in flight at that moment, and an aborted agent waits out its claim, so convergence recovers about 25 s plus one heartbeat interval after the LAST change. Batch scripted edits (apply them, then make one push) instead of editing continuously. To watch it, graph increase(yuzu_server_guardian_reconcile_invalidated_total[10m]) against the audit rows of the operations that cause it (guaranteed_state.rule.create, guaranteed_state.rule.update, guaranteed_state.rule.delete, guaranteed_state.policy_generation.raise, guaranteed_state.push, guaranteed_state.baseline.deploy, and the other guaranteed_state.baseline.* writes) and yuzu_server_guardian_reconcile_in_flight: a few aborts per human edit is normal, and a continuous rate with no matching rows is a stuck loop (a script re-pushing, an automation re-saving rules). No shipped alert covers a stuck loop that leaves agents unconverged: these aborts are deliberately not counted in the result="degraded" series (their audit rows do carry result=degraded), so an alert on result="degraded" would not see them. A suggested example, not shipped: increase(yuzu_server_guardian_reconcile_invalidated_total[10m]) > 0 for three consecutive 10-minute windows with no matching rule or push audit row. To find the affected agents, query the audit log for action=guaranteed_state.reconcile (REST GET /api/v1/audit?action=guaranteed_state.reconcile&limit=1000) and match result of degraded and a detail containing rule bundle invalidated on the client side: the audit query has no filter on the detail text. That route honours only principal, action and limit (capped at 1000, newest first; there is no offset, since or until), so principal=system is optional (the action filter already selects these rows; it only narrows toward the system-initiated ones), but on a busy fleet older aborts are unreachable through REST: read them from the audit store or a SIEM copy. Not covered: (1) a change made through ANOTHER replica is noticed by this one after about 2 s, and if that replica’s bump failed this replica serves its held rule set until it is 60 s old, and an agent reconciled from it stays converged until the next bump; (2) a failure of both the bump and the local hook (a double fault) leaves the same 60 s window; (3) an agent that had already converged before the edit is not re-pushed by a failed bump, because its generation still matches; (4) the instants between a commit and its hook, and between the epoch check and the send, are invisible to the check, so a reconcile in those few instants can still send the old rule set under the old generation. The admit-time fresh read still labels each push with the current generation. An agent whose heartbeat lands inside the cache window is treated as converged and re-reports one heartbeat later, so the delay for such an agent is about 2 s plus one heartbeat interval (about 32 s at the default 30 s).
• Shed is normal under a wave. A shed agent is not an error: it retries on its next heartbeat (about 30 s at the default interval), and which agents are served first is not guaranteed.
• Each replica enforces its own budget. In an HA deployment the three flags apply per server replica, and so do the per-agent claims and caches; there is no cluster-wide budget. Two replicas can therefore both push one agent.

No alert ships for shed, because a threshold needs fleet data. The audit store receives at most one row per admitted push (audit sampling is a follow-up); an admitted push that ends without sending and without aborting (no_session, a fresh generation that is not ahead of the agent, or a failed queue write) writes none. An admission writes the agent’s 25 s claim and nothing releases it early, including when the push ends without sending, so an agent heartbeating generation 0 in a loop is admitted at most once per 25 s. result="no_session" counts only an agent whose session was removed (reaped, revoked or torn down) while a pending-map or gateway-held session id still resolved, or a deregistration race, and writes no audit row. An agent that has Registered but not yet Subscribed already HAS a session (it is installed at Register): it takes the full push path, the send fails, and it shows only as yuzu_server_system_reserved_push_total{result="undelivered"}. There is no server-side per-agent back-off beyond the 25 s per-agent limit: the agent-side back-off for a failed boot re-arm plus this budget cover it.

Example

# HTTP only (development — HTTPS is on by default, must opt out)
./yuzu-server --no-https --web-port 8080

# HTTPS with certificate files (default mode)
./yuzu-server --https-cert /etc/yuzu/server.crt \
  --https-key /etc/yuzu/server.key

# HTTPS with custom cert reload interval (30 seconds)
./yuzu-server --https-cert /etc/yuzu/server.crt \
  --https-key /etc/yuzu/server.key \
  --cert-reload-interval 30

# HTTPS with cert reload disabled
./yuzu-server --https-cert /etc/yuzu/server.crt \
  --https-key /etc/yuzu/server.key \
  --no-cert-reload

# MCP disabled (air-gapped environment)
./yuzu-server --https-cert /etc/yuzu/server.crt \
  --https-key /etc/yuzu/server.key \
  --mcp-disable

# MCP read-only mode (AI can query but not execute)
./yuzu-server --https-cert /etc/yuzu/server.crt \
  --https-key /etc/yuzu/server.key \
  --mcp-read-only

# HTTPS with OIDC SSO
./yuzu-server --https-cert /etc/yuzu/server.crt \
  --https-key /etc/yuzu/server.key \
  --oidc-issuer "https://login.microsoftonline.com/<tenant-guid>/v2.0" \
  --oidc-client-id "YOUR_CLIENT_ID" \
  --oidc-client-secret "YOUR_SECRET" \
  --oidc-redirect-uri "https://yuzu.example.com:8443/auth/callback"

Settings: Integrations page

Settings → Integrations (/settings/integrations, admin only; the MCP and Gateway panels also stay on the /settings page). Every panel reads through admin-only fragments that are also gated by the securable of the matching JSON route (Infrastructure Read for webhooks and offload targets, Directory Read for directory sync), so a fragment is never wider than its JSON twin.

• MCP server enables or disables the endpoint and sets Read-only or Full access, with one Save. The confirmation names the state being saved (MCP off, on in read-only mode, or on with write and execute tools). A change lasts until the server restarts, then the start-up flags (--mcp-disable, --mcp-read-only) apply again; a server started with --mcp-disable does not serve MCP until it is restarted without it.
• Gateway shows the upstream-service state, the listen address, the gateway mode and the number of agents connected through the gateway, the flag guidance for a server where the upstream service is not running, and the gateway environment-variable reference.
• Webhooks lists the configured webhooks (destination host, events, last delivery) and opens a delivery history. You can create and delete a webhook; the create confirmation names the destination host and the data the chosen events carry. A destination with embedded user:password@, or with control characters in the URL or event types, is refused. The signing secret is write-only. There is no restriction on the destination address, and the confirmation is a click-through, not an approval step.
• Offload targets lists the targets (destination host, events, batch size, last delivery), opens a delivery history, and creates and deletes targets under the same rules. The credential is write-only. The create audit row records the name, the destination’s scheme and host and a fingerprint of the URL, never its path or query (see Audit Log).
• Directory sync shows the Microsoft Entra sync state (never synced, running, completed or failed are separate states, and a failed read says so rather than showing an empty panel), the synced user and group counts, and a form that runs a sync. The tenant, client and secret are used for that one request and never stored. “Show directory users” loads the list on request, audited as directory.users.view; it is withheld if the audit row cannot be written and shows the first 200 users. Group-to-role mappings are recorded for reference; nothing reads them at sign-in.

Upgrade note. POST /api/settings/mcp now requires exactly one enabled and exactly one read_only field, each the literal true or false, and answers 400 otherwise. Scripts that omitted a field or sent another value must be updated. Creating a webhook or offload target with user:password@ in the URL, or with a control character in the URL or event types, now answers 400 (existing rows are not changed). Webhook and offload-target audit rows now record the destination’s scheme and host plus a short fingerprint, never the URL path or query; rows written by earlier builds still hold the full URL and should be treated like a credential. These writes (and directory sync, the group-mapping save and POST /api/settings/mcp) write an attempted audit row first and refuse with 503 if it cannot be stored, and refuse a cross-site browser request with 403.

Configuration Files

The server’s configuration file (--config, default /etc/yuzu/yuzu-server.cfg on Linux, C:\ProgramData\Yuzu\yuzu-server.cfg on Windows) and its runtime state files live on disk; the state files are written to --data-dir (default: the directory containing the config file). Apart from those files, the NVD cache and the agent-updates//upload-blobs/ file directories, everything — users, sessions, MFA/TOTP, SCIM and every other server store — lives in PostgreSQL (see PostgreSQL Substrate). There is no auth.db; legacy *.db files left in --data-dir by older releases are only probed at startup to warn, never read.

Backup recommendation: A complete backup is a pg_dump --format=custom of the Yuzu database plus the entire CA/cert directory --ca-dir, captured at the same point in time and restored as a pair — --ca-dir holds default-ca.key (the per-install CA private key) and the secrets KEK files secrets-kek-v<N>.key, without which the secret columns in the dump (TOTP secrets, webhook and plugin-config secrets, runtime-config and offload-target secrets) cannot be decrypted. See Backup — the KEK pairing rule and PostgreSQL Substrate for the procedure. Retain old KEK versions for as long as the backups that need them. Also back up the --data-dir .cfg files above, nvd_cves.db (use sqlite3 nvd_cves.db ".backup ...", NEVER cp against a live DB), and the two blob directories agent-updates/ (or your --update-dir) and upload-blobs/ — the dump holds only the package and upload records that point into them, so restoring the dump without these directories leaves records with no files behind them. Losing default-ca.key forces a full fleet re-enrollment (every agent’s cert chains to that root, and the server refuses to silently re-root — see below). Losing the Postgres auth schema AND yuzu-server.cfg requires re-running --first-run-setup to create a new admin. Losing authentication state alone is a Postgres restore — see docs/ops-runbooks/auth-db-recovery.md. Also keep enablement-anchor (it is a witness of the plugin-enablement store, not part of the dump; scripts/yuzu-backup.sh and scripts/yuzu-restore.sh copy it): restoring the dump without it is reported as unwitnessed, and restoring a dump older than the anchor starts with enablement regressed by design (see Enablement continuity anchor). The internal-CA inventory + CRL history (ca_store schema, ADR-0053) is one of the migrated Postgres stores — back it up with pg_dump/pg_restore, not as a separate local file.

Built-in default certificates — convenience, not production. With no --cert/--key/--https-cert supplied (and without --no-default-certs), the server generates a per-install ECDSA CA + server leaves on first boot so a fresh install is encrypted with zero config. Operational caveats:

• 10-year, no auto-renewal. The server leaves do not auto-renew; the yuzu_server_cert_expiry_timestamp_seconds{cert="default-ca"} gauge + the yuzu-tls alerts (docs/prometheus/yuzu-alerts.yml) warn ahead of expiry, one at a time with rising urgency: YuzuCertificateExpiringSoon (warning, 30 to 7 days left: schedule the renewal), YuzuCertificateExpiryImminent (critical, 7 days to 24 hours: renew today), YuzuCertificateExpiryCritical (critical, under 24 hours: renew now). The gateway reports its own certificate files the same way (yuzu_gw_cert_expiry_timestamp_seconds, alerts YuzuGatewayCertificateExpir*; see Certificate rotation and expiry). Replace defaults before production rollout with operator-provided certs (--cert/--key, --https-cert/--https-key) or, to rotate the built-in set, follow the clean re-root in PKI architecture “Operator runbook”, which removes the default-* files and default-marker.json and also clears the CA inventory in Postgres (ca_store). Never clear the whole --ca-dir: it also holds the secrets key-encryption key (secrets-kek-*.key), and the server refuses to start without it (kek_unresolvable, #5370).
• SAN limitation. Default leaf SANs cover localhost, 127.0.0.1, ::1, and the boot-time hostname only. Reaching the dashboard/agent listener by a LAN IP or a different FQDN needs operator-provided certs (or DNS that resolves to a covered name). A host rename invalidates the SAN — rotate the certs after renaming.
• No silent re-root. If ca_store (the internal-CA Postgres store, ADR-0053) already holds a CA root but the on-disk certs in --ca-dir are missing/corrupt (e.g. a wiped cert dir on a persistent data volume, or ordinary later damage to an established install — a bad partial restore, a lost leaf file), the server refuses to start rather than mint a new CA that would orphan every enrolled agent — unless this exact instance can prove it minted the still-incomplete root (its local CA key file still resolves and cryptographically pairs with the stored root), in which case it resumes automatically and re-mints its own default leaves under the same root (ADR-0053). When that self-heal condition does not hold, restore default-*.{pem,key} from backup (matching the ca_store root), or perform a deliberate clean re-root by clearing ca_store.ca_root/ca_issued/ca_crl_versions directly against Postgres — see docs/pki-architecture.md “Operator runbook” for the full procedure.

Enrollment tokens and the pending-agent approval queue are in PostgreSQL (WS-6 6.2, auth.enrollment_tokens / auth.pending_agents — see PostgreSQL Substrate), not --data-dir files: every server replica shares one authoritative view, and --generate-tokens mints straight into the same store the running server reads. If enrollment-tokens.cfg or pending-agents.cfg is present in the directory holding yuzu-server.cfg or in --data-dir at any boot where the database holds no import record for that file (a fresh database never does), the server imports each file once as live credentials, without checking where it came from, then renames it to <name>.cfg.imported (see the enrollment.import audit action and the yuzu_server_enrollment_import_total metric). A fresh deployment must not start with such files left over from another install.

File permissions (Unix): yuzu-server.cfg is 0600 after every write, and the KEK files in --ca-dir are created 0600. No manual chmod is required.

Windows Defender: No authentication-file exclusion applies — authentication state is in PostgreSQL, not a local database file. See docs/ops-runbooks/auth-db-recovery.md.

Enablement continuity anchor

enablement-anchor (ADR-3006) is this host’s witness that the plugin-enablement store has not gone backwards. It holds three lines: the store’s id, and the highest epoch and counter this host has seen. The tables ride the ordinary pg_dump; the anchor is the one extra artefact, so keep it with the data directory and with every backup. The server never lowers it. Nothing in this release issues desired state, so the counter is 0 and the anchor is in practice a store-identity check; the counter becomes meaningful with the later change that issues desired state, which also keeps the file current after each batch. A dump restored together with an anchor taken at the same time is consistent and silent; the anchor catches a database rolled back, or replaced, while the live file survives.

• Unwitnessed. No anchor was found at boot, so the server cannot tell a restore from a first boot. It counts the boot (yuzu_server_enablement_anchor_unwitnessed_total), reports enablement.anchor_unwitnessed_at_boot on /readyz until the next restart, and writes the anchor from the store’s current state. This is expected at first install and at the first boot after upgrading to the release that adds the store, and needs no action. It is also what a restore onto a host without the file looks like. The same applies when the anchor names a different store but its counter and the store’s counter are both 0: nothing was ever issued, so nothing can have been lost. The server logs one warning naming the old and new store id, rewrites the anchor with the current store id and does not declare a regression.
• Regressed. A PRESENT anchor disagrees with the store in a way that could have lost issued state. The server declares enablement regressed: /readyz shows enablement.regression_pending (readiness is not affected), yuzu_server_enablement_regressed is 1, the critical YuzuEnablementRegressed alert fires, and the boot log has a line containing declared REGRESSED. The reason is one of store_id_mismatch (the anchor names a different store and the anchor’s counter or the store’s counter is above 0: a database replaced, or an image cloned with another deployment’s anchor, after desired state was issued), store_behind (the store’s counter is below the anchor’s: a dump older than the anchor) or anchor_unparseable (the file is readable but is not an anchor: hand-edited, truncated or over 512 bytes). The declaration is permanent in the product: nothing clears it, deleting the file does not, and while it stands the server issues no desired state.
• Unreadable. The file exists but cannot be read (permissions, a directory in its place, descriptor exhaustion). That says nothing about the store, so nothing is declared: the server refuses to start and the log names the cause. Fix the file or its permissions and restart.

Exit from a declared regression in this release. With the server stopped, either restore a dump taken at or after the anchor (this keeps the ordering of states, and is the right answer for store_behind), or, when nothing has been issued, clear the declaration and the anchor together:

UPDATE plugin_config_store.enablement_meta SET regressed_at = NULL, regressed_reason = '' WHERE id = 1;

Then remove or correct enablement-anchor in the data directory and start the server, which writes a fresh anchor from the store. Clearing only one of the two re-declares at the next boot: the SQL alone leaves the mismatching file, and removing the file alone leaves the declaration (the server does not write the anchor while regressed). This is safe in this release only because the counter is 0, so nothing has been issued that an endpoint could hold; once desired state is issued, discarding the ordering is a decision with consequences, and the operator recovery flow that arrives with it replaces this procedure.

High availability. Use one persistent data directory per replica, never shared and never cloned from another deployment. A lagging replica or a failed-over leader has a weaker witness than the replica that wrote the anchor last. One replica’s declaration is cluster-wide (it lives in the database), but the other replicas’ /readyz stays stale until they restart in this release. An ephemeral data directory (a container without a volume for --data-dir) makes the mechanism inert: every boot is unwitnessed, and a data directory the server cannot write sets yuzu_server_enablement_anchor_writable to 0, which raises YuzuEnablementAnchorWriteFailing after 5 minutes.

First-Run Setup

When the server starts for the first time and no yuzu-server.cfg exists, it enters interactive setup mode on the terminal. The setup prompts for:

1. Admin username — the initial administrator account.
2. Admin password — entered twice for confirmation. Stored as a PBKDF2 hash.

After setup completes, the server writes yuzu-server.cfg and starts normally. Subsequent restarts skip the setup prompt.

Headless deployment: For automated or containerized deployments, pre-create yuzu-server.cfg with PBKDF2-hashed password entries before starting the server for the first time — first-run setup is interactive and will exit without a TTY.

Seeded credentials

No image or installer provisions a default account. No image COPYs a yuzu-server.cfg, and there is no built-in account: a server started without a config runs interactive first-run setup, which prompts for an administrator account only (since 0.15.0; earlier versions also prompted for a second user account that could never sign in on PostgreSQL — #5343). Create further accounts in Settings → User Management after the first sign-in.

One checked-in file, deploy/config/uat/yuzu-server.cfg, does hold a single admin entry — with a fixed salt and a fixed PBKDF2 digest committed to git, for the known password adminpassword1. It is orphaned: no script, compose file or image reads it (the UAT rigs each generate their own config at a temp path with a fresh random salt per run). Never copy it into a deployment, and do not read its presence as a supported default — removing it from the tree is tracked separately.

If you seed an account yourself, change its password before exposing the server — sign in and use Settings → User Management → Change password (or POST /api/v1/users/me/password). Editing the config file afterwards does not change it: once the account exists in the PostgreSQL auth store, that row is the only credential checked (#5274). For enterprise deployments, integrate OIDC SSO and disable local accounts.

Settings: General page

Settings → General (/settings/general, admin only; every panel it shows also stays on the /settings page).

• Appearance holds the colour scheme and the table-density preference. The scheme is one of six: Slate (the default), Navy and Jade are dark; Light, Light Baby Pink and Light Hot Pink are light. It applies to every page and repaints as soon as you pick it. The choice is made on this administrator-only page, so a non-administrator account cannot change its scheme. Density (Comfortable or Compact) applies to the Console results table. Both are saved in this browser only.
• Server holds the two runtime settings that change on a running server, written through PUT /api/config/{key}: the agent heartbeat timeout (heartbeat_timeout; seconds of silence after which this server drops an agent connected directly to it) and the log level (log_level). Both apply to this server process only, are stored, and are re-applied at restart; the log level then wins over --log-level. Neither is the dashboard sign-in expiry: that is the read-only Console idle timeout (--session-inactivity-secs) and Console session lifetime (fixed, 8 hours) rows in the server-configuration table below the form. The heartbeat timeout must be 30 seconds or more (a lower value stored by an earlier build is raised to 30 at start, with a log line); keep it well above the agents’ own heartbeat interval, since 30 leaves no margin for a late heartbeat. Runtime changes are audited first: if the attempted audit row cannot be written the change is refused and the page shows the error; if only the outcome row is lost the change stays applied and the page shows a banner.
• Analytics shows the event-collection status and the most recent buffered events (type, source, severity, time; no host or user).

Upgrade Notes

Fresh deployment only for 0.15.0. Entries below that tell you to upgrade, roll back or restore an earlier build are historical and not a supported path (see “Deployment contract (hard cutover)” in Upgrading). Entries describing behaviour of the 0.15.0 build still apply.

vNEXT — the offline user-hive file check is now the default for autoruns, installed_apps, license_scan, registry and echo (related issue #5291; breaking for fleets with non-standard profile roots)

What changed. Agent plugins that mount a logged-out Windows profile’s NTUSER.DAT offline (to read its HKEY_USERS hive without a live session) have always run a file-level check against that hive — the path must not cross a UNC/non-fixed-drive or a reparse-point ancestor, and the file itself must be a regular file owned by the profile, LocalSystem or Administrators, with no more than one hard link (hive_hardlinked). Before this change only one plugin (privacy_permissions) opted into that check; the other five production callers (autoruns, installed_apps, license_scan, registry, and Echo’s mapped-drive-history collector) mounted with no check at all. The check is now the default for all of them.

Who this affects. A profile hive that fails the check — most commonly an FSLogix, UWP-container or other non-persistent-image profile root, or any profile directory not owned by its own user/LocalSystem/Administrators — now surfaces as a named, unreadable-source row instead of being silently mounted and read: autoruns list/catalog and installed_apps list_per_user name the specific refusal reason per profile; license_scan list reports the aggregate reason hive_file_refused on its per_user_hives probe status (the specific per-profile reason goes to the agent log only); registry answers its existing hive-file-refused error (unchanged wording, no reason suffix); Echo’s mapped-drive history retries an affected profile individually, up to 7 attempts over 24 h apart, before giving up on that profile. A fleet with many such profiles (e.g. a VDI/FSLogix rollout) may see these constraint rows appear for a noticeable fraction of devices right after upgrading.

What to do. No action is required — this is a fail-closed tightening of an existing check’s reach, not a new restriction with a workaround; a profile that reads fine today under the new default was never at risk from it. Automation that treats one of the five affected plugins’ previously-silent success as a guarantee (a script that only checks for a non-empty list row count, a SIEM rule that alerts only on an explicit error) should instead treat a hive_*/hive_file_refused/hive-file-refused row the same way it already treats any other named per-profile constraint from these plugins (not_found, privilege_missing, mount_failed) — none of these are new categories of outcome, only a more specific one for this particular cause.

vNEXT - MFA step-up is checked once, before the route’s own checks; the server refuses to start with an unclassified mutating route (breaking order of checks)

What changed. A fresh MFA proof is required on 37 named high-risk routes for MFA-enrolled local accounts and OIDC sessions per --mfa-enforcement; un-enrolled accounts and API-token and MCP-token callers pass, and engine principals are denied. Other mutating routes and all MCP tools are not MFA-step-up gated in this release. The check now runs in the request pipeline before the route’s own authorization and its own same-site copy (and after the server-wide same-site check, which answers a cross-site cookie write 403 first), so a session with a stale or missing proof receives the 401 step-up envelope instead of the route’s 403; engine principals receive 403 on every one of the 37 routes regardless of grants (once their credential resolves and the pre-routing gates and the unwired-store 503 below have been ruled out; during a database outage the credential cannot be resolved and the request is answered 401 unauthorized instead), audited as mfa.step_up.required instead of the per-route handler denial row; and POST /api/settings/mfa/recovery-codes and POST /api/settings/mfa/disable are newly gated. A server with no wired authentication store answers 503 on these routes, and an unreadable store answers 503 to local sessions. Full list of behaviour changes: Upgrading (“Breaking: MFA step-up is checked once”).

Boot refusal. At boot the server checks that every POST/PUT/PATCH/DELETE route registered through the route sink is classified in the step-up table. If one is not, it logs start_web_server(): mutating route has no MFA step-up policy row: <METHOD> <pattern> for each, then aborting: N mutating route(s) are not classified in step_up_policy.hpp, stops the gRPC listeners and exits with status 1 before the web listener starts. The gRPC agent port is already listening when the check fails and keeps accepting connections for about five seconds while the server shuts down, before the process exits (about eleven to twelve seconds in total, measured), so while a restart policy keeps starting it, connected agents and gateways see the server accept connections for about five seconds out of every eleven to twelve and then drop, and the web UI never comes up. A restart policy (restart: always or unless-stopped, systemd Restart=, a Kubernetes pod) turns this into a restart loop, and a rolling deployment halts because the new replica never becomes ready. It can only trigger on a fork or a custom build; stock builds are checked by compiled and source-scan tests before release. Recovery: roll back to the previous binary or image (nothing is migrated), or add the missing row to step_up_policy.hpp and rebuild. No flag disables the check.

Alerts. YuzuStepUpGateUnavailable fires on sustained 503s from the check (see Step-up returns 503); YuzuStepUpWindowDisabled fires while --mfa-step-up-window-secs is 0 or less.

vNEXT - Integrations writes need a same-site Origin on a cookie session, write an attempted audit row first, and store no destination URL (breaking for cookie-reusing scripts)

POST/DELETE /api/webhooks, POST/DELETE /api/v1/offload-targets, POST /api/directory/sync, PUT /api/directory/group-mappings and POST /api/settings/mcp now answer 403 to a request authenticated by a session cookie that carries no Origin/Referer, or one naming another site. A script that reuses a login cookie jar (curl -b) must switch to an API token (token callers are not gated) or send a same-site Origin; behind a proxy that rewrites Host, declare the external origin with --csrf-trusted-origin as described above. Each of these writes also records a result=attempted audit row before it changes anything and answers 503 with Sec-Audit-Failed: true when that row cannot be stored; during an audit-store outage they are unavailable, deleting a webhook or offload target included (the MCP switches can still be turned off with --mcp-disable and a restart; check /healthz stores.audit and the YuzuAuditPersistFailures alert). SIEM rules that filter result != success will now also see the attempted rows. Webhook and offload audit rows record url=<scheme>://<host>[:<port>] url_fp=<12 hex> instead of the full URL, and their log lines name the host (and the webhook or target id where one exists), never the URL; rows written by earlier builds keep the full URL, so rotate any hook URL that appears in one.

vNEXT - POST /api/settings/mcp now refuses a body that omits or repeats enabled or read_only (breaking for scripted callers)

The dashboard route behind the MCP settings form used to read the first enabled and read_only value it found and treat an absent one as false: a request carrying only enabled=true switched Read-only MCP to Full access, and an empty request switched MCP off. It now requires exactly one enabled and exactly one read_only, each true or false, and answers 400 with nothing changed otherwise. The rebuilt panel always sends both. A script that posted a single field must send both.

vNEXT - changing an account’s role or granting elevation eligibility needs your own administrator role (breaking for elevation-only operators)

What changed. POST /api/settings/users/{username}/role and the grant form (eligible=true) of POST /api/v1/users/{username}/elevation-eligibility refuse, with 403, a session that is administrator only through a temporary elevation. The attempt is audited (elevated_session_role_change_blocked, elevated_session_eligibility_grant_blocked).

Who this affects. Operators who rely on an elevated session to change roles or grant eligibility, in particular deployments where every administrator is an elevation-eligible user account.

What to do. Make the change from an account whose own role is admin (a local administrator, or a member of the identity provider’s administrator group). Revoking eligibility is still allowed during an elevation.

vNEXT - --gateway-upstream refuses to start without gateway peer authorization (breaking for own-certificate and plaintext installs)

The gateway-upstream service now admits only a gateway whose certificate key (SPKI SHA-256) is pinned, carries serverAuth and is inside its validity window, and the server exits at boot, before binding a port, when --gateway-upstream is set and that cannot be arranged. Who is affected and what to do:

• Generated default certificates, TLS on: nothing to do; the server pins its own default-gateway.pem. A gateway on another host that does not present that file is refused until its key is pinned.
• Your own certificates: pin the gateway’s key with --gateway-peer-pin or --gateway-peer-pin-file (env YUZU_GATEWAY_PEER_PINS / YUZU_GATEWAY_PEER_PIN_FILE) before upgrading, or the server will not start. This includes a Linux package install whose unit passes --gateway-upstream unconditionally even though you run no gateway: pin, or drop --gateway-upstream in a drop-in.
• Plaintext (--no-tls) rigs: add --insecure-gateway-peer (env YUZU_INSECURE_GATEWAY_PEER=1); this disables the check on every port the service is reachable on. It is refused together with a pin.
• Windows installer: /GATEWAY with your own gRPC certificates needs /GATEWAY_PEER_PIN_FILE=<cert.pem> (preferred) or /GATEWAY_PEER_PIN=<64 hex> the first time, and /GATEWAY /NOTLS together with either is refused.

Check your own unit, env file, compose file or installer arguments before upgrading; the literal commands, the per-deployment table and the rollback steps are in Upgrading (“Breaking: --gateway-upstream now refuses to start without gateway peer authorization”), and the full reference is Gateway upstream peer authorization below. Pins are fixed at boot; changing or withdrawing one is a restart.

vNEXT - changing sign-in provider settings needs your own administrator role (breaking for elevation-only operators)

What changed. POST /api/settings/oidc (Settings > OIDC) and PUT /api/config/{key} for the oidc_* keys refuse, with 403, any session whose own role is not admin: a session that is administrator only through a temporary elevation, and (for the config route) a non-administrator account that holds Infrastructure:Write through a custom or service role, or whose authority is an RBAC Administrator grant only, and any MCP-tier token. The attempt is audited (elevated_session_oidc_change_blocked and elevated_session_oidc_setting_blocked for an elevation, admin_role_required_oidc_setting_blocked for a non-administrator, and token_session_oidc_setting_blocked for an MCP-tier or service-scoped token). Seven other keys, plugin_signing_required, the dex_* alerting keys and audit_retention_days, are administrator-only on the same route: see the next note. Every remaining runtime-config key is unaffected.

Who this affects. Operators who rely on an elevated session to change the OIDC issuer, client, secret, redirect URI or administrator group, in particular deployments where every administrator is an elevation-eligible user account; and automation that writes oidc_* through PUT /api/config/{key} with a token whose creator is not an admin account.

What to do. Make the change from an account whose own role is admin (a local administrator, or a member of the identity provider’s administrator group). If no such account can sign in, use the break-glass account (--break-glass-user, --break-glass-arm) or the direct-SQL fallback in auth-db-recovery.md. A stored oidc_* value overrides the matching --oidc-* flag after a restart, so correct the stored value rather than only the flag.

vNEXT - the runtime-configuration read shows a caller only the overrides it could write (breaking for MCP-tier and non-administrator readers)

What changed. GET /api/config, the new GET /api/v1/config and the MCP tool get_runtime_config leave out the stored override of every key the caller could not write (the oidc_* keys, plugin_signing_required, the DEX alert keys and audit_retention_days), and the effective audit_retention_days. An administrator account’s GET /api/config body is what it was, apart from log_level reading warn instead of warning. heartbeat_timeout is now limited to 30 to 86400 seconds on every write route, and a stored value outside that range is moved to the nearest bound at server start (logged).

Who this affects. Scripts and backups that read GET /api/config with an MCP-tier token or a non-administrator role and expect the administrator-only keys; scripts that compare config.log_level to "warning"; scripts that write a heartbeat_timeout above 86400.

What to do. Read with an administrator API token that is not MCP-tier. Use GET /api/v1/config to see which keys were withheld (withheld_keys). See Upgrading.

vNEXT - seven runtime-config keys are administrator-only on PUT /api/config/{key} (breaking for MCP-tier tokens and non-administrator roles)

What changed. PUT /api/config/{key} for plugin_signing_required, dex_alert_routing, dex_blast_min_devices, dex_blast_window_seconds, dex_blast_cooldown_seconds and dex_cohort_export_key now applies the same caller rule as the Settings routes that also write them, and audit_retention_days joins them by design (it is the stored audit-retention setting, reported by GET /api/config, evidence-bearing, and the lever that will govern enforcement once #483 lands; it is stored only today and a PUT does not change the retention actually applied; it has no Settings route, so this generic route is its only writer): an MCP-tier token, a service-scoped token, or a session whose effective role is not admin (a temporary elevation counts) is refused with 403. The callers that newly reach this refusal are a supervised-tier MCP token (even one minted by an administrator) and, with RBAC on, a non-administrator role holding Infrastructure:Write; service-scoped tokens (and readonly or operator tier MCP tokens) were and remain refused earlier by the permission gate (auth.permission_required). A refusal at the new rule is audited as config.update / denied (token_session_admin_setting_blocked or admin_role_required_setting_blocked) and nothing is stored. The oidc_* keys, which already needed your own administrator role, are unchanged (the handler also refuses a service-scoped token, as defence in depth; the permission gate already does). The role checked is the account’s legacy role or a live temporary elevation, never an RBAC grant: an RBAC Administrator grant alone does not count. Every other key (response_retention_days, guardian_event_retention_days, log_level, heartbeat_timeout) is unchanged and stays writable under Infrastructure:Write.

Who this affects. Automation that writes these keys with a supervised-tier MCP token, even one minted by an administrator, and, with RBAC on, a non-administrator role that holds Infrastructure:Write.

What to do. Change these settings with an administrator account: through the Settings pages (DEX alerts, Plugin Code Signing), or with an administrator API token that is not MCP-tier. audit_retention_days has no Settings page: use PUT /api/config/audit_retention_days with an administrator API token that is not MCP-tier, or an administrator session (a cookie session is a browser write and needs a same-site Origin or Referer; an administrator API token that is not MCP-tier is exempt). PUT only stores the audit_retention_days value: the window the server enforces comes from --audit-retention-days (or YUZU_AUDIT_RETENTION_DAYS) plus a restart, so check enforced in GET /api/v1/settings/data-retention. When #483 lands, every previously stored value becomes live at the next boot, so reviewing the stored values is a precondition of that upgrade. To find earlier writes and for the replay caveat, rolling-upgrade behaviour and the audit query, read the full note in Upgrading (“Breaking: seven runtime-config keys are administrator-only on PUT /api/config/{key}”).

vNEXT — the shipped Docker Compose files keep /etc/yuzu/certs on a volume (#5370; action needed before you recreate a 0.14.0 container)

Fresh deployment only for 0.15.0. This block is not an upgrade path to 0.15.0. If you still run a 0.14.0 Docker Compose stack, the copy-out below remains required before you recreate its server container.

Action required before upgrading from 0.14.0 on Docker Compose: copy /etc/yuzu/certs out of the running server container first (Upgrading → “Docker Compose: copy /etc/yuzu/certs out of the server container before you recreate it”). Affected: a 0.14.0 Docker Compose stack whose server has no named volume on /etc/yuzu/certs while Postgres has one — deploy/docker/docker-compose.yml (the 0.14.0 README quickstart), docker-compose.uat.yml, docker-compose.full-uat.yml, docker-compose.viz-uat.yml, docker-compose.demo.yml used with --keep, any compose copied from them, and Compose Wizard output from before 0.14.1 in Plaintext mode, with ‘Persist generated certs’ unticked, or with named volumes off and an external Postgres. Not affected: docker-compose.reference.yml (which the README quickstart saves as docker-compose.yml since #5419) and docker-compose.reference-gateway.yml (their certs volume).

deploy/docker/docker-compose.yml, docker-compose.uat.yml, docker-compose.full-uat.yml, docker-compose.viz-uat.yml, docker-compose.demo.yml and the Compose Wizard’s Default and Plaintext output (with named volumes on, or with an external Postgres) now mount a server-certs volume on /etc/yuzu/certs, which holds the internal CA and the secrets KEK. In 0.14.0 that directory sat in the container’s own layer, so recreating the container deleted the KEK. Postgres still had it registered, so the server refused to start (kek_unresolvable). The reference composes already persisted it, on their certs volume. The Upgrading section gives the copy-out commands, the diagnostics to run before anything destructive, a rollback that keeps the keys, and the recovery path if the files are already gone. See also What must persist.

vNEXT — a new always-on background sweep auto-cancels executions already stuck at running (#4982, NOT breaking)

What changed. A new clock-guarded background pass, ExecutionTracker::reap_stuck_running_executions (~15-minute cadence, no CLI flag or env var — same unconfigurable-by-design posture as its sibling reapers), recovers an execution row left permanently stranded at status='running' because a post-dispatch bookkeeping write (set_agents_targeted/mark_cancelled) itself failed. It excludes any execution with a still-pending outbox entry or with at least one real per-agent response already recorded — only a genuinely-orphaned row (never reached, or genuinely abandoned by a dispatch that failed its own cleanup) is cancelled.

What an operator sees on this upgrade. Any execution rows ALREADY stranded at running before this build started will usually auto-cancel within roughly 15–30 minutes of the first post-upgrade tick that reaches them, each writing a new execution.cancel audit row with principal="system" on a best-effort basis (if the audit store is unavailable, that pass’s cancellations go unaudited — see yuzu_server_audit_emit_failed_total) — the first system-sourced row that action has ever carried; a SIEM rule or dashboard filter keyed only on human/session principals for execution.cancel will not see these. The 15–30 minute figure assumes a normal-sized backlog: if the pre-existing stranded population is unusually large, the sweep caps itself at 500 cancellations per pass (capped, draining the rest over following passes) or, if stranded rows exceed half of all currently-running executions, declines to act at all (would_wipe) on the assumption that something more systemic is wrong rather than a one-time backlog — in that case the rows stay running and yuzu_exec_tracker_stuck_reap_total{outcome="would_wipe"} holds nonzero until an operator investigates and, if the backlog really is benign, cancels it by hand to bring the ratio back under the sweep’s own threshold. This is otherwise a one-time settling period for pre-existing stuck rows, not an ongoing behaviour change for new dispatches. See docs/user-manual/metrics.md “Execution bookkeeping + stuck-execution reap metrics” for the new yuzu_exec_tracker_stuck_reap_total{outcome} counter and docs/clock-guarded-retention.md for the full design.

vNEXT — the Windows server installer locks its data directory and keeps secrets off the command line (#5196, #5210, #5272; breaking for unattended installs)

Windows server is not a supported deployment (ADR-0035), and the YuzuServer service cannot yet run under the Service Control Manager at all (#5325; see also the open issue #1835, “Windows server binary has the identical SCM control-protocol defect as #1822 (agent)”). This note is for anyone using YuzuServerSetup-*.exe anyway.

What the installer does now. The secrets it handles are kept in %ProgramData%\Yuzu Server, which it locks to exactly Administrators and SYSTEM and verifies before it installs anything:

The service’s command line, which every local user can read, now carries only these paths, and is written with the executable path quoted. Earlier installers meant to pass the OIDC client secret there. In practice, an sc.exe quoting fault dropped every argument, so the service they registered ran with no arguments at all, and the directory was readable by local users. The installer reads the command line back after writing it. If it cannot be written, for example because security software blocks the change, the service is disabled and Setup exits with code 10, rather than leaving the old command line in force.

The server’s own CA, default certificates and key-encryption keys stay where the server keeps them: C:\ProgramData\Yuzu\certs. The server applies its own Administrators-and-SYSTEM permissions to that folder and to those key files when it writes them. Moving them into the locked directory needs server changes first (#5273).

Unattended installs:

• A connection string is required on a fresh install, and on an upgrade from any earlier version, because no earlier version stored one. Give /POSTGRES_DSN_FILE=<file> (preferred) or /POSTGRES_DSN=<connection string>. Without one the install stops with exit code 7. The exception is a YUZU_POSTGRES_DSN environment variable, set machine-wide or for the service: the installer then goes ahead and logs a warning that local users can read it. Earlier installers never passed a connection string, so the service they registered could not start.
• A YUZU_POSTGRES_DSN or YUZU_OIDC_CLIENT_SECRET environment variable cannot be combined with a stored secret of the same kind. The server refuses to start with both, so the installer refuses first and changes nothing. Remove the environment variable, then run the installer again.
• An OIDC client secret is not carried from an earlier version: none stored it anywhere except the service’s command line, and their own command line lost its arguments. Give it with /OIDC_CLIENT_SECRET_FILE= (preferred) or /OIDC_CLIENT_SECRET=. If you had added --oidc-client-secret to the service’s command line by hand and OIDC is enabled on the upgrade (/OIDC_ISSUER), the upgrade refuses until you give the secret again; rotate it in your identity provider afterwards, since local users could read it.
• An upgrade keeps the existing accounts; leave out /ADMIN_USER and /ADMIN_PASS. An upgrade is any run where %ProgramData%\Yuzu Server\yuzu-server.cfg already exists, including an upgrade from an earlier version. The installer can no longer change a stored password: yuzu-server.cfg only seeds the first administrator into an empty database and is not consulted for an account the PostgreSQL auth store already holds (#5274), so a new hash written there would be ignored and the old password would stay valid. A non-empty /ADMIN_PASS= or /OPERATOR_PASS= on an upgrade is therefore refused with exit code 11 before anything is stopped or changed, and the interactive wizard skips the account pages on an upgrade. Change or reset passwords in the product instead (Settings → User Management → Change password / Reset password), or with the direct-SQL fallback in docs/ops-runbooks/auth-db-recovery.md if no administrator can sign in. On a fresh install the passwords must be at least 12 characters and at most 1024 bytes of UTF-8 (the server’s limit; a character outside ASCII takes 2–4 bytes). The installer’s /OPERATOR_USER and /OPERATOR_PASS were removed in 0.15.0 (#5343): the account they wrote went to yuzu-server.cfg only and could never sign in on PostgreSQL. A non-empty value is now refused with exit code 7 (nothing changed; on an upgrade a non-empty /OPERATOR_PASS= is still exit code 11, as above) — create further accounts in Settings → User Management after the install. An upgrade keeps the stored connection string unless a new one is given, and keeps the stored OIDC secret when /OIDC_ISSUER is given again. Repeat your other options (/GATEWAY, /OIDC_ISSUER, /OIDC_CLIENT_ID, …) on an upgrade: they are not remembered. /GATEWAY with your own gRPC certificates also needs /GATEWAY_PEER_PIN_FILE=<cert.pem> (preferred; the leaf certificate only) or /GATEWAY_PEER_PIN=<64 hex> the first time (see Gateway upstream peer authorization, “Windows service”); /GATEWAY with /NOTLS starts the service with --insecure-gateway-peer, and is refused if a pin parameter is also given.
• Upgrade in place; do not uninstall the previous version first. Uninstalling removes the service and the installation record but keeps the old, unlocked data directory. The new installer then cannot tell that an administrator created that directory, and refuses it.
• Exit code 7 means the install was stopped before any file was installed: an input was missing or invalid, or something could not be secured. The reason is in the setup log on a line starting PrepareToInstall:. If the existing service had been stopped by then, it is left stopped, and the message says so (and says if it was disabled). The exception is an upgrade from an unsecured directory that stops before the old directory is moved: the service is then started again. On an upgrade of an already-secured install, the message also lists any files in the data directory that had already been replaced.
• Exit code 10 means the files were installed but the service could not be registered or its command line could not be written. Setup disables the service if it can, and the message and log say whether it did; if not, disable it yourself (sc.exe config YuzuServer start= disabled) until the cause is fixed.
• Exit code 11 means an upgrade was given a non-empty /ADMIN_PASS= or /OPERATOR_PASS= (see above; /OPERATOR_PASS= is itself removed, #5343). Setup stopped before the wizard, before the service was stopped and before anything was written; the reason is in the setup log on a line starting InitializeSetup:. Run it again without the account parameters, then change the password in the product. (Setup’s own temporary folder in %TEMP% is left behind in this case; it holds nothing secret.)
• Secrets on the command line are recorded in the setup log (/LOG=) and by deployment tools such as SCCM (AppEnforce.log) and Intune, including any /ADMIN_PASS=, /POSTGRES_DSN= or /OIDC_CLIENT_SECRET= value. Prefer /POSTGRES_DSN_FILE= and /OIDC_CLIENT_SECRET_FILE=. Stage those files in a pre-step, with permissions only Administrators and SYSTEM can read, and delete them afterwards: the installer copies them and leaves the originals. /ADMIN_PASS= has no file form, so treat those logs as secrets: restrict or purge them after the deployment.

Upgrading from an earlier version. (This describes what the installer does when it finds an earlier installation’s directory. It does not make an upgrade of an earlier build’s database supported in 0.15.0: see “Deployment contract (hard cutover)” in Upgrading.) Its data directory is not locked, so the installer builds a new, locked one. It copies across only plain files that are owned by Administrators or SYSTEM and that no other account can change, from folders that are owned by Administrators or SYSTEM and in which no other account can delete or rename anything: the configuration, postgres.dsn, oidc-client-secret, and the files directly in certs\ and data\. Subdirectories stay behind. The old directory is renamed to %ProgramData%\Yuzu Server.insecure-<date>-<time>; it is never changed or deleted.

If any file that would be copied, or a folder holding one, fails that check, or a file is a link or a hard link, the upgrade stops and names it, and nothing is changed. This also catches a folder that a standard user created before the original install: they stay its owner, so they could have swapped its files for their own. Inspect what is named. If it is yours, remove it, or make Administrators its owner and reset its permissions (icacls <path> /setowner *S-1-5-32-544 /L, then icacls <path> /reset /L), then run the installer again. An unlocked %ProgramData%\Yuzu Server with no Yuzu Server installation registered on the machine is refused, because it cannot be shown that an administrator created it. A junction or symbolic link at that path is always refused.

Fresh deployment only for 0.15.0. This does not make an upgrade of an earlier build supported (see “Deployment contract (hard cutover)” in Upgrading). If the installer has nevertheless run in upgrade mode over an earlier directory, the exposure described in these steps is real whatever the contract says, so carry them out.

After upgrading from an earlier version:

1. Rotate what the old directory exposed. Local users could read it, so treat its contents as disclosed: reset the dashboard passwords, re-issue the TLS keys in certs\, and rotate the OIDC client secret. Reset the passwords in the product (Settings → User Management, or POST /api/v1/users/me/password / POST /api/v1/users/{name}/password): the installer refuses a new /ADMIN_PASS on an upgrade (exit code 11), because the config file only seeds a fresh database and could not change an account that already exists in the PostgreSQL auth store (#5274). The old hashes stay in yuzu-server.cfg — the server logs a boot warning naming each account whose stored password no longer matches it.
2. Move the server’s data subdirectories, if you ran the server. data\agent-updates\ and data\upload-blobs\ (OTA packages and uploads that database rows refer to) stay in the renamed directory. Local users could create files in them, so check their contents. Then stop the server, move them into the new data\, make Administrators their owner and reset their permissions: icacls "%ProgramData%\Yuzu Server\data" /setowner *S-1-5-32-544 /T /L, then icacls "%ProgramData%\Yuzu Server\data" /reset /T /L. If you would rather not trust them, re-upload the packages instead.
3. Delete the renamed directory once the upgrade is confirmed. It still holds the old password hashes and keys.

Constrained Language Mode (WDAC script enforcement, or AppLocker script rules for an install run by an administrator) blocks the .NET calls the installer needs to hash new passwords, so a fresh install with /ADMIN_PASS stops with a message saying so. An upgrade (which keeps the existing accounts and hashes nothing) works, and so does an install run as SYSTEM, which AppLocker exempts.

vNEXT — a hand-edited config listing local users but none with role=admin now fails boot, on every restart (breaking)

What changed. On every boot (not only first boot), the server checks the loaded config’s local user list: if it is non-empty but none of the entries has role=admin, the server now refuses to start (Fatal: the loaded config lists N local user(s) but none has role=admin). Previously, this shape silently promoted the first configured user to Administrator regardless of its declared role — a real bug this change fixes — so the check is not new logic layered on top of prior behavior, it is a correction that happens to also be fail-loud where the old code was silently wrong.

Who this affects. Any deployment whose yuzu-server.cfg local-user list was hand-edited to a shape with no role=admin entry — most plausibly a fleet that migrated fully to SSO/OIDC/SAML and removed or demoted its local admin config entry, relying on the old silent-promotion behavior (or simply never noticing it) rather than on the actual RBAC/session-role state in the database. This check is config-shape-based, independent of whether auth.users already has admins in Postgres — it fires even on a long-running, fully-provisioned deployment if its config happens to carry this shape on the next restart.

Fix, before or immediately after upgrading. Either mark exactly one entry in the local user list role=admin in the config file, or — for an SSO-only fleet — remove all local user entries from the config so this check never triggers. The server logs the config’s user count and refuses to guess; there is no other remediation needed once the config is corrected.

vNEXT — installed_apps list rows carry two more fields (breaking)

What changed. The installed_apps agent plugin’s operator list action (definition crossplatform.software.inventory) now emits seven |-separated fields per row instead of five: app|name|version|publisher|install_date|install_location|bundle_id. install_location is the Windows Uninstall-key InstallLocation (unexpanded) or the location macOS reports; bundle_id is the macOS CFBundleIdentifier. Either is - where the OS has none: every Linux row ends -|-, and many Windows rows have no InstallLocation (182 of 241 on the reference developer workstation). The first five fields keep their position; every field is now escape-aware (\ folds to /, | to \|, CR/LF to a space, a field over 4 KiB is cut), a no-op for every value in the three reference captures. Row shape for query, list_per_user and the daily-sync inventory (ADR-0016) is unchanged, though on macOS the collector list/query/list_per_user share now sorts same-named apps by (name, install_location) instead of leaving their order arbitrary.

Who this affects. Automation reading installed_apps list output from GET /api/v1/responses/{id}, its /export, or MCP query_responses that (a) unpacks or anchors exactly five fields, or (b) treats the last field as install_date. The dashboard is unaffected: it has always split these rows into app plus one remainder cell, so the new fields appear inside that cell (and the search box matches them) but are not separate, sortable or filterable columns. A policy or script that substring-matches the raw output now also matches install paths and bundle identifiers.

Mixed fleets. Upgrade order is server first, so agents keep answering with five fields until they take the release carrying installed_apps 1.2.0 (the definition’s minAgentVersion stays 1.0.0). Agents older than 1.2.0 do not escape | in name, version, publisher or install date, so a row from one of them can split into any count under the escape-aware split: accept a row only when it has exactly 5 or exactly 7 tokens, reject any other count, and read the sixth/seventh columns only for agents known to be on plugin 1.2.0 or later.

Existing deployments keep the old definition. The bundled definition is seeded once and never refreshed on an existing server (see the InstructionStore note in upgrading.md; #2555), so get_definition and discover_instructions keep listing four columns at version 1.0.0. Row content is unaffected. Editing the definition (dashboard YAML editor or PUT /api/instructions/{id}) to declare the two new columns is optional and cosmetic.

Before upgrading, check whether this affects you. Search your scripts, SIEM parsers and saved exports for consumers of installed_apps/crossplatform.software.inventory output and change any fixed five-field pattern to app|name|version|publisher|install_date[|install_location|bundle_id].

vNEXT — re-eval on a result set now refuses instead of broadcasting when its recorded parent set has been deleted (#4306, breaking)

What changed. POST /api/v1/result-sets/{id}/re-eval and MCP reevaluate_result_set previously synthesised the sibling’s dispatch scope from the original’s live, nullable parent_id foreign key. If the original’s parent result set was later deleted (parent_id ... ON DELETE SET NULL), the column read as absent, and an absent parent_id reaching dispatch synthesis meant “broadcast to __all__” — silently turning “re-ask the same narrow question” into “ask the whole visible fleet.” Both routes now refuse (400 RESULT_SET_BAD_REQUEST, reason=parent_gone) when the live parent is gone but the original’s persisted source_payload shows it was narrowed at creation, instead of broadcasting.

Who this affects. Any caller (REST or MCP) whose automation re-evaluates a result set that was originally narrowed to a parent_id, where that parent set has since been deleted. Previously such a call silently succeeded with a 202 (REST) or a materialized/pending result (MCP) dispatched to the entire visible fleet; it now refuses instead — REST returns 400 RESULT_SET_BAD_REQUEST, MCP returns a JSON-RPC error (kInvalidParams) over HTTP 200, per that transport’s existing error-shape convention. No legitimate caller should have been relying on the fleet-wide broadcast — this was the target-erasure defect being fixed — but any automation catching only success responses on this route should add handling for the new 400 reason=parent_gone case: create a fresh result set from the intended parent instead of re-evaluating the orphaned one. A genuinely parentless original (no parent_id was ever supplied at creation) still broadcasts on re-eval, unchanged.

vNEXT — a result-set parent_id alias longer than 64 bytes is no longer accepted on from-echo-query/from-instruction-result (#4734, breaking)

What changed. parent_id accepts either a canonical rs_... id or a per-operator alias (the set’s own name, valid up to 256 bytes). POST /api/v1/result-sets/from-echo-query and /from-instruction-result now bound parent_id to 64 bytes (kResultSetParentIdMaxLen) before attempting alias resolution, closing a gap where an oversized value was copied unbounded into the persisted scope_input_id lineage marker. MCP’s equivalent producer tools already enforced this bound; REST did not — this change brings REST to parity.

Who this affects. Any caller referencing a result set by a name-based alias longer than 64 bytes as parent_id on these two REST routes. Previously such a call resolved the alias and dispatched normally; it now returns 400 (“parent_id must be at most 64 bytes”) before resolution is attempted. Use the set’s canonical rs_... id instead (GET /api/v1/result-sets lists both id and name for every set you own).

vNEXT — GET /api/v1/result-sets can now answer 503; the async result-set producers’ post-dispatch fault code changes from 400 to 500 on REST (#4306, breaking)

What changed. GET /api/v1/result-sets previously always answered 200, even when the underlying store read was degraded — ResultSetStore::list_by_owner returned a plain (possibly empty) container rather than surfacing the failure, so a degraded Postgres read answered 200 with an empty result_sets array indistinguishable from a genuinely-empty owner. It now answers 503 RESULT_SET_STORE_UNAVAILABLE (Retry-After present) on a genuine store-level read failure. Separately, on REST, the three async result-set producers (POST /api/v1/result-sets/from-echo-query, /from-instruction-result, and /{id}/re-eval) previously mapped a post-dispatch store fault — the pending result-set row failing to persist after a real command had already dispatched to agents — to 400. That was a client-error status for a server-side fault; it is now 500 RESULT_SET_STORE_FAULT_AFTER_DISPATCH, matching MCP’s rs_run_async, which already used its kInternalError branch for the identical case (the JSON-RPC error type is unchanged). The same three REST routes also gained a new PRE-dispatch 503 RESULT_SET_STORE_UNAVAILABLE when the per-owner quota cannot be verified before dispatch — nothing is sent in that case, and the request is safe to retry.

Who this affects. Any REST caller that treats GET /api/v1/result-sets as never-erroring, or that pattern-matches the old 400 on the three async producers’ post-dispatch failure path. A 503 should be retried (Retry-After header present); a 500 RESULT_SET_STORE_FAULT_AFTER_DISPATCH means a command already dispatched — do not re-send, poll GET /api/v1/executions/{id} for its outcome instead. MCP’s error type (kInternalError) is unchanged, but the embedded fault-message token was also renamed to RESULT_SET_STORE_FAULT_AFTER_DISPATCH for the same reason as REST — any MCP caller pattern-matching the old RESULT_SET_STORE_UNAVAILABLE token string on this specific post-dispatch branch should update to the new token (the 3 MCP tool descriptions in kTools[] document both tokens explicitly).

vNEXT — the 4 result-set write routes now enforce MCP tier/approval, closing a cross-transport bypass (#5047, breaking)

What changed. POST /api/v1/result-sets, /{id}/pin, /{id}/unpin, and DELETE /api/v1/result-sets/{id} — and their dashboard-fragment equivalents — previously had no RBAC or MCP-tier gate at all beyond ownership; only their MCP tool twins (create_result_set, pin_result_set, unpin_result_set, delete_result_set) enforced the tier/approval belt every other MCP-tiered operation gets. An MCP-tiered bearer token could reach the identical mutation its own tool is gated for simply by calling REST or a dashboard fragment instead of /mcp/v1/. All 8 sites now apply the same belt MCP already did: a readonly- or operator-tier bearer is denied Infrastructure:Write/Infrastructure:Delete outright; a supervised-tier bearer deleting a set must use the MCP ticket-then-recall flow (REST/fragment delete now requires approval on every transport except /mcp/v1/ itself, matching create/pin/unpin’s own approval posture — the routes never granted approval, they simply had no gate to enforce it). Ownership remains the primary, and for a plain (untiered) session the ONLY, gate — this is not a new RBAC securable and does not change who can reach their own result sets; see rbac.md’s “Not RBAC-gated: per-operator result sets”. The still-open, separately-tracked #4309 gap — a plain RBAC session or an API token minted with no mcp_tier skips this belt entirely, on every transport, including MCP itself — is unaffected by this change.

Who this affects. Any integration that mints MCP-tiered bearer tokens (readonly/operator/ supervised) and calls the JSON /api/v1/result-sets* routes or the dashboard fragments directly, rather than going through /mcp/v1/. A readonly- or operator-tier caller that previously reached create/pin/unpin/delete on their own result sets via REST now gets 403 ("MCP token tier does not allow Infrastructure:Write" or :Delete). A supervised-tier caller deleting via REST now gets 403 with a remediation pointing at the MCP ticket-then-recall flow, instead of succeeding directly. A supervised-tier caller’s create/pin/unpin is unaffected (never approval-gated at any tier). An untiered caller (a plain cookie session, or an API token minted with no mcp_tier) sees no behavior change. See rest-api.md’s per-route Errors tables for the exact new status codes.

vNEXT — POST /api/policies/{id}/evaluate and /remediate can now answer 503 where they previously answered 409/400 (#4981; breaking)

What changed. Both routes previously classified a degraded scope evaluation — a from_result_set: reference whose owning set was deleted by a concurrent operation, or a presence-store outage during a fleet-wide scope check — as an ordinary “no agents matched” or “request rejected” outcome: /evaluate returned a clean 409, and /remediate (when agent_ids is supplied) returned a 400 audited as an operator denied. Neither was true — the check could not actually be evaluated. Both routes now answer 503 for this case, audited as error, with the response message (and audit detail) carrying the specific underlying cause rather than a generic string.

Who this affects. Any automation that treats 409 from /evaluate or 400 from /remediate (with agent_ids) as terminal, or that logs either code as an operator denial. A 503 from either route is safe to retry once the underlying condition (a concurrent result-set delete, a presence-store hiccup) clears — typically within seconds.

What to do. Treat 503 on these two routes as retryable infrastructure degradation, distinct from a genuine 409/400. No action needed if your automation already retries on 503 generically.

vNEXT — server TLS listeners now pin a fixed TLS 1.2 cipher allow-list; a previously-set GRPC_SSL_CIPHER_SUITES no longer applies (#4722; breaking)

What changed. The server now unconditionally overwrites GRPC_SSL_CIPHER_SUITES in its own process environment before any gRPC call, and applies the same six-suite ECDHE TLS 1.2 allow-list to the HTTPS dashboard listener and its certificate hot-reload validation. It self-checks the resolved policy at boot and refuses to start if the allow-list resolves to zero usable TLS 1.2 ciphers on the local OpenSSL build. See TLS policy for the exact list and what CI proves about it.

Who this affects. Any deployment that previously set GRPC_SSL_CIPHER_SUITES in the server’s environment to select or restrict its own cipher suites — that variable is now silently ignored, and the server’s fixed allow-list applies instead.

What to do. There is no override flag and no config option to restore the previous behavior — this is a deliberate policy floor, not an oversight. If your environment relied on a non-default GRPC_SSL_CIPHER_SUITES value, remove it (it no longer does anything) and confirm your TLS clients support at least one suite in the pinned list (tls.md); operator-issued RSA certificates remain supported via the pinned ECDHE-RSA suites.

vNEXT — new Guardian T_* diagnostic log lines at info level (#4606; NOT breaking)

What changed. The server now writes one info-level line for every Guardian event it stores for an ordinary rule (ruleless DEX observations are excluded):

Guardian T_server event_id=… agent=… rule=… recv_ns=… committed_ns=… agent_ns=… store_ms=…

The agent writes Guardian T_wire event_id=… domain=… sent=… wire_wall_ns=… for every Guardian event it attempts to send over the Subscribe stream on the legacy detection path (domain=legacy; sent=0 means the link was down or the local write failed, and that event was dropped and is not retried). Ruleless DEX observations are not logged. Only once the Spark path is the live backend (prefer_spark, off by default) does it also write Guardian T_detect … for each outbox entry an evaluation pass stages (a pass evaluates every rule on the watched key and stages up to two entries for each, so a key shared by several rules can produce several lines, and a pass that stages nothing produces none), plus the same T_wire line for the Spark outbox path (there sent=0 means the local write failed and the entry is retried; a down stream logs nothing). They exist to measure detect-to-deliver latency for the Spark cutover benchmark (#4606); they are plain log lines, not metrics or audit events, and no shipped component consumes them. If you do consume them, join on the agent and event_id (the agent-side lines carry no agent field, so take the agent from which log the line came from and from the server line’s agent=), and never on log-file line order: a T_wire line can appear before its own T_detect line.

What the lines contain. Only identifiers, times and a few flags and counters: the event id, the agent id, the rule id, instants or elapsed times, and fields such as sent, accepted and seq. They contain no event detail, no detected or expected value, no user name, process name or path. (The one exception among the lines named here is the server’s Guardian ingest error warning, which also prints the store’s error message; it has control bytes and line separators folded, keeps its spaces, and is shortened to 256 bytes. Two further server lines print their agent and one other name neutralised: the observer-threw warning prints the observation type and the agent, and the line for an ignored __guard__ action prints the action name and the agent. The observer-threw warning also prints the exception text, which is not neutralised.) Ids are neutralised before they are written: every byte outside printable ASCII, and any space, = or , (a line break included), becomes _, and an id longer than 256 bytes is shortened to 256 by keeping its head and its last 24 bytes (the part that tells two events apart), identically on the agent and the server and on the server’s Guardian ingest replay, conflict, error, oversized-detail and parse-failure lines, so an operator-chosen rule id such as Disk Full appears as Disk_Full and a non-ASCII rule name appears as underscores. The event id printed on the server’s replay, conflict and oversized-detail lines is this neutralised form, so to find that event in the store, match on the agent, the rule and the time rather than pasting the id. This covers the lines named here. It does not extend to the server’s other log lines that print an agent id read back from the session table or the enrollment store (the Heartbeat, BatchHeartbeat, ReportInventory and ProxyInventory ingest paths and the shared ingestion modules they call, the Subscribe stream open and close lines, the agent registry’s other log lines that print a session-table id (removal, rollback, session-timeout, sweep and dispatch-routing lines), and the enrollment store’s own agent-id log lines in auth_db.cpp (the admin-denied warnings and the consumed, added-to-pending, approved, denied and removed-from-queue lines) and the server’s command-forward, asset-tag push and quarantine dispatch-denied lines in server.cpp): those print the id as stored. The enrollment store has already refused control bytes and , in it, so it cannot add a newline-delimited line, but an id enrolled or imported before the new-id character check may still contain a space, = or a non-ASCII byte, and so can add same-line key=value tokens. AuthDB’s agent-id check also accepts U+0085, U+2028 and U+2029, which a log reader that splits on Unicode line breaks (a splitlines()-style reader) treats as line breaks, so such an id can still appear to start a new line to that reader. The exception text of the ingest catch-alls on those paths is printed as is. #4665 closed the rest of this list: every other rule-id, Spark-key, path, hive, service-name and unit-name print site across the Guardian/Spark subsystem — the legacy file, registry and service guards, SparkEngine, the Guardian engine’s arm/baseline/rule-parsing messages, the Spark runtime’s own lines that name only a watched key, the server’s Guardian push enforce-downgrade warning, and the event store’s own error lines — is now neutralised too. A rule id on any of these lines is wrapped the same way as on the lines above (log_id_token: control bytes, DEL, space, =, , and every non-ASCII byte fold to _, then a 256-byte head/tail shorten), so the same Disk_Full-style search still finds it. A path, registry hive/key, service or unit name embedded in a '...'-quoted fragment is wrapped with the newer, more permissive log_key_token (common/include/yuzu/log_token.hpp): it preserves spaces and non-ASCII bytes, so a real Windows path or a non-ASCII name still reads naturally, and folds only control bytes, DEL, and three Unicode line-separator sequences (NEL, LINE SEPARATOR, PARAGRAPH SEPARATOR) to _ — the minimum needed to stop a forged physical log line. On any of these lines a new physical line cannot be forged from an id or key. The agent’s hostname on the “Agent registered” line is not wrapped this way: it takes the same closed-set fold as an id (control bytes, space, =, , and every non-ASCII byte become _, 256-byte head/tail shorten), so a hostname such as x' plugins=0 id=victim ' appears as x'_plugins_0_id_victim_' and cannot add a same-line token. The registration-path lines that fold agent-supplied values are the registration, update-check and update-download request logs, the agent certificate signing logs on the registration path, and the agent registry’s registration and revoke logs; the update-download admission and identity-rejection lines fold control characters only, as described below. A same-line key=value token can still be forged where a value is wrapped with log_key_token (paths, registry keys, service and unit names, the Guardian ingest error text, and on the registration lines tag keys and store error text), and likewise where the control-byte-only fold is used (gateway node names, cluster ids, session ids and plugin names on the gateway stream-status, heartbeat and inventory lines, and the update-download admission and identity-rejection lines), because that wrapper keeps spaces and =. The control-byte-only fold also leaves U+0085, U+2028 and U+2029 in place, so a log reader that splits on Unicode line breaks can still see a line break in those gateway node names, cluster ids, session ids, plugin names and update-download lines (the exception is the session ids on the ProxyRegister adopted, lost-race and success lines and on four stream-status lines, which are wrapped with log_key_token and so have those three code points folded too). Store and database error text in these log lines is shortened to 256 bytes (head and tail kept), so the later hosts of a multi-host libpq message can be cut from the log line; the connection pool keeps the full text in memory for its last-error accessor. A regression-net test (tests/test_guardian_spark_log_injection_tripwire.py) fails the build on a future unwrapped call site in this subsystem — a lexical scanner with documented blind spots (for example it cannot trace a value wrapped several lines above its print site through local-variable construction, nor a value copied into a renamed local; the full list is in the test’s own header comment). They are ordinary log output, so how long they are kept is decided by your log pipeline, not by the Guardian event retention period, and they are not an audit record.

Impact. Log volume only. There is no API, schema, metric, alert or wire change, and no change to any detection, dispatch or ingest decision. Nothing to do on upgrade.

• Today, on the legacy path: up to one server line and one agent line per Guardian event, so it tracks your Guardian event volume. The agent debounces drift events per rule (default 1000 ms, the rule parameter event_debounce_ms), which collapses rapid drifts; a rule configured with event_debounce_ms of 0 emits every drift, and a return to compliant is never debounced, so a rule that flaps can still log at its flap rate.
• Once the Spark path is live: up to two agent lines per event (T_detect and T_wire) plus the server line. Lifecycle events, replayed events and health events raised by a subscription fault or its recovery log T_wire with no T_detect (health entries raised by an evaluation pass do get one), a retried send logs T_wire again, lifecycle journal replays re-send on every reconnect (with no info-level server line), a rule stuck in an unknown or error state re-emits on the errored-refresh cadence (default 5 minutes, and in practice no faster than the rule type’s convergence sweep), and each evaluation pass rejected by a full outbox logs another accepted=0 T_detect line with no rate limit of its own.
• Where the lines go: wherever a log file is in use (--log-file on the server or the agent; a Windows service agent defaults to yuzu-agent.log under its data directory) the file sink rotates at 50 MB and keeps the active file plus up to 5 rotated files by default (up to about 300 MB per sink; --log-max-size in bytes, --log-max-files), so an event storm shortens how far back your logs reach. Without a log file the lines go to the console and Yuzu applies no rotation: retention and any rate limiting belong to your service manager or container runtime (a journald rate limit can drop lines, including unrelated warnings).
• T_server is now asynchronous too, on its own dedicated logger (#4666 PR-4); the agent side has been asynchronous since PR-2. The thread that reads the agent’s stream, or the ForwardGuardianMessage handler when the agent is behind a gateway, resolves T_server through a small bounded async logger built solely for this one line (its own 1024-slot queue — a fixed ~408 KiB of RSS, paid once at boot whether or not anything is ever logged (about 1/8 the agent-side hand-off’s 3.34 MB, sized for one diagnostic line) — and one worker thread, overrun_oldest eviction under sustained overload — the same eviction policy the agent-side hand-off uses, and just as on the agent, overrun_oldest drops are silent in this release, with no counter or alert to point at them yet), so that thread now enqueues and returns rather than blocking on sink I/O, mirroring the agent-side mechanism much more lightly (no teardown watchdog: the server’s hard-exit machinery is signal-driven, not self-armed — the first SIGTERM takes the graceful stop path, so a sink that stalls at exactly the wrong moment leaves a wedged pool join bounded only by a second signal or the deployment’s external stop deadline, 210 seconds in the shipped systemd/Compose configs — an accepted, not eliminated, exposure). This is a dedicated per-line change, unlike the agent side’s global default-logger swap: the server’s OTHER Guardian ingest lines on the same code path — the idempotent-redelivery debug line, the event-collision and store/ingest-error warnings, the oversized-detail_json/parse-failure warnings, and the observation-only blast-radius/alert-router warning pair (fires only for the ruleless observation event type) — are unaffected and remain on the ordinary synchronous default logger, so a blocked sink still blocks the ingest thread on any of THOSE lines. One consequence: because T_server now drains through its own queue and worker, a T_server line can now appear later in the log stream than an adjacent Conflict/redelivery/error line from a LATER event on the same code path, even though the events themselves were ingested in order — the existing “never trust log-file line order” guidance above (for T_wire/T_detect) applies to T_server relative to those sibling lines too. On the agent, a guard worker (legacy path) or the Spark consumer, convergence and send threads call the same bare spdlog:: functions as before, but main.cpp now installs the async hand-off logger (see “Agent logging is now asynchronous” under systemd Units, further down this page) as the process’s spdlog default logger, so every one of those calls, including the Spark runtime’s own arm-committed/late-arm/sweep-residue lines, now enqueues and returns rather than blocking on sink I/O, on any platform where the agent image and libyuzu_agent_core share ONE registry (confirmed on both Linux and Windows, the latter measured 2026-10-03 — see docs/spark-rebuild-baselines/4666-mi1b-windows-registry-verdict.md, and “spdlog registry identity across images” in docs/darwin-compat.md, which also has the measured macOS exception — two separate registries there). This is the mechanism, not a dedicated per-line change, for every OTHER agent-side line: the docs/spark-flip-gate.md section 7 precondition (synchronous benchmark and Spark-runtime log writes) is addressed by this global default-logger swap on the agent, plus the server’s own dedicated T_server logger above, not by threading a bounded-wait call through each individual T_detect/T_wire/arm-committed site.

If log volume matters. There is no dedicated switch for these lines. The only lever is the log level, and it is blunt: warn also suppresses every other info line, including on the server the authentication and session lines (for example User '…' authenticated, Local session created) and the API-token created/revoked lines, which a SIEM ingesting the server log would then lose, and the Guardian arm/commit messages on an agent. It also silences the agent’s own “Received signal, shutting down…” line (an ordinary info-level call since #4666 PR-2): at --log-level warn or above that line is absent entirely on a clean SIGINT/SIGTERM/Ctrl-C stop, so if you rely on it to confirm an intentional stop, keep agents at info or below, or check the process exit code (0 = clean) instead. The audit log is a separate store and is not affected by the log level. Prefer applying it to agents only, and weigh it before applying it fleet-wide.

• At startup: --log-level warn (env YUZU_LOG_LEVEL) on the server and/or the agent; this needs a restart. Level names are lowercase and matching is case sensitive, so WARN is not warn: an unrecognised value given this way is treated as off, not rejected, so a typo silences everything. On an agent, --verbose forces trace whatever --log-level says.
• Without a restart: on the server, the log_level runtime-configuration key (PUT /api/config/log_level, its twin PUT /api/v1/config/log_level, or the MCP tool set_runtime_config, which a supervised-tier agentic worker may call: needs Infrastructure:Write, applied immediately and persisted, so it survives a restart, and GET /api/v1/config shows who last set it and when; an invalid value is rejected; see REST API: Runtime Configuration); on an agent, the agent_actions plugin’s set_log_level action, which is in-process only and lasts until that agent restarts (see the agent --log-level flag in device-management.md).
• Precedence: a log_level stored through runtime configuration is re-applied when the server boots, after the command line and environment, so it overrides --log-level. If --log-level warn appears to have no effect, check the stored value first; and if you stored warn to quieten these lines, remember to store info again afterwards.

Retiring or gating these lines once the benchmark concludes is recorded in docs/spark-flip-gate.md §7.

vNEXT — gateway-fronted agents stay dispatchable across circuit-recovery replays (HA WS-4 4.4, #4246 #6; NOT breaking)

New, non-breaking, purely additive. No operator action required.

Before this change, a gateway that lost and regained its connection to the server (a replica failover or an ordinary network blip) would replay its held agent registrations — and on EVERY such replay, the server wiped that agent’s placement (gateway_node/capabilities) before deciding whether to reuse or refuse the session, silently making the agent unreachable via that gateway until it happened to reconnect on its own. This was reachable on a single, otherwise-healthy replica; no core restart was required.

This section applies to circuit-recovery replays, not to a server-only restart while a gateway stays connected; for that case see What happens when the server restarts (a gateway at this version re-registers the sessions the server reports unknown).

What changes: the server now decides adopt-vs-refuse for a replayed session before installing anything, the gateway re-announces the agent’s own connection to converge placement on a successful adopt, and a genuinely superseded/stale replay is refused outright rather than silently accepted. New Prometheus counters (yuzu_gw_upstream_notify_dropped_total, yuzu_server_gateway_route_desync_total{outcome="session_superseded"}) give visibility into both the normal drop path and a refused replay. At fleet-wide reconnect-storm scale a re-announcement can still be dropped under load; that case is no longer stuck forever — the row now ages out and is purged within the route’s existing lease TTL+grace window instead of being kept alive indefinitely by ordinary heartbeat renewals. This makes the stuck row observable (a new drop counter, plus the shortfall desync outcome on the next heartbeat) and eligible for reclaim, but it is not an instant fix — actual re-convergence still needs the next circuit-recovery replay or the agent’s own reconnect. See docs/adr/2002-high-availability-architecture.md §7c for the full mechanism and two remaining known limitations: (1) a large fleet recovering from an outage longer than the lease grace window sees a transient wave of reclaim activity rather than instant convergence, and (2) a replay refused because the routing directory itself was degraded at that moment is not retried within that recovery cycle and can strand an agent server-unknown until its own next reconnect (#4634).

Update (#4632, 2026-10-05): a gateway that includes #4632 holds a re-announcement it cannot send at once (10 notifications already in flight, or the circuit breaker not closed) and sends it first in first out, instead of dropping it as described above. It is lost at admission, and counted in yuzu_gw_upstream_notify_dropped_total{reason="at_capacity"}, only when the store is full: notify_pending_max keys are held, counting the key of a half open probe that is out. A notification lost after admission (a failed or killed worker, or a gateway stop with entries held) is not counted there. See “Stream Status Notification” in gateway.md.

vNEXT — multi-cluster gateway mode now binds each agent to its own cluster, closing a claim-then-answer hijack (#4669; NOT breaking, gateway-fronted multi-cluster deployments only)

New, non-breaking, purely additive. No operator action required for single-gateway (non-multi-cluster) deployments — read on only if you run --gateway-cluster-addr (multi-cluster gateway mode).

Before this change, a rogue or compromised gateway process could re-register an agent identity that was already approved and live on a DIFFERENT cluster (ProxyRegister needs no per-agent secret for an already-approved agent), then legitimately answer for it — intercepting command payloads and forging terminal results for that agent, despite each --gateway-cluster-addr entry conceptually representing a separate trust zone.

What changes: each agent now durably binds to the first cluster that legitimately confirms its connection (trust-on-first-use). A later claim from a DIFFERENT cluster for an already-bound agent is refused — for the ordinary case, before anything is published to the routing directory or the in-memory dispatch path; on a definitive conflict caught only at the durable write (a rare defense-in-depth case — the fast pre-check missed it, e.g. under a transient store hiccup), an in-memory placement already published moments earlier is reverted rather than left live — and the refusal is audited (gateway.cluster_affinity_violation, see audit-log.md) and counted (yuzu_server_gateway_route_desync_total{op="announce_connected",outcome="cluster_affinity_violation"}, alerted via YuzuGatewayClusterAffinityViolation). An agent’s affinity re-binds when the previously-bound cluster has been genuinely unreachable for the full stale-route grace window and no session is currently claiming the row — a single rogue registration alone (with no completed connection) parks the row instead, preserving its affinity, precisely so a rogue cannot force a re-home just by registering (an operator-forced re-home route is tracked as a follow-up, #4696 — not available in this release).

Rollout window — read this if you run multi-cluster gateway mode today. Every agent that was already connected before you upgrade to this version has NO bound affinity until its NEXT CONNECTED notification — until then, that agent is still open to a same-shape claim from any cluster, exactly as before this fix. Long-lived gateway↔agent connections may not reconnect on their own for a long time. To close this window immediately across your whole fleet, force your agents to reconnect after upgrading — each forced reconnect binds that agent’s affinity right away rather than waiting on an organic reconnect. Restarting your gateway processes is one way to force that, but restart them only if you accept the risk described in gateway.md “What happens when the server restarts”: a gateway restart disconnects every agent it fronts, and on agents without the #5183 fix (shipped in 0.15.0) it can leave released agents wedged (if you do, go one node at a time).

This does not provide full trust-zone isolation between clusters — a gateway’s claimed cluster_id is still not cryptographically bound to its actual identity (mitigation 2, tracked separately, not implemented). Do not present multi-cluster gateway mode as providing that guarantee. Multi-cluster gateway mode has no production deployments as of this release.

vNEXT — a command forwarded to a gateway-connected agent always resolves instead of getting stuck at RUNNING (#4672; NOT breaking)

New, non-breaking, purely additive. No operator action required.

Before this change, a command dispatched to a gateway-fronted agent (multi-cluster gateway mode — see ha-postgres.md and ADR-2002 §7) could get stuck at RUNNING forever with no terminal signal if the forward itself failed: the gateway’s mgmt-plane peer pin rejecting this server’s certificate, the gateway staying unreachable after retries, a target cluster with no configured --gateway-cluster-addr, or a response that could not be attributed to the intended agent. The executions drawer and any API caller polling that command’s status saw it idle indefinitely.

What changes: every one of those cases now resolves the command to a terminal FAILURE, at most once, with error_detail (the field the REST/MCP response surfaces — there is no separate structured error.code) prefixed with a specific reason code you can use to diagnose the cause, e.g. [gateway_unavailable] Gateway unreachable after 3 attempts — command not delivered:

Recovery: there is no automatic re-drive for a gateway-forward failure — re-dispatch the command once the underlying cause is fixed. Automatic durable retry is tracked as a follow-up in #4690. One case is not yet covered by this fix: a gateway response stream that closes cleanly with zero frames still leaves the command stuck at RUNNING (tracked separately, #4691).

vNEXT — human API-token self-rotation is now reachable under the default config, and covers your own MCP-tiered/scoped tokens (#2963; NOT breaking)

New, non-breaking, purely additive. No operator action required.

Before this change, POST /api/v1/tokens/{id}/rotate/.../confirm (and the MCP twins rotate_api_token/confirm_api_token_rotation) composed the shipped RBAC-off default with the store’s self-service-only ownership check into something reachable by nobody but an admin out of the box — a plain non-admin owner of a token got 403 trying to rotate their own credential. Separately, a dashboard/cookie session could never rotate or confirm its own MCP-tiered or service-scoped token — only that exact token’s own credential could, which was backwards precisely when the token’s secret is under suspicion.

What changes: any authenticated, non-admin owner of a token can now self-rotate it under the default configuration, and a plain interactive session can now rotate/confirm any of its own tokens regardless of that token’s own tier/scope. Nothing that previously succeeded now fails — this only widens which previously-403/400’d callers now get 200. The minted successor still always inherits the token’s own tier/scope and expiry verbatim; no caller can mint a credential broader than the one it replaces. A token within 24 hours of its own expiry still cannot be rotated (mint a new one instead) — unchanged, by design.

See docs/user-manual/authentication.md “Rotating a Token” for the full operator-facing detail, and docs/security-reviews/2963-token-rotation-default-permission-2026-09-17.md for the decision record.

vNEXT — CRL publishing is serialised in Postgres, and a missed CRL now republishes itself (HA WS-6 6.1, #4126; NOT breaking)

Every CRL publish (startup, an operator revoke, a subordinate-CA import, and the freshness re-publish) now runs as one Postgres transaction under a lock on the ca_store.ca_crl_versions table, instead of behind a lock inside one server process. The ca_store schema migrates to v4: v3 adds a nullable revoked_count column on ca_crl_versions (nothing is backfilled), v4 adds a trigger that refuses to delete or change a revoked ca_issued row. A subordinate-CA import (POST /api/v1/ca/import-chain) now waits for any CRL publish already in progress before it swaps the root; if that takes longer than the database’s lock timeout (10 s by default) the import fails with a database error and can simply be retried.

What changes on every deployment, including single-server:

• A CRL publish can now fail on lock contention. A publish that waits more than 5 s for the table lock gives up. On POST /api/v1/ca/revoke and MCP revoke_certificate that shows as crl_republished:false plus a ca.crl.published failure audit; every trigger increments yuzu_server_ca_crl_publish_failures_total. The revocation itself still takes effect immediately server-side.
• A revocation missing from the served CRL is now republished automatically. Previously, if a revoke’s own CRL publish failed, the revocation stayed out of GET /api/v1/ca/crl until the next revoke or until the CRL was within 24 h of its nextUpdate (up to ~6 days); retrying the revoke returns “already revoked” and does not publish. The freshness pass now also republishes, on its next 15 s tick, whenever the latest CRL was not built from the current revoked set (after a failed attempt it waits 5 minutes before trying again). You may occasionally see a redundant CRL version shortly after a revoke.
• A revoked default server certificate now stays revoked. Previously, regenerating the built-in default certificates (for example after changing --cert-san) deleted their old inventory rows, including revoked ones, so a revoked default leaf was accepted again and dropped from the CRL. Revoked rows are now kept, and the database now refuses to delete them (migration v4 trigger) — during a rolling upgrade an older server’s default-cert purge fails with “failed to purge prior default-cert inventory rows” instead. This does not restore a revocation already lost that way: the purged serial is no longer in the inventory, so revoking it again returns 404. This only affects you if you revoked a default server certificate (for example because its key may have leaked) and the default certificates were regenerated before you upgraded to this release. Check first: find the serial in the audit log (the ca.cert.revoked event for that revocation; if that event has aged out of audit retention — 365 days by default — and you cannot establish the serial another way, you cannot tell whether it was lost: treat it as lost), then page through GET /api/v1/ca/issued with an unconfined account (follow offset until has_more is false; a management-group-confined account, or one with no Inventory:Read grant, is refused that list) or decode the CRL (curl … /api/v1/ca/crl | openssl crl -inform DER -noout -text). If the serial is still listed as revoked, nothing was lost. If it is missing, re-root the internal CA with the clean re-root in docs/pki-architecture.md (“Deliberate clean re-root”) — POST /api/v1/ca/import-chain is not enough, because it keeps the issuing key the leaked certificate chains to. A re-root re-enrolls the whole fleet.

Single-server remains the only supported topology. If you nevertheless run two server versions against one database during an upgrade, a publish from the older binary does not take the lock and can still publish a CRL that omits a revocation the newer binary just recorded; the newer binary’s freshness pass republishes to cover it (and can do so on every tick for as long as the older binary keeps publishing).

Rollback to the previous release is safe: an older binary boots against the v4 schema and ignores the new column. It will not self-heal a missed CRL (rolling forward again republishes once), and its default-cert inventory purge fails while any revoked default leaf exists (logged, harmless) because the trigger stays in place — which is the point: it cannot un-revoke anything. Multi-replica PKI also still needs WS-6 slices 6.2 (enrollment) and 6.3 (CA key and KEK custody): today the freshness pass runs only on the elected leader, and a leader whose CA directory lacks the CA key can never publish.

vNEXT — the server now elects a background-work leader at startup (HA WS-3; NOT breaking)

New, non-breaking, and inert on a single-server deployment. As one step toward making a second server replica safe, the singleton background loops that dispatch — scheduled instructions, policy remediation, quarantine containment reconciliation, and the CRL freshness re-publish — now run only on a fenced leader elected over a dedicated Postgres coordination connection (ADR-2002 §3). On a single server the sole replica is always the leader, so behaviour is unchanged.

The elector alone is not a reason to run a second server replica. It ensures only the leader attempts the dispatching loops. The correctness guarantee that a paused ex-leader cannot still commit a leader-driven dispatch is the epoch fence in the claim write (the command outbox, since the follow-on slice 3.3; operator-triggered remediation is arbitrated by its own durable per-target claim instead). A supported active-active deployment additionally needs gateway-fronted routing, shared agent presence, PKI HA and the rest of the safe-to-scale gate (docs/ha-delivery-matrix.md); that gate is not lifted. Single-server is the only supported topology today.

What you will see, on every deployment including single-server, are new startup log lines — these are routine, not a fault:

• leader_elector: coordination connection established (host=… port=… dbname=…; dedicated, never-recycled)
• leader_elector: election loop started (poll=5s)
• leader_elector: acquired leadership 'server_background_leader' at epoch N (holder_id=srv-…)

If instead you see [HA] leader_elector could not open its coordination connection; FencedLeaderOnly background loops are PAUSED …, the server could not reach its Postgres coordination connection: it keeps serving and re-tries every election cycle (so a transient blip self-heals within seconds), but while the message persists the leader-only background loops (the four above, plus command-outbox delivery) do not run. This is fail-closed by design — a paused loop never double-dispatches — and a persistent occurrence is a Postgres-reachability problem to investigate, not a server bug. (The operator-triggered paths are unaffected: a manual policy remediation or evaluation, and an operator CRL revoke, run on whichever replica received the request — and a remediation/evaluation is also completed on that same replica, so it still reaches a terminal verdict even while that replica is not the leader. Only the leader-owned scheduling half — the automatic due-policy dispatch and the periodic CRL freshness re-publish — pauses.)

Leader election: metrics and alerts (HA WS-11)

Running more than one server replica is not supported today (the safe-to-scale gate in docs/ha-delivery-matrix.md is not lifted); the multi-replica wording below is for pilots and pre-GA testing. On the supported single-server deployment YuzuLeaderSplitBrain cannot fire (one series); in practice only YuzuLeaderNoLeader and YuzuLeaderCoordinationStale fire, and YuzuLeaderFlapping only on a Postgres connection flap or a restart loop whose acquire is delayed.

Each replica exposes the elector’s state as yuzu_server_leader_* series on its own /metrics (reference: metrics.md “Leader election metrics”), and docs/prometheus/yuzu-alerts.yml ships the yuzu-ha-leader group. Reload your Prometheus rule file to pick it up. Prometheus must scrape each replica individually (not through the load balancer), and unrelated Yuzu deployments sharing one Prometheus need separate jobs (the shipped scrape configs all use job_name: 'yuzu-server', so a second stack must be renamed), because the alerts aggregate sum by (job). /readyz does not include leadership: a replica that cannot lead is a ready follower, so these alerts are the only signal that the leader-only passes (scheduled dispatch, policy dispatch, quarantine reconciliation, the command-outbox delivery loop and the CRL re-publish) are paused. The thresholds below (2 m, 5 m, 6 acquisitions per 30 m, 120 s) are initial claims with no multi-replica fleet data behind them; tune them from your own history.

Every one of these alerts needs a successful scrape, and no server up == 0 rule ships. If the primary freezes while its kernel keeps acknowledging packets, the /metrics handler’s own database queries hang too, the scrape fails, every yuzu_server_leader_* series goes stale at once and every rule below evaluates empty. Configure an up{job="<your server job>"} == 0 alert (the shipped scrape configs name the job yuzu-server; use yours, with a for: of a few scrape intervals) and probe /readyz from a blackbox exporter or your load balancer as the backstop. A hung scrape becomes up == 0 only after Prometheus’s scrape_timeout. No server dead-man’s-switch rule ships: #2956 tracked it and was closed without building it.

Find the replica that is the leader with yuzu_server_leader_is_leader == 1; the cluster’s authoritative record is SELECT * FROM leader_elector.leader_state; (its holder_id matches the (holder_id=srv-…) text in the acquired leadership log line of the process that holds it).

• YuzuLeaderSplitBrain (critical). More than one replica believes it leads for over 2 minutes. First, compare each replica’s coordination host/dbname in its startup log line (coordination connection established (host=… dbname=…)). Against the real elector the usual cause is replicas pointed at different databases (a split --postgres-dsn, or a Patroni double-primary). In that case each replica holds a valid lock and a valid epoch in its own database, so the epoch fence does not span them: both replicas run every leader-only pass, and on a forked database the same scheduled occurrence can be sent twice (only the agent’s command_id dedup, which is effectively-once, absorbs it). Then establish which database is authoritative: patronictl list, and compare SELECT system_identifier FROM pg_control_system(); on each side (two different identifiers are two different clusters; hostnames, aliases and a VIP can make “different” hosts the same cluster, and a Patroni demotion followed by pg_rewind discards the demoted side’s divergent writes, so do not demote on a log-line comparison alone). Then fix the DSN, or fence or demote the wrong primary in Patroni, before restarting any replica: a restart just reconnects to whichever primary the DSN resolves to. Only when every replica shares one database is the stale replica the one whose yuzu_server_leader_epoch differs from leader_state.current_leader_epoch; restart it. Within one database fenced claims stay safe (assuming no committed transaction was lost on a promotion, that is, synchronous replication, the shipped quorum default; the fence is not claimed safe across a lossy asynchronous promotion) and what runs twice is the unfenced background passes. What it does not catch: a paused ex-leader (the case the epoch fence exists for) — its process is frozen, so its series goes stale, not 1, and a frozen process serves no /metrics, so no alert on its own series can evaluate. That case is covered by the fence itself (and YuzuLeaderNoLeader fires after 5 minutes if no other replica takes over); YuzuLeaderCoordinationStale is the signal for a leader that is alive and serving /metrics but whose coordination connection has stopped answering. This alert is verified by Prometheus rule unit cases only; a genuine two-leader state cannot be produced against one database.
• YuzuLeaderNoLeader (critical). No replica has believed it leads for over 5 minutes, so the leader-only passes are paused on every replica while /readyz stays green. Step 0: scrape every replica’s /metrics directly and check up for each. If any replica reads yuzu_server_leader_is_leader 1 this is a scrape problem, not a leader problem (a leader whose own scrape target is down leaves only the followers’ zeros, which sums to 0). Otherwise check, in order: the server log for [HA] leader_elector could not open its coordination connection and the yuzu_server_leader_acquire_failures_total{cause} series (connect rising means the dedicated connection cannot be made, or its schema migration failed: reach the primary directly, not through a transaction-mode pooler); then, in Postgres, SELECT a.pid, a.client_addr, a.state, a.backend_start, a.state_change FROM pg_locks l JOIN pg_stat_activity a USING (pid) WHERE l.locktype = 'advisory' AND l.classid = 1735291205; Compare client_addr with the replicas: a lock held by a backend that no live replica owns is a zombie lock holder (a half-open connection the server side has not noticed; Postgres notices only via its own keepalives, which can take hours by default). On a single replica the zombie is typically that replica’s own previous connection after a network blip of 15 seconds or more (backend_start older than the server process, state idle), and the replica’s acquire_failures_total stays flat because losing the try-lock to a holder is the normal follower outcome and is not counted. Only that backend should be terminated: SELECT pg_terminate_backend(<pid>); (this needs superuser or membership of pg_signal_backend), after which a replica acquires within one election cycle. Terminating a live leader’s backend forces a failover. Fenced claims stay safe; for up to about 8 seconds (one heartbeat interval, plus any pass already in flight) before the old leader notices, its unfenced passes (schedule tick, policy dispatch, quarantine reconcile, CRL re-publish) can overlap the successor’s, and any duplicate dispatch is absorbed only by the agent’s command_id dedup. During a restart of the only replica the series is absent and the alert does not fire. Silence this alert during planned Postgres maintenance or a switchover that will take more than 5 minutes: scope the silence to the job, end it at the window plus about 10 minutes, and afterwards confirm a replica reads yuzu_server_leader_is_leader 1.
• YuzuLeaderFlapping (warning). Six or more acquisitions in 30 minutes across a multi-replica deployment: a network or Postgres failover flap, or a handover loop. Compare with YuzuServerPostgresUnreachable and yuzu_server_leader_lost_total{cause="liveness_failed"}. A rolling upgrade of an N-replica deployment legitimately adds up to N acquisitions; the threshold leaves room for four replicas. It usually does not detect a single replica in a restart loop: the counter is synced at scrape time, so a boot whose leadership is acquired before its first scrape reads 1 with no rise, whereas a boot whose acquire is delayed (slow or unreachable Postgres, a lock still held by the previous process) is counted. Do not rely on this rule for that failure; use YuzuServerRestartLoop (which has its own documented limits).
• YuzuLeaderCoordinationStale (critical). A replica still reads leader but its coordination connection has not answered for over 120 seconds. This is the only series that moves when a network partition lets the peer’s kernel keep acknowledging packets and only the dedicated coordination connection wedges: the heartbeat then blocks inside a database call that has no client-side time bound against a frozen-but-acknowledging backend (about 100 seconds is merely how long one measured pause lasted), the replica’s leadership belief stays 1, so the count of leaders stays at one and both YuzuLeaderSplitBrain and YuzuLeaderNoLeader stay silent while that replica’s leader-only passes keep attempting, and its fenced claims cannot commit (its unfenced passes may still succeed). Fenced claims still cannot double-dispatch (the epoch predicate is evaluated in the claim’s write on an ordinary pooled connection). Detection takes about 4–5 minutes (age above 120 s, held for 2 minutes, plus a scrape interval); a shorter freeze never alerts. Check connectivity from that replica to the primary. If the primary is reachable, terminate the stuck backend as above (that ends the lock and unblocks the stale replica, which then drops its belief; its unfenced passes can briefly overlap the successor’s, with the same command_id-dedup caveat); otherwise restart the replica. A restart can take as long as the supervisor’s stop timeout, or need a second signal, while the coordination call is wedged; if the old backend still holds the advisory lock afterwards, terminate it (see YuzuLeaderNoLeader). Blind spot: when the whole primary is frozen the scrape itself fails (see the note above), so this rule cannot evaluate: rely on the up == 0 backstop.

vNEXT — Guardian compliance % and the fleet honesty banner now reflect Linux Service Guards correctly (#4252; NOT breaking)

New, non-breaking, and inert on a fleet with no deployed Service-type Guards. The /guardian dashboard’s compliance rollup previously treated every non-Windows agent as “not implemented” for every Guard type — correct for Registry/File (genuinely Windows-only), but wrong for Service, which arms (observe-only) on Linux today. A Linux agent with a real compliant/drifted status row for a deployed Service Guard was folded into that Guard’s “not implemented” bucket a second time, corrupting the ”% compliant” headline and the fleet/by-baseline breakdowns.

What you will see, if your fleet has Linux endpoints with deployed Service-type Guards: the affected Guard(s)’ ”% compliant” figure will likely move — typically upward, since the double-count previously inflated the denominator without inflating the numerator. If some of those Linux Service Guards never actually arm (see the known limitations below), part of any upward movement is those pairs going from “counted as not implemented” to “not counted at all” — also raises the %, but is coverage disappearing, not new enforcement; don’t read the whole jump as pure correctness without checking D-Bus/systemd reachability on the affected endpoints. The fleet honesty banner (the list of agents flagged as running on a platform Guardian doesn’t fully cover) also narrows: it now names an agent only if it owns an actually-unenforced Guard pair, not merely for being on a less-covered platform, so a Linux agent whose only deployed Guard is a Service Guard may disappear from that banner. Nothing about how a Guard is authored, deployed, or enforced changes — this is a dashboard-reporting correctness fix only, with no REST/MCP-visible effect (the /api/v1/guaranteed-state/status endpoint never applied this fold and is unaffected).

Two known, accepted limitations of this fold — see guaranteed-state.md for detail and metrics.md’s yuzu_server_guardian_platform_matrix_stale_total entry for the diagnostic signal: a Service Guard that never actually arms on a genuinely-supported platform (no system D-Bus — every containerized/compose Linux agent, a disabled build flag, or an invalid unit name) now silently drops out of the denominator rather than showing “not implemented”; and a status row surviving a guard-type revision on the same rule id can render a now-unenforced pair as compliant with no marker on that rule (tracked as #4263, not yet fixed).

vNEXT — scheduled instruction fires now go through a durable command outbox (HA WS-3 3.3; breaking for SIEM/audit-count assurance)

Scheduled instruction fires no longer dispatch to agents inline from the poller. The poller now (a) creates the tracked execution row and (b) commits a durable pending occurrence to a new born-on-PG store, CommandOutboxStore (schema command_outbox_store); a leader-gated delivery loop drains that outbox and performs the actual wire dispatch. No operator action is required — the store’s schema migration runs automatically on upgrade (fresh-start/ no-backfill, since this store never existed before 3.3).

Behaviour changes you will see:

• Added dispatch latency. A scheduled fire is still enqueued within ~30 seconds of its due time (the poller cadence is unchanged), but actual dispatch to agents now follows within ~5 seconds of enqueue via the delivery loop — up to ~35 seconds total, where it was previously inline/ immediate.
• The audit trail for one scheduled fire is now two events, not one. instruction.schedule_fired (result queued) marks the enqueue; a new command.outbox_delivered (result success/failure/denied) marks the actual delivery outcome. Any SIEM correlation rule or audit-count assurance built on “one audit event per scheduled fire” must be updated to expect the pair, or to key off command.outbox_delivered for the delivery outcome.
• New metrics. Eight counters and a backlog gauge — yuzu_server_command_outbox_* — cover enqueue-side degrade and the delivery loop’s outcomes (delivered / no-agents / denied / retry / errors / decode-failed / degrade) plus yuzu_server_command_outbox_pending, the primary signal that scheduled dispatch has stalled. Full reference: docs/user-manual/metrics.md “Command outbox delivery metrics”.
• ADR-1007 per-device concurrency is no longer enforced for scheduled fires. The inline path previously resolved a concurrency_mode and gated on it; the outbox delivery path dispatches on the plain confined path without that claim. This is a deliberate, tracked gap — restoring per-device concurrency enforcement for the outbox path is a follow-up, not an oversight.

Rollback note: a pending outbox row committed by a 3.3-or-later binary is invisible to a pre-3.3 binary, which has no outbox reader. Rolling back mid-flight leaves that occurrence undelivered until you roll forward again — a narrow window, and no data corruption (the row stays durable in Postgres and delivers as soon as a 3.3-or-later binary is running and holds leadership).

vNEXT — manual policy remediation is now claimed durably, cross-replica (HA WS-3 3.4; breaking for automation asserting agents == len(agent_ids))

POST /api/policies/{id}/remediate now arbitrates its per-target claim through a durable per-(policy, agent) row in PolicyStore (policy_status.remediation_claim_at), instead of an in-process-only guard. This closes the cross-replica gap noted in the ADR-0056 Follow-ups: two replicas racing a remediate call for the same policy/agent can no longer both dispatch a fix. (This is the guarantee for when the safe-to-scale gate lifts, ADR-2002 §7c; exactly one server replica is supported today.)

Scope of the guarantee. This holds between replicas running the same schema version. A rolling upgrade across the migration boundary can transiently run an old binary that predates the durable claim alongside a new one; that mixed-version window is covered today only by the safe-to-scale gate (docs/ha-delivery-matrix.md, which is procedural and not lifted), pending a durable cluster-capability admission gate (#6528). The exposure is one-directional: a row written by an old binary carries claim-generation 0, a value a new binary never mints. Separately, this is effectively-once, not exactly-once (consistent with ADR-2002): a replica that pauses for longer than fixing_stale_seconds between winning a claim and actually sending a command over gRPC — the fix, or the subsequent post-fix verify — can still dispatch that one send after a sibling has reclaimed the target — the wire send is not itself transactionally fenced. The durable claim prevents concurrent or duplicate claims; it bounds, but does not make impossible, a single late duplicate dispatch from a long-paused claim-holder.

Behaviour changes you will see:

• A new 409 cause. POST /api/policies/{id}/remediate can now refuse with 409 and body message "remediation already in flight or retry cap reached for this policy" when a target is already claimed for remediation (by this replica or a sibling) or has exhausted its fix-retry cap for this policy — in addition to the existing no-fix-instruction and no-non-compliant-agents 409 causes.
• agents in the 202 response is now the delivered count, not the attempted count (breaking). A claimed-but-undelivered target (offline, quarantined, plugin absent) releases its claim without consuming a retry attempt and is excluded from agents. Automation asserting agents == len(agent_ids) (or == number of non-compliant agents for an omitted agent_ids) must be updated to tolerate agents being smaller than the number of targets requested.

Migration note: schema migrations v2–v4 add policy_status.remediation_claim_at (v2), policy_status.remediation_claim_gen (v3, the ABA claim-generation fence), and the remediation_claim_seq sequence (v4) — all BIGINT NOT NULL DEFAULT 0 columns plus one sequence. They run automatically on upgrade, are metadata-only, and require no operator action and no downtime.

Post-restart note: after a restart, a durable claim left behind by the previous process may briefly block re-remediation of the agents it was mid-flight for. This self-heals once the claim ages past the staleness window (fixing_stale_seconds, default 1800s) — the same window claim_due_policies’s own stranded-fixing sweep uses.

vNEXT — gateway management plane now pins its peer (#1422, breaking for custom gateway configs)

The gateway’s :50063 command plane requires, on any network-reachable listener, strict mTLS plus an SPKI peer pin (auth_fun => fun yuzu_gw_authz:check_mgmt_peer/1 and {yuzu_gw, mgmt_peer_pins}) — and the gateway refuses to boot without them (mode identity: without a resolvable server identity; mode identity_or_pin: with neither a pin nor a resolvable identity, #6016). Two upgrade orderings fail, in opposite directions:

• New gateway image + your old custom mTLS mgmt config → boot refusal. Add the auth_fun and mgmt_peer_pins entries (copy the shape from gateway/config/sys.config.prod) before pulling the image. The refusal log names every missing piece.
• New repo-checkout config + an old (pre-pin) gateway image → the config references yuzu_gw_authz, which the old image does not ship: every mgmt stream dies with undef and command forwarding is silently dead. Update the image and the mounted config together — a dev checkout’s reference-gateway-sys.config is ahead of the pinned release image until the next release ships.

BYO-cert installs: point mgmt_peer_pins at your server certificate — which must carry the serverAuth EKU — or paste its SPKI SHA-256; pin-list edits take effect on gateway restart. A pin is not the only way to admit the server any more (#6016): with the built-in certificates the gateway can instead admit it by the identity URI SAN yuzu://<install>/server that the install CA puts on the server’s certificate (mgmt_peer_auth_mode identity_or_pin or identity), which needs no gateway change when the server’s key is renewed when no pin is configured (a pin kept in identity mode must follow the renewal). It is an opt-in: the shipped reference configs and an unset mode are pin, and the identity trusts the issuing CA’s issuance policy, so it must not be enabled behind a CA that lets a requester choose a URI SAN. Rolling the gateway back to a binary from before #6016 ignores these settings: the deny-list is no longer honoured, and a configuration with no pins (dropped for identity, or identity_or_pin running on the identity alone) refuses to boot (no_mgmt_peer_pins); restore the pins, matching the key the server presents now, first (see the “Rolling back” paragraph of that section). The rollback is a gateway restart, so do it only if the node’s agents can take the disconnect, because agents without the #5183 fix stay wedged after it. Bring-your-own certificates, --no-default-certs and a server given its own --cert/--key carry no identity and stay on pins. See Authorizing the server by its identity and docs/adr/2004-gateway-mgmt-server-identity.md. {allow_insecure_mgmt, true} is a lab-rig acknowledgement only; never set it where :50063 is reachable from untrusted networks.

vNEXT — the wifi plugin’s Linux output changes shape (breaking)

The wifi plugin moves to reading NetworkManager directly over D-Bus on Linux (plugin version 1.1.0). Three operator-visible changes to the records it emits:

• connected field 6 changes meaning on Linux. It now carries the network interface (for example wlan0). Earlier agents put the NetworkManager connection profile name there. Any dashboard query, export or script that reads that field as a profile name needs updating.
• Unsecured networks report Open, not NONE. Previously the value depended on which acquisition path answered; all paths now agree on Open.
• WPA3 and enhanced-open networks are reported distinctly. They previously appeared as WPA2, so a fleet may show WPA3 networks for the first time after upgrading. This is a reporting change, not a change on the network.

During a staged rollout the two shapes are mixed, and the record does not say which one it is. An un-upgraded agent reports the profile name in field 6 and an upgraded agent reports the interface, with no discriminator in the row — unlike the firewall change, where a backend| value distinguished per device. Until the fleet is fully upgraded, treat field 6 as free text: match on agent version rather than on the field’s content, or defer any query that keys on it. SELECT on the plugin version, not on the value.

On macOS 14 and later list_networks still normally returns no networks. airport was removed in that release, and the system_profiler fallback the plugin now tries needs a Location Services grant that a background daemon does not normally have. Without it the plugin reports a single wifi|info row (wifi:macos_scan_requires_location_services) instead of an empty scan, so the cause is visible; networks appear only on a host where Location Services has been granted.

vNEXT — event_logs acquires natively; Windows message text and count semantics change (breaking)

The event_logs plugin now reads the event log in-process — wevtapi on Windows, sd_journal on Linux (falling back to a bounded journalctl invocation where libsystemd is unavailable) — replacing the previous PowerShell and shell-out legs. Automation that parses event_logs rows needs review before upgrading; the changes fail silently (fewer rows, or a rule that stops firing), not with an error.

What changes for a row consumer:

• Windows message column is now the event’s EventData parameter values, space-joined, instead of the provider-formatted message template. A rule regex-matching template prose (for example "The user account was locked out") will stop matching; match on event ID and provider instead.
• Windows timestamps are the event’s UTC SystemTime at full precision, where the previous leg emitted the PowerShell-rendered local time.
• Windows count bounds the events examined, not the matches returned — the filter is applied within the newest count events. Linux and macOS return up to count matches. When the window fills without satisfying the query the result now carries a constrained status rather than reporting an absence.
• Linux keyword filtering is a case-insensitive substring match on both rungs, where the shell-out leg used journalctl --grep regular expressions.
• hours and count reject trailing or leading non-digits ("12x", " 24", "+8") and fall back to their defaults; the previous parse silently accepted the leading digits.
• A failed, denied, or bounded read now reports a typed status (permission_denied / unavailable / constrained) instead of an empty result. Consumers that treated “no rows” as “healthy” will now see the difference — this is the point of the change, but it is a behaviour change.

Mixed-fleet blend during rollout. Upgraded and non-upgraded agents emit different message columns for the same Windows event, and there is no per-row field identifying which leg produced it. Expect a blended view until the fleet is fully upgraded, and prefer event ID plus provider for any rule that must hold across both.

vNEXT — a duplicate SCIM externalId now refuses to boot (ADR-2001, CC6.8) (breaking)

What changed. This release adds a partial unique index, scim_resources_external_id_uniq ON scim_resources (external_id) WHERE external_id IS NOT NULL, applied by the ScimStore migration that runs at construction — unconditionally, regardless of whether --scim-enable is set. Postgres itself detects any pre-existing duplicate non-empty external_id and raises a unique_violation, which fails the migration; ScimStore then reports !is_open() and the server refuses to start, the same fail-closed posture every born-on-PG store uses on a failed migration. This is deliberate, not a bug to work around: a duplicate externalId is exactly the mis-link hazard ADR-2001’s SCIM↔OIDC token-revoke linkage exists to prevent (a duplicate would let link formation pick a SCIM resource arbitrarily and revoke — or fail to revoke — the wrong principal’s tokens), so the index is a stronger posture than what shipped before, not a regression to relax.

Who this affects. Any deployment with existing SCIM data (has ever run --scim-enable, even if it isn’t set on this upgrade) where two or more scim_resources rows share the same non-empty external_id — most likely from an IdP connector bug, a manual data-fix gone wrong, or a resource re-provisioned under a new SCIM id without the old row ever being cleaned up.

Before upgrading, run this against your existing database to check:

SELECT external_id, COUNT(*) FROM scim_store.scim_resources
WHERE external_id IS NOT NULL
GROUP BY external_id
HAVING COUNT(*) > 1;

If it returns any rows, the upgrade will fail to boot until you resolve the duplicates first — deactivate or delete the stale/incorrect row(s) for each external_id this query surfaces (keep whichever row is the IdP’s actual current resource), then re-run the query until it returns nothing. If it returns no rows, this note does not affect you and the migration applies silently on next boot.

vNEXT — pre-auth request-body caps now apply to (almost) every route, not just /mcp/ (#2407) (breaking)

What changed. Before this release, only the /mcp/ ingress was capped ahead of authentication (4 MiB, #2437); every other route was bounded only by httplib’s own 100 MiB server-wide default, so an unauthenticated caller could make the server buffer up to 100 MiB per connection on any route. The server now resolves a per-route-class cap from a single table (body_cap_policy.hpp) at the same pre-routing chokepoint, before the body is read at all, including GET/HEAD requests (a GET declaring a Content-Length still buffers a body regardless of method) and the four unauthenticated health-probe paths (/livez, /readyz, /health, /api/health), which were previously the last unauthenticated 100 MiB buffer on the server because their early-return sat above the cap; they now hit the cap first and remain exempt from the on-behalf-of guard and the general rate-limit bucket (/health and /api/health requests that carry a credential header are counted in a separate per-source bucket, see Health Endpoints). Separately, a request declaring a compressed (non-identity) Content-Encoding is now refused outright (415) on every class, because httplib decompresses transparently and bounds only the decompressed size against its own 100 MiB global limit; a sub-cap compressed body could otherwise expand to ~100 MiB before this cap (or any handler) ever saw it. Full per-class numbers and the error contract: docs/user-manual/rest-api.md “Pre-Auth Request Body Caps”.

What a client sees.

• 413 — the declared Content-Length exceeds the resolved class’s cap.
• 411 — the body cannot be sized in advance by framing (chunked, or no Content-Length) and this route’s class requires one it can size. Today that is /mcp/ only; every other class still admits an unmeasurable body up to httplib’s 100 MiB backstop, unchanged from before this release.
• 415 — the request carries a Content-Encoding header other than identity. Refused on every class, unconditionally — this one is new in this release and does not depend on the class opting into anything. No shipped Yuzu route accepts a compressed request body, so this should not affect a conforming client; a client or intermediary that compresses request bodies (gzip/br/deflate) must stop doing so against this server. Send an identity-encoded body by omitting Content-Encoding entirely — do not send Content-Encoding: identity explicitly, which reaches httplib’s own decompressor lookup, finds nothing to do, and today gets a bare 415 with none of this section’s error-shape guarantees (no A4 envelope, no yuzu_body_cap_rejected_total increment); see the 415 row in docs/user-manual/rest-api.md for the mechanism.

Who this affects. Any integration that was sending a body larger than its route’s new class cap and relying on httplib’s 100 MiB backstop to accept it. The most likely surface to notice: a very large POST /api/v1/guaranteed-state/rules, /api/workflows, or /api/product-packs body — those three classes are capped at a generous but explicit 16 MiB (a judgment call, not a measured contract, because no aggregate size limit for that content exists yet) rather than the previous 100 MiB backstop. Most other routes had a smaller handler-level check already (response templates, CA import) — for those, the effect really is just an earlier rejection at the same byte count (before the body is buffered rather than after). Also newly affected: any client or intermediary that sends a compressed request body — that is now refused outright with 415 on every class (see “What changed” above); no shipped Yuzu client compresses request bodies, so this is expected to affect only a custom integration or a reverse proxy doing transparent compression. One class is not the simple “earlier rejection, same effect” case, and clients should not assume “no behavior change” from it: Echo queries (POST /api/dashboard/echo-execute, POST /api/v1/result-sets/from-echo-query) — these two classes’ caps as originally shipped bounded the RAW body at the same byte count their handlers check on a DECODED/PARSED value, which is not the same number; that unit mismatch caused some legitimate queries to be rejected here that the handler would have admitted. Both caps carry a margin now (see docs/user-manual/rest-api.md for the sizing), so this class is now also a clean earlier-rejection-only change going forward. SCIM is no longer a special case either (fixed as part of this same change, D7): the pre-routing rejection now publishes SCIM’s own RFC 7644 §3.12 application/scim+json error shape, matching what a strict SCIM client already expected from the handler’s own (now-superseded) check.

How to raise a cap. Edit the table in server/core/src/body_cap_policy.hpp (with review) and the matching row in docs/user-manual/rest-api.md. Do not reach for httplib’s global Server::set_payload_max_length — that knob is shared by every route on the listener, including the ~70 MiB live-query bundle route and the OTA agent-binary upload, so a single global value can’t fit every route class at once. Rejections are visible per class via yuzu_body_cap_rejected_total{path_class,reason} — see docs/user-manual/metrics.md.

vNEXT — /auto deployment execute now enforces per-device Execution:Execute confinement (breaking)

What changed. /auto deployment advance (staging + executing an installer on a pre-flight run’s go-cohort) previously dispatched every content_dist.stage/content_dist.execute_staged command under the server’s own system authority, bypassing the caller-identity check the chokepoint performs for every other operator-facing dispatch surface — the check the route’s own permission gate depends on to confine which devices an operator’s Execution:Execute grant actually reaches. It now dispatches under the triggering operator’s real identity and — as of a second, related fix landing in the same release — is confined to that operator’s Execution:Execute-visible device set at the point of dispatch, the same per-device confinement RestApiV1/WorkflowRoutes/BundleOrchestrator/McpServer/DashboardRoutes already enforce.

Who this affects. Only deployments where a role that can trigger /auto deploy (Infrastructure:Read + SoftwareDeployment:Execute, per the permissions table in preflight.md) holds a management-group-scoped Execution:Execute grant — narrower than the full fleet — rather than a global one. For that role, a deployment whose go-cohort includes a device outside the role’s Execution:Execute scope will now correctly skip that device (visible as skipped in the deployment’s device list) instead of executing on it. A role with a global Execution:Execute grant sees no change.

Before upgrading, check for any role combining a scoped Execution:Execute grant with SoftwareDeployment:Execute — that combination is the only one affected. If none of your deployment-triggering roles hold a scoped (rather than global) Execution:Execute grant, this note does not affect you.

vNEXT — an authorization-topology floor now applies regardless of RBAC, and engine-principal reads move off Security:Read (#2376) (breaking)

This note has two independent breaking directions — read both, they affect different deployments.

Direction 1 — RBAC-disabled installs lose three non-admin reads.

Who this affects. Any deployment running with [rbac] enabled = false (the shipped default) where a non-admin, authenticated session was relying on reading one of: the fleet-wide access-review export (/api/v1/access-reviews*), GET /api/v1/rbac/roles, or the engine-principal inventory/roles (/api/v1/engine-principals* and the list_engine_principals/get_engine_principal/list_engine_roles MCP tools). RBAC-enabled deployments are unaffected by this direction (see “Why” below).

Why. RBAC ships disabled by default. With RBAC disabled, the legacy authorization fallback previously allowed every authenticated non-engine session to perform every Read — including reads of the authorization topology itself. A plain non-admin user on a default, just-installed server could read who holds what role and pull the complete access-review grant population that exists to be SOC 2 CC6.2 evidence. As of this release, AccessReview:Read, UserManagement:Read, and EnginePrincipal:Read (see Direction 2) require the admin session role regardless of the RBAC toggle. This is not configurable — there is no setting that reopens it. Full design rationale: docs/security-reviews/authz-topology-floor-2026-08-05.md.

What does NOT change. A live RBAC grant is unaffected — under RBAC-enabled enforcement, a non-admin holding the seeded Reviewer role still reaches the access-review export exactly as before; a non-admin holding UserManagement:Read/EnginePrincipal:Read (e.g. the built-in Viewer role holds both) still reaches the other two reads. The floor only engages inside the RBAC-off legacy fallback.

What to do. If a non-admin session needs one of these three reads, enable RBAC and grant the matching role:

[rbac]
enabled = true

• Access-review export → assign the built-in Reviewer role.
• /rbac/roles → the built-in Viewer role already holds UserManagement:Read, or grant it directly.
• Engine-principal reads → the built-in Viewer role already holds EnginePrincipal:Read (see Direction 2), or grant it directly.

See rbac.md “The authorization topology floor” and “Before enabling RBAC in production” before flipping the toggle on a live fleet — turning RBAC on also applies management-group-scoped device visibility immediately.

Direction 2 — RBAC-enabled deployments with a custom role granted Security:Read for engine-principal access must re-grant.

Who this affects. Any RBAC-enabled deployment with a custom role (not Administrator/Viewer, which are updated automatically — see below) that was granted Security:Read specifically so its holders could reach GET /api/v1/engine-principals* or the list_engine_principals/get_engine_principal/list_engine_roles MCP tools.

Why. Those reads moved off Security:Read onto a new, narrower EnginePrincipal:Read securable (the same move #2324 made cutting AccessReview away from AuditLog:*) — Security:Read also gates unrelated operational reads (quarantine visibility, CA issued-certs, /ca/root-csr, KEK status) that stay on Security:Read unchanged and un-floored.

What changes automatically. No schema migration is required — the new EnginePrincipal securable and its Administrator (full CRUD) and Viewer (Read) grants are seeded idempotently on every server boot (RbacStore::seed_defaults()’s INSERT OR IGNORE loops), so the two built-in roles pick it up on the next boot automatically, with no operator action, matching exactly the access they held via Security:Read before this change.

What to do. A custom role does not get this automatic re-seed. Check whether any custom role was granted Security:Read for this purpose, and if so, grant it EnginePrincipal:Read too — via the Settings UI or RbacStore::set_permission() directly. Security:Read alone no longer reaches /api/v1/engine-principals* or its MCP twins after this upgrade.

Direction 3 — API consumers of GET /api/v1/discover/permissions (and the MCP discover_permissions twin) lose the role grid unless they hold UserManagement:Read.

Who this affects. Anything reading the discovery catalogue’s roles[].permissions[] under a principal that holds Infrastructure:Read but not UserManagement:Read.

What changes. The route still returns 200 and the full securable_types/operations taxonomy — that half is unchanged and still needs only Infrastructure:Read. The roles key is replaced by "roles_omitted": true plus a roles_omitted_reason. A consumer that iterates roles must handle its absence; one that checks roles_omitted gets an unambiguous answer.

Why. That grid is authorization topology, and strictly more of it than GET /api/v1/rbac/roles discloses — which this same change floors. Leaving it on the unfloored Infrastructure:Read, which every authenticated session holds on an RBAC-off install, left the floor reachable around. Flooring Infrastructure:Read itself was rejected: it gates ordinary operational reads and would have repeated the too-coarse mistake this change avoided with Security:Read.

What to do. Grant the consuming principal UserManagement:Read if it legitimately needs the grid, or update it to read roles_omitted. Do not treat a missing roles key as “no roles exist” — that is precisely why the omission is declared.

Check your automation, not just your operators — this is the failure that hides. Neither breaking direction produces a startup warning or a migration prompt. Both surface only as a 403 at some later moment, and that moment may be inside a script rather than in front of a person. Audit anything non-interactive that reads the three floored surfaces (/api/v1/access-reviews/export, /api/v1/rbac/roles, /api/v1/engine-principals* and their MCP twins) under a non-admin token:

• A compliance evidence collector is the dangerous case. If it pulls the access-review export on a schedule and treats a non-2xx as “no grants to report” rather than as a hard failure, it will keep succeeding while silently collecting nothing — potentially for a whole audit period. Confirm yours fails loudly on 403, and re-check the first collection after upgrade.
• A monitoring or inventory job on a non-admin token will start logging 403s rather than breaking visibly.

Either enable RBAC and grant the job’s principal the appropriate role (Reviewer for access reviews, UserManagement:Read for /rbac/roles, EnginePrincipal:Read for engine-principal reads), or move the job to an admin credential.

One caveat on revoking a built-in role’s grant. Because the seed loops above run on every boot, a grant you deliberately remove from a built-in role — for example revoking Viewer’s EnginePrincipal:Read — is re-inserted on the next restart, with no audit line distinguishing it from an operator-set grant. This is long-standing behaviour for every seeded securable, not new here, but it is worth knowing before you narrow a built-in role: express the narrowing as a custom role or an explicit deny row instead, both of which survive a restart.

vNEXT — three new securables for directory-sync and enrollment reads; Viewer auto-gains Directory:Read (#4031) (breaking)

Who this affects. RBAC-enabled deployments with a custom role that reads AD/Entra directory-synced users, or that previously reached the directory-sync status / enrollment auto-approve rules / pending-agent list / OIDC config surfaces before this release added dedicated REST v1 routes for them.

What changes automatically. Three new securables — Directory, Enrollment, OidcConfig — are seeded idempotently on every boot (RbacStore::seed_defaults()), same mechanism as EnginePrincipal above. Their built-in-role grants are not symmetric, unlike EnginePrincipal:

• Directory:Read is seeded to both Administrator and Viewer — matching the precedent set for other identity-adjacent PII reads (UserManagement). A custom role that previously relied on not inheriting directory-user PII visibility from a Viewer-equivalent grant set should check whether that matters for its use — this is a genuine widening of what Viewer-derived roles can see, not a like-for-like securable split the way EnginePrincipal:Read was. RBAC-enabled deployments get more than a widening here, though: Directory was never seeded to any role before this release, despite GET /api/directory/users//directory/status//directory/sync already gating on it — under RBAC-enabled enforcement this denied every role, including Administrator, not just non-Viewer custom roles. If your RBAC-enabled deployment could never reach the legacy directory-sync routes even as an admin, this release fixes that dead zone; the “Viewer gains new PII visibility” framing above only tells the RBAC-disabled half of the story.
• Enrollment:Read and OidcConfig:Read are seeded to Administrator only — Viewer deliberately does NOT gain either, since these gate the fleet’s enrollment admission policy and SSO configuration rather than identity/inventory data. Both are also added to the authorization topology floor (see the #2376 note above), so an RBAC-disabled install denies them to a non-admin the same way it now denies AccessReview:Read/UserManagement:Read/EnginePrincipal:Read.

What to do. No action needed for Administrator/Viewer — the seed loop picks these up automatically. If a custom role needs Enrollment:Read/OidcConfig:Read, grant it directly (no built-in non-admin role holds either). If a custom role’s exposure to directory-user PII via an inherited Viewer-shaped permission set is a concern, review it explicitly — the auto-grant is intentional but new.

Also new in this release: the legacy GET /api/directory/users route (pre-existing, not new in #4031) previously issued no audit call at all despite returning PII; it is now audited as directory.users.view, though via a fire-and-forget path that cannot detect a dropped audit row — see audit-log.md’s directory.users.view row for the full three-way posture (REST v1 fail-closed / MCP set-and-proceed-with-signal / legacy silent).

vNEXT — the audit trail is admin-only while RBAC is disabled (#5771) (breaking)

Who this affects. Deployments with RBAC enforcement off (the shipped default; see GET /api/v1/rbac/enforcement) where a non-admin, authenticated session or API token reads the audit trail — a SIEM export, an evidence-collection script or a dashboard user signed in as a plain user. RBAC-enabled deployments are unaffected.

What changes. AuditLog:Read joins the authorization topology floor (see the #2376 note above). With RBAC disabled, every audit read surface now requires the admin session role: GET /api/audit, GET /api/v1/audit, GET /api/v1/audit/auth-sample, GET /api/v1/engine-principals/audit/no-admin, the MCP query_audit_log and audit_engine_no_admin tools, and the yuzu://audit/recent resource. A non-admin gets 403, audited with a detail beginning topology floor: non-admin role denied AuditLog:Read (match on the prefix: an MCP-tier token’s row appends (mcp_tier=<tier>)) and counted in yuzu_auth_topology_floor_denied_total{permission="AuditLog:Read"}. Why: the trail is the SOC 2 CC6.1/CC7.2 evidence chain, and the legacy route also returns each actor’s role and source IP.

What does not change. An admin-role session, and an MCP token minted by one, still read the trail. Under RBAC-enabled enforcement a role holding AuditLog:Read (the seeded Administrator, PlatformEngineer, Operator, ITServiceOwner and Viewer, or any custom role granted it) is admitted exactly as before. Engine principals and service-scoped tokens were already refused with RBAC off.

What to do. If a non-admin identity needs the trail, either enable RBAC and assign it a role that holds AuditLog:Read, or run the automation as an admin-owned token. See rbac.md “The authorization topology floor”.

vNEXT — approval tickets outstanding at the upgrade must be re-requested (#2442) (breaking)

Who this affects. Any deployment holding an MCP approval that was granted but not yet redeemed at the moment of upgrade — that is, one an agentic worker will present back to an MCP tool call. This is independent of the mcp. prefix entry below: it does not matter whether you author any mcp.-prefixed definitions, and a deployment running only Yuzu-supplied content is affected if it has an MCP approval in flight.

Scheduled approvals are not affected. A schedule redeems its own approval by matching the schedule id, not through the MCP recall, so the origin check never sees it and a scheduled run outstanding across the upgrade still fires. If nothing is awaiting an MCP recall when you upgrade, there is nothing to do.

Rolling BACK across this release re-opens the check, and rolling forward does not close it again — but as of this same release, it does not stay open either. The back-fill runs once, as a schema migration. An older server started against an already-migrated database does not re-run it and does not refuse to start; approvals it mints while you are rolled back record no minting surface at all (the column is never set). Rolling forward will not revisit those rows, because the migration has already run — but the MCP mint in this same release always declares its own surface (see the “MCP mint now declares its own surface” entry below), so a row with no recorded surface is no longer the value that grants: it is refused at recall exactly like a declared foreign surface, not exempted. There is no window in which a rollback-minted ticket is redeemable once you are back on a current server. Treat a roll-back as a window during which such tickets simply won’t work later — mint fresh ones after rolling forward rather than relying on anything minted during the roll-back.

What happens. This release records which surface minted each approval, and rows that predate the column carry no surface at all. Rather than assume one they may not have come from, the upgrade labels them with a sentinel that fails closed — so an approval granted before the upgrade is refused at recall. It is reported as approval already used (one-time ticket) even though it was never consumed: that message is deliberately identical for every refusal so the recall cannot be used to probe which surface minted a ticket, and that is why this note exists rather than the error explaining itself.

Recover by calling the tool again without approval_id to mint a fresh ticket and have it re-approved. Nothing is lost but the review. The affected population is only what is outstanding at upgrade time, it does not grow afterwards, and approvals expire after 7 days regardless. On the server side the refusal is distinguishable: the audit row records refused: foreign_origin, not the uniform client message.

vNEXT — the MCP mint now declares its own surface; undeclared tickets stop redeeming (#2442) (breaking)

Who this affects. Any deployment holding an MCP approval that was granted but not yet redeemed at the moment of upgrade — the same population as the entry above, widened. The previous release closed the check for a ticket a REST or scheduled mint declared as its own surface; an MCP-minted ticket was still exempt, because the MCP gate could not yet declare kMcp itself. This release removes that exemption: the MCP mint now stamps every new ticket kMcp explicitly, and an undeclared ticket (the value an omitted origin argument used to produce) is refused at recall exactly like a declared foreign surface.

What happens. Same mechanics as the entry above, same client message (approval already used (one-time ticket)), same audit distinction (refused: foreign_origin). The only difference is which population is now caught: any MCP ticket minted by a pre-upgrade server — every one of them, since none could declare kMcp before this release — that is still outstanding when you upgrade.

Recover the same way: call the tool again without approval_id. The affected population does not grow after the upgrade — every ticket minted by the new server declares its surface at mint time, so nothing outstanding from this point on can hit this case.

vNEXT — MCP approval recall is now bound to its submitter (#2442) (intentional compatibility break, no supported flow affected)

What changes. An approval ticket’s approval_id is a bearer capability — presenting it is what authorizes the recall — and an id can be disclosed to a third party: GET /api/approvals returns the full id to any principal holding Approval:Read, seeded to the Viewer role. Until this release, a Viewer who also held the target tool’s own RBAC permission could take another operator’s approved ticket id from that listing and redeem it themselves. The MCP recall now refuses a ticket whose recorded submitter does not match the recalling principal, the same way it already refuses one minted on a foreign surface — same client message, a distinct audit token (refused: foreign_submitter) for the log.

Who this affects. No supported flow: the one production redemption path in this codebase (the MCP recall) has only ever redeemed a ticket as the same principal that minted it, verified by an exhaustive sweep of every mint and consume call site in the tree, and the non-binding was documented as this issue’s own hazard, never as an intended capability. What this cannot rule out is an external integration that relied on the old behavior to hand an approval_id between two different authenticated principals deliberately — nothing in this codebase does that, but an operator’s own tooling might have. If something in your deployment presented a ticket as a different principal than the one that minted it, that now fails; there is no supported replacement for it today (see #2442 if you need one). Nothing to do at upgrade for the ordinary case: there is no outstanding-ticket population affected by this specific change, because a legitimately-outstanding ticket was always going to be recalled by its own submitter.

vNEXT — the mcp. instruction-definition id prefix is reserved (#2442) (breaking)

Who this affects. Anyone whose instruction definitions include an id beginning mcp.. No shipped or bundled content uses that prefix, so a deployment running only Yuzu-supplied content needs no action for this item — but the outstanding-approval entry above applies regardless of what content you run.

Why. mcp.<tool> names an MCP approval ticket. At the time this shipped, the MCP recall matched a ticket on its definition id and scope expression alone, so a definition authored under that prefix could line up with an MCP tool’s canonical arguments — a ticket raised on one surface being redeemable on another. (Two later releases closed the redemption side of this directly, by binding the recall to the ticket’s recorded origin and separately to its submitter — see the two entries above. This prefix reservation is the narrower, mint-time-adjacent half: it stops a definition from being authored under mcp. at all.) Consuming a confused ticket still required the schema check, the tier gate, per-handler RBAC and a human approval, so this closes a namespace confusion rather than an open escalation.

What changes. Creating a definition whose id starts mcp. is refused with a 400 on every authoring route that accepts an explicit id — POST /api/instructions, POST /api/instructions/yaml, and POST /api/instructions/import — and such a definition is skipped (counted in defs_errored) at boot auto-import. The dashboard’s YAML validator refuses it too, so Validate and Save agree on the create path. Product-pack install is unaffected because it never carries a declared id through at all: the store assigns one, so a pack’s own id, reserved or not, has never been honoured.

What does NOT change. The store applies the rule at creation only, so a definition that already carries such an id keeps executing and can still be saved through PUT /api/instructions/{id} — an update cannot originate an id, so blocking it there would strand content rather than protect anything. One exception, and it is the reason to rename rather than live with it: the dashboard’s YAML editor runs the same validation on every save, create or update, so an existing mcp.-prefixed definition cannot be edited there if its document declares metadata.id (one that omits it still saves, since the validator only checks a declared id).

What to do. Before upgrading, check for affected content. Query the instruction database directly rather than the REST list route: GET /api/v1/definitions caps its result at 100 definitions, and the shipped content alone exceeds that, so an API-based check can report a false all-clear. Use a case-sensitive glob-style match — Postgres LIKE is case-sensitive by default, but ~ (POSIX regex) is used below for an explicit anchor — the reservation is case-sensitive: MCP.foo is a different id everywhere else in the system and is not reserved. (Superseded — instruction_definitions moved to PostgreSQL under ADR-0058; this is no longer a sqlite3 query against instructions.db, see the PostgreSQL Substrate section below.)

psql "$YUZU_POSTGRES_DSN" -c \
  "SELECT id FROM instruction_store.instruction_definitions WHERE id ~ '^mcp\.';"

Any id listed keeps executing after the upgrade and can still be edited through PUT /api/instructions/{id}, but re-importing an export of it will fail, and so will saving it from the dashboard’s YAML editor (that path validates the id on every save, not only on create). Rename those definitions before upgrading — create a replacement under a different id and delete the original.

vNEXT: the guardian. instruction-definition id prefix is reserved (#6062) (breaking)

Who this affects. Anyone whose instruction definitions include an id beginning guardian.. No shipped or bundled content uses that prefix, so a deployment running only Yuzu-supplied content needs no action for this item. Operator-authored definitions are the population this cannot rule out: the prefix reads like a natural name for a Guardian-related definition, and nothing before this release refused it.

Why. guardian.<...> is the namespace reserved for Guardian rule-deploy approval tickets (four-eyes epic #6060, ADR-1009). Keeping authored definitions out of it keeps the two namespaces separable in the store, on the Approvals page and in the audit log, in the same way mcp. is kept apart for MCP tickets (previous entry). No Guardian deploy approval exists yet in this release; the reservation lands first so that no definition has to be renamed later, when a ticket that uses the prefix does exist.

What changes. Creating a definition whose id starts guardian. is refused with a 400 on every authoring route that accepts an explicit id (POST /api/instructions, POST /api/instructions/yaml, POST /api/instructions/import) and such a definition is skipped (counted in defs_errored) at boot auto-import. The dashboard’s YAML validator refuses it as well, so Validate and Save agree. The error text is definition id may not use the reserved 'guardian.' prefix (reserved for Guardian deploy approvals); the mcp. text is unchanged, so a SIEM or script rule keyed on it keeps matching exactly what it matched before. The rule is case-sensitive and exact-prefix: Guardian.foo, my.guardian.foo and guardianx.foo are not reserved.

What does NOT change. As for mcp., the store applies the rule at creation only. A definition that already carries a guardian. id keeps executing and can still be saved through PUT /api/instructions/{id}. The one exception is the dashboard’s YAML editor, which runs the same validation on every save, create or update: an existing guardian.-prefixed definition cannot be edited there if its document declares metadata.id (a document that omits it still saves). Re-importing an export of such a definition also fails.

What to do. Before upgrading, look for affected ids. Query the instruction database directly rather than the REST list route: GET /api/v1/definitions caps its result at 100 definitions and the shipped content alone exceeds that, so an API-based check can report a false all-clear. The query is read-only and case-sensitive, matching the reservation:

psql "$YUZU_POSTGRES_DSN" -c \
  "SELECT id FROM instruction_store.instruction_definitions WHERE id ~ '^guardian\.';"

An empty result means nothing to do. For any id listed, create a replacement under a different id (for example site.guardian-baseline-check) and delete the original; update any schedule, workflow or script that refers to the old id first. Definitions you leave in place keep executing, but you lose the dashboard YAML editor and re-import for them.

vNEXT — behind a reverse proxy, declare your external origin or CSRF-gated dashboard actions keep failing (#2537)

Who this affects. Anyone running the dashboard behind nginx, Envoy, HAProxy, an ALB or Cloudflare where the proxy rewrites the Host header rather than passing the client’s value through. Direct access, and proxies configured to preserve Host, are unaffected and need no change. For nginx, the proxy_set_header Host $http_host; snippet in Security Hardening forwards the client’s Host with its port.

The symptom. Every dashboard write fails with 403 cross-site request refused: cookie-session writes require a same-site Origin or Referer, and that is decided by one server-wide check, applied after authentication and before step-up evaluation, challenge emission and route-handler effects. The dashboard renders fine, sign-in works and reads work; every write fails, including creating an API token and the step-up elevation request. Most dashboard pages do not display the refusal (the page does not swap an error response into itself), so the visible symptom is usually a button that does nothing. The response body, an error envelope with a remediation that names the flag, is visible in the browser’s network tab. The refusal is also recorded as a csrf.denied audit row, counted in yuzu_server_csrf_denied_total{path_class,reason}, and logged as a warning that names the flag (at most one log per 60 seconds per reason; the counter records all of them). The per-route message texts that earlier releases showed for individual dashboard actions are now produced only for a token caller of a route that keeps a non-token-exempt copy of the check.

Cookie-session requests that dispatch work, persist state or force a refresh of server state (the guarded requests) are held to the same rule, but a Sec-Fetch-Site header that is present decides alone: exactly one header with value same-origin or none is admitted and any other value is refused with origin_mismatch, without consulting Origin or Referer. So behind a TLS-terminating proxy, where the browser talks to the proxy over HTTPS, a proxy that rewrites Host does not refuse a browser’s own guarded requests. Only a client that sends no Sec-Fetch-Site (a script, Safari before 16.4, or a browser talking plain HTTP to a non-loopback address, which sends no fetch metadata) falls back to the Origin or Referer comparison that --csrf-trusted-origin feeds. A same-origin GET carries no Origin, so for those clients the guarded requests depend on the Referer the browser sends; the server’s default Referrer-Policy supplies it, and a proxy or page policy of no-referrer would break them. A repeated Sec-Fetch-Site header is refused and is recorded with reason origin_mismatch, even when a good Origin is sent; to cure a refusal caused by a Sec-Fetch-Site header, send it once with value same-origin or none, or remove the header so that a same-site Origin or Referer decides.

Alerts. The shipped yuzu-web-writes group in docs/prometheus/yuzu-alerts.yml watches the counter: YuzuCookieWritesRefusedSameSite (warning) fires when at least 3 origin_mismatch or referer_mismatch refusals occur within 15 minutes, which is the signature of a proxy that rewrites Host, and YuzuSameSiteGateInternalError (warning) fires on any gate_error in 15 minutes. no_origin_no_referer is deliberately not alerted, because a script without the header is expected noise. The 15 minute window and the threshold of 3 are tentative and have not been tuned against fleet data.

What the lockout does and does not do.

• Restart required: the origin list is read at start, there is no reload. After changing the flag or YUZU_CSRF_TRUSTED_ORIGIN, restart every server replica; replicas with different values answer the same request 200 on one and 403 on another.
• /livez and /readyz stay green, so a load-balancer or orchestrator health check never detects it.
• Ordinary reads are not blocked by the check (the guarded requests are held to it, as above), but a refusal is not free: each one writes an audit row, and behind a proxy every operator shares the proxy’s single per-IP rate-limit bucket. With a slow or degraded PostgreSQL a burst of refusals can therefore slow other requests down.
• Bypass path: if you cannot create an API token through the proxy, reach the server directly (for example an SSH tunnel to the web port, so Host matches the browser’s Origin), sign in and create one, or restart with the flag. A script that connects directly must send an Origin naming the address it connects to (for example http://localhost:8080; https://localhost:8443 when the server terminates TLS itself, where an http:// origin is refused). Direct access needs a local sign-in: a deployment that offers only OIDC or SAML sign-in redirects to the public URL, so for those only restarting with the flag works.
• The check cannot be disabled: the remedies are the flag, an API token or rolling back.
• An Origin: null request cannot be fixed by configuration: the reserved token null is refused as an entry, and Referer is not consulted once Origin is present.

Why. The same-site check compares the browser’s Origin against the server’s Host. When the proxy substitutes its own Host, those two legitimately differ: the browser says https://yuzu.customer.example while the server sees yuzu-server:8080. The check was correct and the deployment was correct; there was simply no way to tell the server its external name.

The fix — declare the external origin:

yuzu-server --csrf-trusted-origin https://yuzu.customer.example
# or
YUZU_CSRF_TRUSTED_ORIGIN=https://yuzu.customer.example

Repeatable, and a single value may be comma-separated:

--csrf-trusted-origin https://yuzu.example --csrf-trusted-origin https://yuzu-dr.example
--csrf-trusted-origin "https://yuzu.example,https://yuzu-dr.example"

Two accepted forms, and the difference matters:

Prefer the scheme-qualified form. A scheme-qualified https:// entry also pins the scheme for that host: the server behind a TLS-terminating proxy listens in plain HTTP and cannot see the public scheme, so an https://yuzu.example entry is the operator’s declaration that https is the public scheme, and an http://yuzu.example Origin is then refused even when Host is forwarded and matches. Declare it even when the proxy forwards Host unchanged. A server that terminates TLS itself (HTTPS enabled) needs no declaration for its own address: it requires the Origin or Referer to be https:// and treats only :443 as the default port, so https://yuzu.example:80 is a different origin from Host: yuzu.example:443. Remaining limit: on a plain-HTTP listener with no https:// entry declared for the host, an http:// Origin naming the same host and port as Host is still treated as same-site.

Entries are case-insensitive, may carry a port, and any path is ignored. Wildcards are not supported — *.example is accepted by the parser but will never match anything, deliberately. The reserved token null is refused as an entry: it is the serialisation of an opaque origin (sandboxed iframes, redirected cross-origin POSTs, file:// documents), not a host, so trusting it would admit all of them at once.

Port handling follows RFC 6454 — a port is dropped only when it is the default for that entry’s scheme:

The last row is the one to read twice. An earlier version collapsed a bare yuzu.example:443 to yuzu.example, which then trusted both http://yuzu.example and https://yuzu.example — one declared origin silently trusting a second, which is the whole defect this canonicalisation exists to prevent. Guessing https would be a guess, and keeping the port would make the entry unmatchable (the request side canonicalises https://yuzu.example:443 to yuzu.example). So it is refused, and you write https://yuzu.example or http://yuzu.example instead.

A bare entry with no port is still deliberately loose: yuzu.example covers https://yuzu.example, https://yuzu.example:443, http://yuzu.example and http://yuzu.example:80 — but not https://yuzu.example:80, which is a different origin.

Entries are also rejected if they are empty, host-less (:443), or carry userinfo (u@host) — the last because the request side always fails closed on @, so accepting it would put an entry in the boot log that could never match.

Rejections are reported. If any supplied value is refused, the server warns at boot with the accepted-versus-supplied counts. Before this, an all-invalid config produced boot output identical to not passing the flag at all, and the first symptom was the same opaque 403 the flag exists to remove.

Confirming it took. The server prints the accepted set once at start-up, to standard output (check the container or service log). It is emitted before the log level and --log-file are configured, so --log-level does not hide it and it is not written to --log-file:

CSRF same-site gate: accepting 1 operator-declared external origin(s) in addition to the request Host — https://yuzu.customer.example

If that line is absent, or lists something other than what your users type into the address bar, the 403s will continue. It is the fastest diagnosis for a mistyped entry.

When no valid entry is configured the server instead prints one line stating the rule and that no --csrf-trusted-origin value was accepted, with the same advice for a Host-rewriting proxy.

What has NOT changed. A request whose Origin already matches Host behaves exactly as before, so an unproxied deployment sees no difference for browsers. Leaving the flag unset means same-host only, which is correct for any deployment that does not rewrite Host. A request authenticated only by an API, MCP or engine token is not subject to this check.

What did change. A cookie-session write that carries neither Origin nor Referer is refused 403, including one from curl or other automation; use an API token or send the header. The guarded requests (cookie-session requests that dispatch work, persist state or force a refresh of server state) are refused 403 unless they carry one Sec-Fetch-Site header with value same-origin or none, or, when they carry no Sec-Fetch-Site header, a same-site Origin or Referer.

Deliberately not implemented: X-Forwarded-Host is never consulted. Reading it would let anyone able to reach the server’s port declare their own external hostname and defeat the CSRF check outright. Gating it on a trusted-proxy CIDR does not rescue that — on the container networks the reference composes use, “inside the CIDR” is usually every sibling container, and the failure mode is silent: the dashboard keeps working while the control is dead. A config value cannot be set by an attacker, which is why the trust anchor here is a flag and not a header.

For the same reason, never configure the proxy to inject or rewrite Origin or Referer: that turns the control off for that deployment. Declare the public origin with --csrf-trusted-origin instead.

vNEXT — a device target that was supplied but names nothing is now refused, not widened to the fleet (#2500)

What changed. Three REST surfaces treated a targeting argument the caller supplied that resolved to no devices as identical to one they never sent — and “no target named” meant broadcast. POST /api/command with {"plugin":"service","action":"restart","agent_ids":[1,2,3]} restarted the service on every connected agent under plain Execution:Execute, with no approval step, and returned a success response. So did {"agent_ids": []}, which is what any device filter that matched nothing produces. Each of these is now 400.

What breaks. Requests that previously succeeded and now fail:

The two remediation rows matter for the same reason as the rest: on that route an absent agent_ids means “every non-compliant agent in this policy”, so a supplied selector that named nothing — or a scope, which that route cannot act on — quietly became a fleet-wide mutating remediation. Omitting agent_ids entirely still targets every non-compliant agent, unchanged.

The instructions-execute row is the one to read twice: "scope": "" + empty agent_ids = broadcast was documented behaviour in rest-api.md, so a client written against the published contract is affected. The from-* row covers from-echo-query, from-instruction-result, re-eval and from-inventory-query.

One thing got LOOSER, not tighter. POST /api/command previously rejected "scope": "__all__" with 400 invalid scope — the scope parser has no rule for it — while the sibling instruction-execute route broadcast on the same string. It now broadcasts on both. This grants no new access: broadcast was already reachable on that route by omitting targeting entirely, under the same Execution:Execute, so __all__ only gives a name to something a caller could already do. But if you have a script that sends __all__ and treats the 400 as a no-op, it will now dispatch to the fleet.

What still works, unchanged. Omitting both agent_ids and scope broadcasts to all connected agents, on every route. So does an explicit "scope": "__all__" — the ground scope kind already advertised by /discover/scope-kinds and by the MCP execute_instruction schema. If you have a client sending "scope": "" to mean “everything”, change it to omit the field or to send "__all__"; both are supported and neither is deprecated.

Stored data — narrower than it first looks. Only requests that explicitly send "scope": "__all__" record scope_expression = "__all__" on the execution row, and those already did so before this change. Broadcasting by omitting both fields is unchanged and still records "" — the execution row is written from the raw request value, before the omitted-means-__all__ mapping is applied for dispatch, and the mapped value is never written back. The one practical change is that the dashboard’s “All agents” button now sends __all__ explicitly, so rows created that way look different from before. A saved query selecting historical broadcasts by scope_expression = '' still matches everything except dashboard-initiated ones.

Dashboard users driving the UI in a browser need do nothing — the Instructions execute dialog’s “All agents” option sends __all__ instead of an empty string.

But automation that POSTs the dashboard forms directly does need attention. As of the Wave-1 foundations change, /api/dashboard/execute and /api/dashboard/echo-execute distinguish an OMITTED scope from one SUPPLIED as empty (scope=): omitted still means the whole fleet, but supplied-but-empty is now refused and dispatches to nobody, where it previously broadcast. If a script builds the form body unconditionally and leaves scope blank when no device is selected, it will stop dispatching — which is the intended outcome, but a silent one. Send scope=__all__ to keep the fleet-wide behaviour deliberately.

Detecting affected clients — and the limit of what is possible. There is no reliable way to find them before upgrading. The audit trail records the OUTCOME of a dispatch, not the request shape that produced it: command.dispatch|success stores plugin:action -> N agent(s) and instruction.execute|success stores agents=<sent>, and neither preserves the agent_ids or scope the caller actually sent. So a historical broadcast that was deliberate and one that was an accidentally-widened three-device request are indistinguishable in existing rows. The closest available pre-upgrade signal is reviewing automation you believe targets a subset for dispatches whose agent count is suspiciously close to your full fleet size. (The stored detail uses a literal \u2192 arrow, not ->, so match on the agent count rather than the separator.)

After upgrading, refusals are counted by yuzu_server_dispatch_target_rejected_total{route,reason} (all series pre-seeded at boot, so absent() stays meaningful) and audited as command.dispatch|denied (detail=reason=<reason> <plugin>:<action>), instruction.execute|denied (detail=reason=<reason>) or result_set.create|denied (detail=reason=<reason> source_kind=<kind>). The same action’s failure result (#4496 + follow-up, the POST /api/v1/result-sets/from-inventory-query producer and its MCP twin) carries detail=reason=store_degraded|query_truncated|poison_excluded|parse_error_excluded source_kind=inventory_query - only the latter three of those four are counted on yuzu_server_dispatch_target_rejected_total (route="result_set_inventory_query"); store_degraded is a store-availability failure, not a targeting-shape refusal, so it is not on this series. The YuzuDispatchTargetRejected alert fires when the 15-minute increase exceeds 3 — deliberately not on every single refusal, because a rule that pages on one malformed request gets silenced. Use the audit rows, not the alert, to find individual offenders.

query_truncated’s failure mode is structural, not a per-record near-miss - plan its runbook step separately from poison_excluded/parse_error_excluded. poison_excluded and parse_error_excluded are both per-record and self-heal once the offending record is fixed or excluded - they are DIFFERENT causes (over-nested data_json vs. data_json that fails to parse as JSON at all), each with its own reason so an operator can tell which guard excluded a record, but the same “per-record, self-healing” runbook shape applies to both. query_truncated fires whenever the generic-inventory read backing both producer routes exceeds the hard-coded 5,000-row cap or the 8 MiB aggregate payload cap - there is no pagination on this path today. On a fleet whose inventory has grown past either cap, EVERY subsequent call to POST /api/v1/result-sets/from-inventory-query or its MCP twin refuses with query_truncated, and the alert never clears on its own. If you need to silence YuzuDispatchTargetRejected on such a fleet before the fix lands, scope the Alertmanager silence to reason="query_truncated" AND route="result_set_inventory_query" specifically - never the bare alertname, which would also hide poison_excluded/parse_error_excluded, genuine near-miss signals that must stay visible. Tracked fix: #2633 (InventoryStore::query row cap (5000): keyset pagination + limit+1 truncation probe).

vNEXT - result-set instruction_id/params fields are now bound-checked (#4373) (intentional compatibility break, no supported flow affected)

What changed. POST /api/v1/result-sets/from-instruction-result and POST /api/v1/result-sets/{id}/re-eval had no bound on the instruction_id or params fields feeding an InstructionDefinition dispatch: an over-keyed/oversized params object could reach fleet-wide dispatch, and an oversized instruction_id could reach an unbounded instruction_store lookup (a real registered instruction id is capped at 128 characters, so an oversized id could never match one and dispatch - it reached the not-found fallback instead). Both routes now enforce the same caps MCP’s create_result_set_from_instruction_result/reevaluate_result_set tools enforce: instruction_id at 256 bytes, params at 32 keys / 256-byte keys / 64 KiB values.

What breaks. Requests that previously succeeded and now fail:

Who this affects. Callers sending a field past a numeric/count bound (instruction_id, params count/key-length/value-length, both at the same values MCP’s equivalent tools enforce - MCP’s handler-side bounds landed within days of this fix, in the same unreleased cycle, not a long-standing MCP/REST gap), AND separately callers sending a wrong-typed params, name, or source_kind (not a numeric bound at all - a shape/type mismatch, always rejected regardless of size). No supported flow constructs any of these fields anywhere near the numeric limits or with the wrong JSON type, so no compliant client is affected either way.

vNEXT — POST /mcp/v1/ can now hold its response open as an SSE stream (2f PR 3b)

A tools/call for execute_instruction that carries _meta.progressToken and an SSE-capable Accept now receives its progress on the POST response itself rather than on the session’s GET stream. Opt-in and per request: a call that sends neither, or only one of the two, behaves exactly as before and is byte-identical on the wire. Plain POST clients (mcp-remote, Claude Desktop) are unaffected.

Three operator-visible consequences:

• Sizing. A streamed POST holds an HTTP worker for up to its response cap (120 s), enforced on a busy execution too (#2739): after the cap expires the bridge delivers one final drain of already-latched progress and then settles, so the bound is the cap plus at most two ~3 s pump ticks plus one bounded progress drain (a single latest-wins snapshot since #2412) and its socket-write time (the server’s 30 s write timeout, set_write_timeout in server.cpp) — worst case ~156 s, not the execution’s duration. It leases from the same held-open budget as the GET channel, so total concurrency is unchanged — but TimeoutStopSec and any container termination grace period should be sized comfortably above that ~156 s bound (the shipped systemd unit and every shipped compose file use 210 s); 30 s — which suited GET alone — is the figure to move away from. Under-sizing SIGKILLs mid-drain and silently drops in-flight streams on deploy. Update (#3042): the ~156 s figure above bounds a single streamed-POST call during ordinary (non-shutdown) operation — it is not how long ServerImpl::stop() itself waits on one. Since #3042, graceful shutdown close-signals every live MCP session up front, so a streamed POST held open across an ordinary stop() ends within about one pump tick (~3 s), not the 120 s cap; the underlying execution is unaffected and stays fetchable by execution_id. What still bounds shutdown is a stream stuck mid-write to a blackholed or drip-feeding peer (the 30 s write timeout) — see docs/mcp-server.md’s Shutdown section for the current mechanism. The 210 s TimeoutStopSec recommendation above remains a safe, comfortably conservative choice; it is no longer the tight bound its original derivation implied.
• Per-principal ceiling. --mcp-max-streams-per-principal governs the GET channel. The streamed-POST allowance is a fixed 4 concurrent calls per principal — numerically the same as, but counted and enforced separately from, any single session’s own replay-ring pin-slot count — so the steady-state per-principal held-open sum across both channels is --mcp-max-streams-per-principal + 4. That sum is not a hard ceiling: a GET-channel reconnect keeps the superseded connection’s count until it finishes draining, and McpStreamState bounds this to one pending handover per session, not per principal — a principal with every one of its GET sessions mid-handover at once can transiently double the GET component. Lowering the flag to contain a noisy principal does not reduce its streamed-POST concurrency.
• Reverse proxies. The response sets X-Accel-Buffering: no, which only nginx honours. Envoy, HAProxy, ALB and Cloudflare need their own no-buffering opt-out, or the stream is buffered and dead clients are not detected.

--mcp-no-streaming remains the kill switch and now also degrades the streamed POST arm back to plain JSON, not just GET/DELETE.

vNEXT — streamed POST (--mcp-enable-streamed-post) now ships ON by default (breaking)

The feature described in the vNEXT entry above already existed; this entry is about its default changing. Every prerequisite that gated the on-by-default flip (#2739, #2740, #2785, #2789) is fixed, so a server started with no MCP flags now serves streamed POST for any execute_instruction call that sends _meta.progressToken with an SSE-capable Accept — no configuration required.

What this means for an upgrading operator, restated from the Sizing and per-principal bullets above because they are no longer conditional on opting in — they now apply to every deployment by default:

• TimeoutStopSec (or your container runtime’s equivalent termination grace) must be sized comfortably above the ~156 s worst-case bound derived in the Sizing bullet above (120 s cap + pump ticks + the 30 s write timeout), not the ~30 s that sufficed for the GET channel alone. The shipped systemd unit and every shipped compose file that runs yuzu-server use 210 s — use that as your own starting point if you’re not deploying from one of those. This was previously only a concern for operators who had explicitly enabled the flag; it is now the default posture for every deployment that takes no action.
• **A principal’s steady-state held-open sum is `—mcp-max-streams-per-principal
  • 4`** — the streamed-POST allowance is no longer contingent on opting in, it applies to every deployment by default. That sum is not a hard ceiling (see the Per-principal ceiling bullet above for the GET-reconnect transient).
• If you’re behind a non-nginx reverse proxy (Envoy, HAProxy, ALB, Cloudflare) or running an MCP client whose HTTP transport can’t consume a streamed response body, verify SSE-on-POST actually reaches your client before upgrading (see the Reverse proxies bullet above) — otherwise a call that previously answered promptly can appear to hang for up to the ~156 s worst-case bound (see the Sizing bullet above), not just the 120 s cap. Pass --no-mcp-streamed-post if you’d rather verify after upgrading.

To opt out, pass --no-mcp-streamed-post (or set YUZU_MCP_ENABLE_STREAMED_POST=false) to keep the pre-flip plain-POST-only behavior. --mcp-no-streaming remains the broader kill switch and disables streamed POST along with the rest of the Streamable HTTP transport.

vNEXT — MCP stream revalidation rides the tick, and the pin-drift alert moves to a new counter

Two operator-visible changes to held-open MCP GET SSE streams. Neither changes a revocation bound: the documented one-tick single-server figure still holds and is now pinned by tests.

Credential re-checks are once per tick, and busy streams stop sending heartbeat filler. A stream delivering frames continuously previously re-validated its credential and slid its session TTL on every frame; both now run once per ~3 s tick, and a pass that delivered real frames skips the redundant heartbeat. If anything on your side counts event: heartbeat frames as a liveness signal, key on any delivered frame instead.

The admission-drift reading moved counters. yuzu_mcp_stream_final_unpinned_total used to carry the “replay-ring pin accounting has drifted” reading; that reading now belongs to the new yuzu_mcp_stream_pin_displaced_total (the server now displaces the oldest pinned terminal instead of committing the newest unprotected, which is strictly less bad). If you alert on final_unpinned_total > 0, keep that rule and add the same rule for pin_displaced_total — reference rules for both ship in docs/prometheus/yuzu-alerts.yml, and the response procedure is docs/ops-runbooks/mcp-stream-pin-displacement.md (diagnostic only — no restart).

vNEXT — engine-principal streams: liveness re-checks are cached, and the outage grace window is measured differently (#2367)

Two operator-visible changes to held-open MCP/SSE streams authenticated by an engine principal. Nothing changes for human or API-token streams.

Liveness re-checks are cached for 15 seconds. Every held-open stream re-validates its credential on each ~3 s heartbeat tick. For engine principals that check previously read PostgreSQL every tick, which under a connection-pool brownout was self-amplifying and could starve ordinary request traffic — not just streaming. It is now served from a short cache, and a store that cannot be reached is rate-limited rather than re-asked on every tick.

Fresh authorization is unaffected: creating a session and on-behalf-of target checks still read through on every call, so revoking an engine principal stops new sessions immediately. On the server handling the revoke, live streams are still cut on the next tick — the cache is invalidated as part of that write. Across replicas, a revoke reaches another replica’s stream within the 15 s TTL plus a tick. If your compliance posture quotes a revocation-latency figure for engine principals, that is the number.

A stream may now end sooner during a PostgreSQL outage. The 60 s indeterminate grace window is now measured from the stream’s last authoritative credential confirmation rather than from the moment the outage was noticed. Previously a stream could ride cached answers and then collect a full fresh 60 s window on top of them; total survival past a real confirmation is now bounded by the window itself. In practice a stream can close with auth_unavailable up to ~15 s earlier than before. Clients reconnect and resume via Last-Event-ID as they already do, and durable results remain fetchable by execution_id.

New observability. Four metrics (yuzu_server_engine_revalidate_cache_hits_total, ..._misses_total, ..._cache_size, ..._backoff_suppressed_total — see metrics.md). The last is the brownout signal: it moves only while the store is unreachable. The server also emits a startup warning when effective SSE stream capacity exceeds 16x --postgres-pool-size; if you see it, watch yuzu_pg_acquire_wait_seconds and yuzu_pg_pool_in_use before raising either.

No configuration changes are required and no flags were added.

vNEXT — MCP supervised-tier calls now enforce the published input schema pre-approval (#2405)

Approval-gated MCP tools/call arguments are now validated against the tool’s published inputSchema BEFORE an approval ticket is minted or consumed. Most callers only fail earlier (junk args that would have failed after admin approval now answer -32602 immediately, with no ticket created or burned).

Previously-succeeding shapes that now break — supervised-tier execute_instruction / execute_bundle, plus any approval-gated tool taking an integer parameter:

• execute_instruction / execute_bundle with a params object containing a non-string value (number/bool/array/object) — previously silently stringified by the handler after approval. Stringify client-side.
• execute_bundle with steps: [] or more than 32 steps (the schema’s minItems/maxItems, matching the handler’s own 1–32 bound).
• execute_instruction with more than 10 000 agent_ids, or any single agent_ids element longer than 128 bytes.
• execute_instruction with plugin/action longer than 128 bytes, or scope / a params value longer than 8192 bytes. (execute_bundle’s per-step plugin/action/params carry no length bound and so are unaffected by these bounds — the non-string rule above still applies to them.)
• Any approval-gated tool given an integral float where the schema declares integer — e.g. mint_engine_credential {"ttl_days": 90.0} or rotate_engine_credential {"overlap_days": 7.0}. These previously “succeeded” in the worst way: the server ignored the value and silently applied the parameter’s default. Send a JSON integer (90, not 90.0) — note that Python’s json.dumps emits 90.0 for a float-typed value.

The bound-based rejections above were already the published schema contract (added in track 2f) but were client-advisory until this change; they are now enforced on the approval-gated path. Operator and readonly callers of these two tools are unchanged — schema validation runs where requires_approval is true, which for Execution:Execute is the supervised tier only, so those bounds stay advisory on the other tiers (full every-path enforcement is tracked in #2437). Note the gate keys on the operation, not the tier: delete_tag is approval-gated on operator as well, so its arguments are validated there too — it simply carries no length or count bound today, so nothing breaks. Any approval ticket already minted-and-approved for one of the shapes above becomes unrecallable fail-closed (it is never consumed) and expires on the normal 7-day approval TTL; re-submit the corrected call for a fresh ticket.

Also note: a recall whose arguments fail schema validation now answers -32602 before the ticket is even looked up — clients that pattern-matched -32003 for every recall failure should treat -32602 as “fix the arguments, ticket untouched”. Full detail: docs/mcp-server.md “Pre-approval input-schema validation”.

vNEXT — MCP tool annotations are now truthful; write tools carry destructiveHint:true for the first time

Track 2g PR 2. Every MCP tool now advertises the four standard annotation hints (readOnlyHint/destructiveHint/idempotentHint/openWorldHint), generated from a single-source classification and enforced truthful by a CI cross-check test. Operator-visible effect: an agentic worker that renders a confirmation prompt off destructiveHint will, for the first time, prompt on the write tools that previously carried no annotation (execute_instruction, execute_bundle, set_tag, delete_tag, quarantine_device, revoke_certificate, and more), and three previously false-safe hints (confirm_engine_rotation, close_access_review) are corrected. The hints are advisory UX only — the tier + maker-checker approval gate is unchanged and remains the enforcement. Because MCP advertises tools.listChanged:false, long-lived clients should reconnect after this deploy to pick up the corrected hints. Not a breaking change (no previously-working call is rejected). Full detail: docs/user-manual/mcp.md “Available Tools”. (Since superseded for confirm_engine_rotation: #2384 added a required token_id argument that pins the confirm to the exact pending rotation, so its idempotentHint is corrected back to true — and the new required argument IS a breaking change for that one unreleased tool/route.)

vNEXT — confirm_engine_rotation replay-after-resolution is now a terminal error (#2404)

Direct sequel to the #2384 entry above. A confirm (REST POST /api/v1/engine-principals/{id}/credentials/confirm or MCP confirm_engine_rotation) replayed after the rotation already resolved — a network-dropped 200, a double-submit, or a client racing the auto-revoke sweep — now returns a terminal 409 / MCP kInvalidParams (rotation already confirmed / no rotation in flight ... already the sole active credential), instead of the previous retryable 503. A confirm that finds more than two active credentials likewise now returns a terminal 400 (was 503). Only a genuinely-empty read and an unrecognized two-credential pair stay 503.

Not a breaking change by this file’s convention — no previously-succeeding call is rejected; the first, real confirm still returns 200 identically. But worth a glance for any integration that pattern-matches 503 => retry: such a client will now loop forever against a permanently-503 call under the old behavior, which is exactly the livelock this fixes — after the upgrade it gets a terminal 409/400 and should stop and (if it genuinely needs a new credential) call rotate, not replay confirm. Agentic MCP clients honouring idempotentHint:true get the correct terminal answer automatically. Full detail: docs/user-manual/rest-api.md (confirm error table) and docs/user-manual/engine-principals.md.

vNEXT — API-token rotation confirm-identity binding now survives a restart; first schema change to api_tokens (#2961)

Migration v3 adds api_tokens.rotation_initiator (durable twin of the confirm-identity binding described in docs/auth-architecture.md “Confirm-identity binding survives a server restart”). This is the FIRST schema change ever applied to api_tokens since it shipped — every authenticated request validates a Bearer token against this table, so it is one of the hottest tables on the server.

Not breaking, but worth knowing before a rolling upgrade. ALTER TABLE ... ADD COLUMN ... DEFAULT '' takes Postgres’s briefest possible form (a metadata-only “fast default” on PG 11+, not a table rewrite) but is still an ACCESS EXCLUSIVE DDL statement: it must wait for every transaction currently touching api_tokens to finish before it can run, and every new transaction on the table queues behind it in turn while it waits. During a rolling upgrade, an outgoing replica’s still-open request handling can hold such a transaction open. This is bounded and loud, not silent: the pool’s lock_timeout (10s, pg_pool.hpp) cancels the migration’s ALTER if it queues that long, and ApiTokenStore construction fails closed exactly as it does for any other migration failure — the new binary refuses to start with a clear log line rather than serving degraded or hanging the table. If you see this on a rolling upgrade, retry once traffic quiesces (a load balancer draining the outgoing replica is usually enough); it is not a data integrity concern either way.

vNEXT — a device quarantined while offline now re-contains itself automatically on reconnect (#3425) (breaking)

QuarantineStore gains schema v2: two columns on quarantine_records (last_applied_at, last_confirmed_at, both BIGINT NOT NULL DEFAULT 0) tracking whether a new background component, QuarantineContainmentReconciler, has re-applied and confirmed a device’s endpoint firewall since it last reconnected. Same ACCESS EXCLUSIVE migration-lock note as the API-token entry above applies here too — negligible in practice, since quarantine_records is a small, manually-curated security-event table, not a hot path.

No operator action needed for the reconciler itself. Previously, a device quarantined while offline stayed contained at the control plane (the #881 dispatch gate) indefinitely, but its own firewall was never (re-)applied until someone noticed and manually re-issued the quarantine_device MCP call. On upgrade, every pre-existing active quarantine record starts unconfirmed, so expect one automatic re-application attempt per connected contained device shortly after the new binary starts serving — this is idempotent and is the correct, intended behaviour (those are exactly the devices whose endpoint containment was never independently confirmed). See docs/user-manual/security-hardening.md “Reconnect re-application (#3425)” for the mechanism and the new yuzu_server_quarantine_endpoint_unconfirmed{reachability} / yuzu_server_quarantine_reapply_total{result} / yuzu_server_quarantine_reconciler_tick_healthy metrics.

BREAKING for callers of POST /api/v1/quarantine. This route now validates whitelist at write time (≤512 chars, [0-9A-Fa-f.:] tokens only) and rejects a malformed value with 400 instead of persisting it — a caller relying on the old permissiveness for a CIDR range or hostname entry now gets 400 where it previously got 201. This route only ever creates a NEW record (it already refuses with 400 if the device is already quarantined), so no existing containment is ever lost by this change — but a caller that fires a quarantine request and ignores a 400 response now gets zero protection for that device, where before a malformed-but-persisted record still left it denied at the #881 control-plane dispatch gate (its endpoint firewall was never actually enforceable either way, since the same malformed value could never be dispatched). Check the response status; do not assume success.

QuarantineContainmentReconciler is always on, with no configuration surface — no CLI flag or env var disables or tunes it (matching the #881 dispatch gate it complements, which is the same way). Its cadence (20s tick), per-agent timing (60s minimum reapply interval, 15-minute backoff cap), and per-tick dispatch cap (50 agents) are fixed constexpr constants in quarantine_containment_reconciler.hpp, not runtime-configurable.

vNEXT — the rotation sweep now carries the full clock-guarded-retention shape (#2964)

What changed. ApiTokenStore::sweep_expired_rotations — the 60-second background sweep that auto-revokes rotation predecessors for both engine-credential and human API-token overlap pairs — previously issued its delete on a bare local wall-clock comparison, capped per tick but with no persisted clock anchor and no anomaly detection. It now reads a Postgres-authoritative clock, persists a durable anchor in a new rotation_retention_meta table (schema v4), and classifies every tick through the same guard audit_store’s retention pass uses — a clock anomaly or a not-yet-trustworthy anchor declines the tick outright rather than revoke on an unverified reading, leaving both credentials in every affected pair active for at least one more tick. Detail: docs/user-manual/authentication.md “Rotating a Token”, docs/user-manual/engine-principals.md “Rotate the credential (overlap-pair model)”, and docs/user-manual/metrics.md “Rotation-sweep clock guard metrics”.

HA / multi-replica. Unlike audit_store’s retention pass, this sweep is single-writer by construction: before it will classify anything, it takes a store-wide PostgreSQL session advisory lock in its own key namespace. This is what makes the per-tick auto-revoke cap (≤200) a cluster-wide cap rather than a per-replica one that would otherwise let an N-replica deployment auto-revoke up to N x 200 predecessors in one tick. A replica that does not win the lock on a given tick records a SkippedLock outcome and does nothing that tick — expected, routine behaviour on a genuinely multi-replica deployment, where roughly (N-1)/N of ticks land there for any given replica. On a single-replica deployment — the default — this is not “routine leader-election contention” at all: one dedicated sweep thread on a sequential 60-second loop cannot lose an election against itself, so SkippedLock there can only mean a second writer holding the same session lock — a zombie process left over from a crashed instance, a botched blue-green cutover overlapping two live instances briefly, or an unauthorised second server pointed at the same DSN. If you run single-replica, treat any sustained non-zero yuzu_rotation_sweep_lock_skipped_total as a fault to investigate, not background noise; see docs/ops-runbooks/rotation-sweep-clock-guard.md.

vNEXT — API-token rotation confirm now requires proof of possession (#3015) (breaking)

What changed. confirm on a rotation — REST POST /api/v1/tokens/{id}/confirm and POST /api/v1/engine-principals/{id}/credentials/confirm, plus the MCP twins confirm_api_token_rotation/confirm_engine_rotation — previously admitted on caller identity plus the successor’s token_id alone. A caller who recovered an unknown successor’s token_id out-of-band (a support ticket, a log line) could confirm — and thereby revoke the predecessor for — a rotation whose secret they never received. All four confirm surfaces now additionally require the raw successor secret in the request body/args, verified with a constant-time hash comparison against the successor’s stored hash, checked LAST — strictly after ownership, pair-state, the token_id pin, tier, scope, and the initiator binding have all already passed — before the predecessor is touched.

Who this affects. Any caller confirming with only token_id: REST now returns 400 instead of succeeding; MCP returns kInvalidParams. A caller confirming with a wrong secret gets REST 403 / MCP kPermissionDenied. Correctly-installing automation is unaffected — the secret required here is the same raw value the rotate response already returns exactly once (REST data.token), so automation that installs the successor from that response and passes it straight to confirm sees no behavior change.

If you lose the rotate response before confirming, you can no longer confirm — the secret cannot be manufactured from the token_id alone. Two recovery paths:

1. Wait for the automatic overlap-window sweep. Proof of possession gates the immediate, explicit confirm call only — the 60-second background sweep is unaffected by this change and still auto-revokes the predecessor on its own schedule with no secret required, provided the successor secret was actually installed and presented (used) at least once.
2. Revoke the unknown successor and start a new rotation (DELETE /api/v1/tokens/{token_id} or the engine-principal twin) — keeps the predecessor working immediately, at the cost of restarting the rotation.

Full detail: authentication.md “Rotating a Token”, engine-principals.md “Rotate the credential”, and mcp.md rows 61/71.

vNEXT — Guardian status routes gain real data, new denial/failure modes (#2298 item 6d) (breaking)

What changed. GET /api/v1/guaranteed-state/status and /status/{agent_id} previously returned a hardcoded placeholder (errored_rules: 0, always 200). errored_rules is now real, derived from the same per-agent compliance census (guardian_agent_rule_status) the dashboard’s Unhealthy Guards card reads, intersected against the live rule catalogue. compliant_rules/drifted_rules remain placeholder 0 pending full status ingest, tracked separately.

What a client sees.

• 403 on the fleet route (/status, no {agent_id}) — new. A service-scoped API token is now refused outright rather than admitted to a fleet-wide aggregate outside its own scope: this route aggregates across every agent’s census, and the underlying permission check does not apply a service-scoped token’s own service-tag confinement (it only checks a role grant), so admitting it here would have let a token scoped to one service read a fleet-wide count.
• A narrower errored_rules count (still 200) on the fleet route for a management-group-confined (not global) GuaranteedState:Read grant — new. This route moved from a bare global permission check (any authenticated GuaranteedState:Read holder got the unfiltered fleet-wide count) to AuthRoutes::require_list_read — the route’s SOLE gate (ADR-0017 admit-then-filter; never stacked with the flat require_permission, which does not consult management groups and cannot compose with a separate confinement check bolted on afterward). An earlier, unreleased attempt at this exact fix stacked a direct authorize_list_read call BEHIND the flat require_permission gate instead of replacing it — that composition never actually confined anyone (a confined caller was denied by the flat gate before the confinement check ever ran) and was corrected before shipping; nothing described below was ever live under that broken attempt. A caller with no GuaranteedState:Read grant at all (global or via any management group) now gets 403 instead of 200 (this is a SEPARATE outcome from the narrower-count case above — a confined grant that resolves to at least a scope, even an empty one, never gets a 403 from the confinement decision itself; a 403 from this route means either a service-scoped token, no usable grant, or, rarer, a fail-closed authorization-store fault, which denies even a caller who otherwise holds a resolvable confined grant — see rest-api.md for the full 4xx taxonomy). A caller whose grant is confined to specific management groups now sees errored_rules scoped to their visible agents only, not the whole fleet — including 0 if their groups contain no agents at all, which is still 200, not 403 (a real grant that resolves to an empty visible set is a legitimate answer, not a denial). total_rules is unaffected by this change on either route: it counts the rule catalogue, which has no agent dimension to confine. A global (non-group-confined) grant is unaffected on the fleet route too — it still sees the full fleet count, same as before.
• 403 on the per-agent route (/status/{agent_id}) — new. This route moved from a bare global permission check (any authenticated GuaranteedState:Read holder got 200, even with a management-group scope that does not cover the requested device) to the same per-device scoped check GET /guaranteed-state/device-compliance uses. A caller whose GuaranteedState:Read grant is confined to management groups that do not include the requested agent_id now gets 403 instead of a 200 carrying placeholder-zero data for a device outside their scope. A global (non-group-confined) grant is unaffected — it still passes fleet-wide, same as before.
• 503 on the per-agent route (/status/{agent_id}) — new failure mode. This route now performs a behavioral-PII access audit (guardian.device.view, same verb as GET /guaranteed-state/device-compliance and GET /dex/devices/{id}) before serving per-device data, and fails closed (503 + Sec-Audit-Failed: true) if that audit row cannot persist — the audit subsystem being unavailable is not itself new, but this route could not previously return 503 for it because it served no real per-device data before.
• 503 on either route if the Guaranteed State store degrades — new; previously the route degraded silently to 0 on any store fault.

Who this affects. Any integration polling either route that (a) uses a service-scoped API token against the fleet route — that call now needs either a non-service-scoped credential or a per-agent call against /status/{agent_id} instead, and the per-agent route is not an unconditional substitute: require_scoped_permission additionally checks that the target agent_id’s own service tag matches the token’s scope (tag_store-backed), so a service-scoped token still gets 403 there for any device outside its own service, distinct from the fleet route’s simpler “any service-scoped token, unconditionally” denial; (b) holds a management-group-confined (not global) GuaranteedState:Read grant and polls /status/{agent_id} for a device outside that scope — that call now gets 403 where it previously got 200 with placeholder data, the same confinement /guaranteed-state/device-compliance has always enforced; (c) holds a management-group-confined grant and polls the fleet route (/status) — that call still gets 200, but errored_rules now reflects only the caller’s visible agents rather than the whole fleet; a real grant that resolves to ZERO visible agents (an empty or agent-less management group) is still 200 with errored_rules: 0 (ADR-0017 INV-2 — a real grant that is simply narrow is not a denial), distinct from holding no GuaranteedState:Read grant anywhere, which is 403 on either route; or (d) treats every response as 200 — both routes can now return 403/503 and a client that does not already retry on 5xx (standard practice for every other Guaranteed State route) should add that handling. No change for a global-permission, non-service-scoped caller on the happy path beyond errored_rules becoming a real, changing number instead of a constant 0.

vNEXT — macOS antivirus posture is now probed, not asserted

The antivirus plugin’s macOS leg previously hardcoded av|XProtect|active without reading anything, and its third-party checks grepped for the wrong process name. Visible after upgrading agents:

1. av|XProtect|<state> is now a real probe of the XProtect definition bundle: active comes with a new xprotect_version|<n> row; unknown means the bundle was unreadable (never assumed active). Third-party EDR/AV detected via endpoint-security system extensions emit an additional edr|<bundle id>|<version> row each. Integrations keying on the old always-present av|XProtect|active row should treat unknown as a signal to investigate, not as product-absent.
2. The status action on macOS returns real XProtect data (definition version, freshness, Remediator/MRT versions) instead of status|not_available. A new darwin-only definition security.antivirus.xprotect_status exposes it; being a new id, it seeds on upgraded installs at next boot (unlike edited descriptions, which reach fresh installs only — the amended security.antivirus.products text lands there alone).
3. Mixed-fleet blend during rollout: agents not yet upgraded keep emitting the hardcoded XProtect row and process-grep results. Not a server bug; the presence of an xprotect_version row identifies an upgraded agent.

vNEXT — macOS firewall state now reports the Application Firewall (backend key changed)

The firewall plugin’s macOS state action previously reported the pf packet filter — which is off by default and unrelated to the Application Firewall a Mac admin means — so a Mac with the real firewall on could read disabled. Three things are visible after upgrading agents:

1. The backend row value changes from pf to appfirewall, and state now reflects socketfilterfw --getglobalstate. Integrations keying on backend|pf or treating state as pf state must switch to the new rows. Two additive rows appear: mode|block_all (only when block-all is set) and pf|<state> (the demoted pf signal; unknown on agents not running as root). state|unknown means the check was unreadable — never assumed safe.
2. Mixed-fleet blend during rollout: agents not yet upgraded keep emitting backend|pf + pf-based state. Expect both shapes side by side until the fleet is fully upgraded — not a server bug; the backend row disambiguates per device.
3. Existing installs keep the old definition description. Bundled definitions seed insert-or-skip by id at boot, so the corrected security.firewall.state description (v1.1.0) lands on fresh installs only; upgraded fleets get the corrected behavior regardless. To refresh the text, delete security.firewall.state and re-import it via POST /api/instructions/import — do not edit it in the dashboard YAML editor, which drops the definition’s spec.visualization on save.

vNEXT — Linux firewall state may now read unknown more often for ufw/iptables hosts (correctness fix)

The firewall plugin’s Linux ufw/iptables legs previously reported a parsed active/inactive state as soon as the underlying subprocess exited 0 — even if the read had actually timed out or been truncated before finishing, which could report a stale or simply wrong verdict with full confidence. state| is now gated on the same completeness check ruleset| already used (tool_ran && exit_code==0 && !timed_out && !output_truncated), matching the nftables leg’s own honest-degrade behavior added alongside it.

1. Hosts whose ufw status/iptables -S reads were marginal (slow, near-timeout, or hitting the output cap) will now read state|unknown more often post-upgrade, for the SAME underlying firewall state as before. This is the fix working as intended — a previously false-confident active/inactive becomes an honest “couldn’t tell” — not a detection regression. ruleset|unknown already had this behavior; state| now matches it.
2. No row shape changed. unknown was already a valid state value on every backend (it is the nftables leg’s own honest-degrade outcome); this only changes which reads reach it. Integrations already handling state|unknown need no changes.
3. Mixed-fleet blend during rollout: agents not yet upgraded keep the prior completeness posture for ufw/iptables; a wider spread of state|unknown across the fleet for these backends specifically identifies upgraded agents whose reads were genuinely marginal, not a server-side fault.

vNEXT — KEK rotation is now durably rate-limited (#2530) (breaking)

Before this release, POST /api/v1/secrets/kek/rotate was rate-limited only by a 5-minute process-local cooldown — cheap, but per-process and restart-clearable, so an install could in practice rotate roughly every 5 minutes (or more often across a restart, or across several servers pointed at the same database). This release replaces that with a durable, database-backed rate limit, --kek-min-rotate-interval (default 3600 seconds = 1 hour, env YUZU_KEK_MIN_ROTATE_INTERVAL), read from secrets.kek_meta.created_at on the database server’s own clock — it survives a restart and is shared cluster-wide by every server pointed at the same database. An install that could previously rotate every ~5 minutes can now be durably refused (429) for up to an hour by default. If your operational tooling or runbooks assume a short rotation cadence is always available (smoke tests immediately after install, scripted rotation drills, etc.), review them against the new default before upgrading — a fresh install’s very first rotate attempt is also refused for up to this interval, because KEK v1 is minted at boot with created_at = now(). Lower --kek-min-rotate-interval if your operational model genuinely needs more frequent rotation, but read the “runaway/abuse guard, not a rotation-schedule setting” caveat in “Key management (secrets KEK)” first — this flag also bounds how quickly you can rotate in a genuine emergency. See that section and docs/prometheus/yuzu-alerts.yml (group yuzu-secrets) for the full contract, the new VersionCeiling/QueryCanceled/ClockAnomaly failure modes, and the new observability surface.

vNEXT — API/MCP bearer tokens invalidated on upgrade (ApiTokenStore → Postgres, ADR-0030) (breaking)

The API/MCP bearer-token store moves from SQLite (api-tokens.db) to the PostgreSQL substrate as a fresh-start cutover with no data migration — every pre-upgrade API token and MCP token stops working the instant the new server starts (interactive cookie-session/SSO login is unaffected). Re-mint every API/MCP bearer token after upgrading (POST /api/v1/tokens) and update the credential wherever it is stored; plan a maintenance window and notify automation owners, since all bearer-token integrations break at once. A boot-time warning names the legacy file (inert, removable). Full detail + multi-instance caveat: the ## ⚠️ Breaking section in docs/user-manual/upgrading.md and ADR-0030.

vNEXT — GET /mcp/v1/ is a live SSE channel; ALL streaming surfaces now share one worker budget

Track 2f PR 2, plus ADR-0034. Three things change for an operator:

1. GET /mcp/v1/ no longer returns 405. It is the MCP session’s server→client SSE channel (heartbeats, Last-Event-ID resume). Nothing publishes onto it yet — notifications/progress arrives in the next rung — so a client that never issues a GET is unaffected. Behind a reverse proxy, note these are held-open responses: the server sets X-Accel-Buffering: no (nginx honours it); Envoy, HAProxy, ALB and Cloudflare need their own no-buffering opt-out.

2. Every held-open SSE response now leases from one shared budget — MCP’s GET channel, MCP streamed POST, GET /api/v1/events, the dashboard executions drawer, and the legacy /events stream. cpp-httplib is thread-per-connection, so each of those pins a worker for its entire life; previously only MCP was counted, which meant the plain-REST reserve was arithmetic rather than a guarantee. A cap hit returns 429 with Retry-After; a live stream is never evicted. This closes a starvation path that existed before MCP streaming shipped at all: enough dashboard tabs could exhaust the pool and stall plain REST.

3. The worker pool is derived from --max-sse-streams (default 128), not the other way round. The old default sized itself from httplib’s accidental 32-thread pool and yielded 12 streams — on a platform designed for hundreds of agentic clients. A blocked thread burns no CPU; its resident cost is a fraction of a virtual, platform-dependent stack reservation and has not yet been measured (ADR-0034), so the pool is sized for the workload you declare rather than off a per-thread constant. Watch yuzu_http_held_open_responses / yuzu_http_held_open_capacity; the ceiling is thread-count, and the durable fix (moving long-lived connections off the thread-per-connection server) is recorded in ADR-0034.

vNEXT — MCP notification POSTs now answer 202 (was 204); Streamable HTTP sessions added

The /mcp/v1/ endpoint gains the MCP-spec Streamable HTTP transport (track 2f, PR 1). Three changes are visible to existing clients:

1. A JSON-RPC notification POST (a request with no id, e.g. notifications/initialized) now returns HTTP 202 Accepted instead of 204 No Content. The body stays empty. This is a spec MUST and applies regardless of the kill switch below. Affected: only a strict client that asserts the status is exactly 204 (or asserts an empty-body status other than 202) — the reference clients (mcp-remote, Claude Desktop) treat any 2xx-with-empty-body as success and are unaffected. Adjust such assertions to accept 202.
2. initialize responses now carry an additive Mcp-Session-Id header. Clients that don’t use it can safely ignore it (plain-POST flows are otherwise byte-identical). Clients that do may present it on later requests; an unknown/expired/foreign id returns 404, at which point the client re-initializes (sessions are in-memory, so a server restart has the same effect). DELETE /mcp/v1/ ends a session.
3. initialize now negotiates the protocol revision instead of always returning 2025-03-26. A client that sends params.protocolVersion of a supported revision (2025-03-26 or 2025-06-18) gets that value echoed; anything else (including no protocolVersion) still returns 2025-03-26. Affected: only a newer client that requests 2025-06-18 — a legacy client requesting 2025-03-26 (or nothing) sees no change. Negotiation is independent of the --mcp-no-streaming kill switch.

New CLI flags (all optional):

• --mcp-no-streaming (YUZU_MCP_NO_STREAMING) — disable the Streamable HTTP transport: no session minting, GET/DELETE /mcp/v1/ → 405, plain JSON-RPC POST only. The 202 notification status still applies. Use this if a buffering reverse proxy interferes with streaming.
• --mcp-allowed-origin <value> (YUZU_MCP_ALLOWED_ORIGINS, repeatable) — an allowed Origin header value (scheme://host:port, exact match) for DNS-rebinding defence. An absent Origin is always allowed (the endpoint requires a credential); an empty allowlist rejects any present Origin (the secure default) — browser-based MCP clients must be allowlisted explicitly. Non-browser clients (which send no Origin) need no configuration.

vNEXT — DEX per-application sampling (procperf) is a new telemetry category

This release adds per-application resource sampling (top-N processes by CPU and working set, by image name) to the Echo edge warehouse. Superseded: it is now on by default (procperf_enabled=true on every agent, including a one-time reset of any stored false at the echo.db schema v8 upgrade — see Upgrading); this release originally shipped it off. It is a distinct telemetry category, separate from the device-level performance sampling (perf_enabled, on by default, no per-app identity) that shipped in the prior release on Windows — on Linux, device-level sampling starts automatically on upgrade to this release (see the upgrade checklist in the user manual’s Upgrading page and the Echo manual’s upgrade note; opt out per host with perf_enabled=false). To disable per-app sampling, set procperf_enabled=false via an Echo configure instruction (fleet-wide or per-device). The data is image names only (no command lines), 7-day raw / 31-day hourly retention, and is captured in the Workstream E data inventory in docs/enterprise-readiness-soc2-first-customer.md. Echo warehouse tables are now also created on every database open, so upgraded agents need no manual table-creation step. New webhook egress: the dex.signal event (operator-routed signals) — see the security questionnaire note in the assurance package if you answer data-egress questions.

vNEXT — POST /login returns 202 for MFA-enrolled users (breaking)

Programmatic clients (CI pipelines, health checks, curl scripts) that call POST /login and treat anything other than HTTP 200 + {"status":"ok"} as failure will silently break the first time an authenticating user enrolls in TOTP MFA via Settings → Security and sign-in → Two-factor sign-in. The new response is HTTP 202 with body {"status":"mfa_required","mfa_pending_token":"<opaque>","expires_in":120} — handle this branch by posting mfa_pending_token + the 6-digit TOTP code (or a XXXX-XXXX-XXXX-XXXX recovery code) to POST /login/mfa to mint the session cookie. See docs/user-manual/authentication.md § Multi-Factor Authentication for the full flow.

MFA CLI flags: --mfa-enforcement (default optional; admin-only/required now enforce — see the breaking note in docs/user-manual/upgrading.md), --mfa-step-up-window-secs (default 300), --mfa-login-pending-secs (default 120), and the break-glass --mfa-reset <username> (clears a locked-out user’s MFA and exits, writing an mfa.reset.breakglass audit row — see docs/ops-runbooks/auth-db-recovery.md). Recovery code format changed from XXXXX-XXXXX (50 bits) to XXXX-XXXX-XXXX-XXXX (80 bits) — codes printed by earlier PR1 commits remain valid until consumed or regenerated. The break-glass procedure for a user who has lost both their authenticator and all recovery codes — and the recovery path for an operator locked out by an enforcement misconfiguration (SSO IdP not asserting amr, or a sole admin who could not enroll) — lives at docs/ops-runbooks/auth-db-recovery.md.

vNEXT — service-scoped API tokens can no longer read or mutate the confirmed fleet-wide Guardian/DEX/network/inventory/Echo/Schedule surfaces found as of this release (breaking)

What changed. A pre-existing gap let a service-scoped API token read identity-linked, fleet-wide device data — and, for Guardian Baselines, MUTATE what every agent enforces — across every reporting agent, not just its own service’s agents. This note does not claim the underlying class is fully closed — three tracked issues (#3123 device-discovery, #3124 response/execution data, #3125 inventory data) document further confirmed instances on surfaces this release does not touch, found by an independent adversarial review during this branch’s own governance run; see those issues for the current list. The confinement check on these fleet-wide (no single agent_id) reads only ever verified the token’s role, never its service scope, and several of the affected reads had no per-open audit trail at all. Fixed across REST, the dashboard, and MCP: GET /api/v1/guaranteed-state/events (no-agent_id shape), GET /api/v1/dex/signals/{obs_type}, GET /api/v1/dex/perf/devices, GET /api/v1/network/devices and their MCP twins now deny a service-scoped token outright and are access-audited (dex.signal.view / dex.perf.device.view / network.device.view — the latter two had no audit coverage at all before this release); the Guardian dashboard’s fleet status, guards list, event timeline, per-Guard drilldown, baselines list, and per-Baseline fragments, plus the /fragments/dex/perf/devices, /fragments/network/devices, /fragments/dex/overview, /fragments/dex/app, and /fragments/dex/catalogue/signal dashboard fragments now deny a service-scoped token the same way. The same confinement-gap class extends beyond GuaranteedState:Read: GET /fragments/devices/list (Infrastructure:Read), GET /fragments/inventory/devices (Inventory:Read — GDPR-personal-data serial/system_uuid/primary_mac columns), and both Echo frame device pickers (GET /fragments/echo/process-tree, GET /fragments/echo/capture-sources) now deny a service-scoped token too. It also reached a MUTATING surface: POST /fragments/auto/run (the /auto Pre-flight dispatch) resolved its device cohort the same unconfined way before dispatching the configured checks — now denied the same way, before any dispatch occurs. A different shape of the same root cause also reached three more /auto Pre-flight routes: GET /fragments/auto (the saved-runs rail), GET /fragments/auto/result (the run result poll), and POST /fragments/auto/delete (run delete) scope by session->username alone, and a service-scoped API token shares its creating principal’s username (ApiToken::principal_id) — so a token scoped to e.g. one IT service could read back, poll, or delete a fleet-wide pre-flight run its own principal created interactively. All three now deny a service-scoped token the same way. The same owner-scoping shape reached the /auto Deploy stage too: GET /fragments/auto/deploy (the deploy config form) and POST /fragments/auto/deploy/delete (deployment delete) now deny a service-scoped token for the same reason; GET /fragments/auto/deploy/result (the deployment result poll) now denies too, and — because it also re-invokes the deployment engine’s mutating advance step on every call — this additionally narrows how far an already-in-flight deployment can progress for a service-scoped caller. The same owner-scoping shape reached the Schedule API too, and worse: POST /api/schedules (create) and POST /api/schedules/{id}/enable (re-enable) now deny a service-scoped token outright, even one holding both Schedule:Write and Execution:Execute — a recurring schedule dispatches fleet-wide through ScheduleRunner with no per-fire confinement at all, unattended, and reaching it needs no pre-existing state (unlike every route above). DELETE /api/schedules/{id} (delete) denies the same way as the ordinary owner-scoping class. Disabling a schedule (enable=false) is deliberately not denied by this per-file check — the kill switch is intended to stay reachable even for a service-scoped token. Caveat added 2026-08-21 (#3290 Phase 2 bucket 1a): that guarantee does not currently hold in practice — require_permission(Schedule,Write) denies a service-scoped token unconditionally at the top of this handler, before enabled is even parsed, so a disable is denied the same as an enable. Pre-existing, not introduced by this note’s own fixes; tracked as #3378. A separate, worse-than-username-scoping gap in the same feature area: GET /api/schedules (REST) and the MCP list_schedules tool now also deny a service-scoped token, because ITServiceOwner grants full CRUD on Schedule and the underlying query had no owner/service filter of any kind, so Schedule:Read alone let a service-scoped token enumerate every schedule from every other service. GET /fragments/schedules (the dashboard twin) is fixed the same way, and separately gains an RBAC gate it never had — this fragment was previously reachable by any authenticated session, regardless of role or grant. Independently, POST /api/schedules/{id}/enable had a request-parsing bug that silently reinterpreted the standards-compliant JSON boolean {"enabled": false} as enabled=true — this is now fixed; see “What to do” below for who needs to check their integration. Two more instances were found and fixed during this branch’s own governance review, both in files already touched above: the Guardian dashboard’s six MUTATING fragments (guard create, guard enable/disable, baseline create/deploy/delete/update) never got the same deny their read-only siblings got in the same earlier commit — worst of all the fixes in this note, since a service-scoped token could deploy a Baseline (a fleet-wide, full_sync operation) outside its own service; and GET /fragments/inventory/find/results, GET /api/v1/inventory/software, and MCP query_installed_software (the software-search family) never got the deny their sibling /fragments/inventory/devices got. See docs/user-manual/audit-log.md for the full list of new/changed audit verbs and docs/enterprise-readiness-soc2-first-customer.md “The machine-health audit exemption” for the narrowed CC7.2 scope.

Who this affects. Any integration authenticating with a service-scoped API token (a token bound to one service’s agents) that currently calls any of the routes/tools/fragments above will start receiving 403 instead of fleet-wide data on this upgrade — this is the intended fix, not a regression. Separately, any integration calling POST /api/schedules/{id}/enable with enabled as a native JSON boolean will see its enabled: false calls actually disable the schedule for the first time — see “What to do” below. Ordinary (non-service-scoped) operator sessions are unaffected by the confinement changes.

What to do. An integration that needs this data should use a token scoped appropriately for the surface it reads: a global (non-service-scoped) credential for a fleet-wide aggregate view, or the existing per-device REST/MCP reads (.../{agent_id} shapes), which remain available and stay confined to the token’s own service. For the /auto Pre-flight and Deploy rail/result/config/delete routes there is no per-device equivalent — a service-scoped token cannot manage its own principal’s fleet-wide pre-flight runs or deployments at all; use a non-service-scoped credential for pre-flight and deployment workflows. The Schedule API has no per-device equivalent either — a service-scoped token cannot list, create, arm, or (see the #3378 caveat above) disable a runaway schedule; use a non-service-scoped credential for schedule management, including as the kill-switch for an incident involving a service-scoped-triggered schedule. Guardian Guard/Baseline mutation and fleet-wide software search have no per-device equivalent either — use a non-service-scoped credential for those too. For the enabled parsing fix: if your integration sends {"enabled": false} as a real JSON boolean and has been relying on (or working around) it actually re-enabling the schedule, update it — that was always a bug, and the workaround is now unnecessary and will produce the opposite of the intended effect.

vNEXT — service-scoped API tokens are now denied by default everywhere, not just on the routes named above (guardian-confinement-2298 PR 3 — “the flip”) (breaking)

What changed. The note immediately above this one (PR 2 of this same series) closed an enumerated list of routes/tools/fragments — real fixes, but each one required someone to find and name the affected surface first. This release replaces that approach with a default-flip: AuthRoutes::require_permission’s service-scoped branch previously admitted any operation the ITServiceOwner role happened to grant, and that role holds broad CRUD across most securables — so a token bound to one IT service’s agents could, in practice, reach fleet-wide data anywhere that role’s grants reached, whether or not anyone had found and fixed that specific route yet. A (securable, operation) pair must now also clear a server-side allow-list that ships empty, so every route not yet migrated to real per-request confinement denies a service-scoped token outright — including routes nobody has found yet. Mirrored at the MCP tools/call dispatch layer via a new per-tool ServiceScopeClass classification (denied is the default; confined and global_safe are explicit, reviewed exceptions). This supersedes the PR 2 note’s scope: those routes stay fixed the same way, but so does everything else that shares the same require_permission/require_scoped_permission/MCP tools/call gate. See docs/adr/1006-service-scope-default-deny.md for the full design.

Who this affects. Any integration authenticating with a service-scoped API token that currently reaches fleet-wide data or actions through a route not on the seeded-empty allow-list will start receiving 403 on this upgrade — this is the intended fix, not a regression. This is a strictly larger set than the PR 2 note above: that note’s fixes were routes someone had already found; this flip additionally denies routes nobody has found yet, the moment they’re reached. Ordinary (non-service-scoped) operator sessions are unaffected.

What to do. Same guidance as the PR 2 note above: use a global (non-service-scoped) credential for anything that genuinely needs fleet-wide reach, or the per-device REST/MCP reads that stay confined to the token’s own service. There is no per-deployment or per-operator way to widen the allow-list — it is compile-time, and widening it for a specific (securable, operation) pair is a security decision requiring security-guardian sign-off, not an admin setting. If a service-scoped token’s workflow breaks on this upgrade and there is no per-device equivalent for what it was doing, that workflow needs a non-service-scoped credential going forward; there is no config flag to opt back into the old admit-by-default behavior for one route or one deployment.

vNEXT — GET /api/v1/inventory/software and MCP query_installed_software diverge from the rest of the service-scoped-deny family: filtered data, not a 403 (#3290 Phase 2) (still in this vNEXT batch)

What changed. Both notes immediately above this one (PR 2’s enumerated fixes, then “the flip”) describe GET /api/v1/inventory/software and MCP query_installed_software as denying a service-scoped API token outright (403). This still-unreleased batch changes that a second time, before either prior note has shipped: both surfaces now migrate onto AuthRoutes::require_fleet_read, a real per-request confinement gate — a correctly-confined service-scoped token gets a filtered 200 (rows limited to its own service-tagged agents, intersected with any management-group grant), not a 403. This is a least-privilege improvement, not a widening: no caller sees any row it did not already hold a grant for. (A service-scoped caller’s devices_omitted count is 0 while its scope is within the SQL push-down bound of 10 000 visible devices; only for a wider scope can it be nonzero, as the count of distinct out-of-scope devices within the scanned window(s) when a name-filtered query matches software outside its own scope — a bounded, audited existence-only signal, reviewed and accepted, see docs/security-reviews/service-scope-phase2-migrations-2026-08.md’s UP-2 ruling — not a row-level disclosure.) The two prior notes’ own routes/tools are otherwise unaffected — only this one route pair’s end-state changes. Because all three notes land in the same unreleased version, an operator reading only the two notes above would be told to expect a permanent 403 from these two surfaces that will not, in fact, occur. Full behavior table (every caller class, not just this one): docs/security-reviews/service-scope-phase2-migrations-2026-08.md.

Who this affects. Any integration using a service-scoped API token against these two specific surfaces. If it was built to expect (or work around) the 403 the notes above describe, it will now receive real data instead — code that specifically branches on a 403 here to mean “not available to this token” should be updated; code that simply surfaces the error to an operator needs no change, since the success case is a strict improvement.

What to do. Nothing required for the common case. If your integration has error-handling logic keyed specifically on a 403 from these two surfaces, update it to handle the real filtered result instead.

vNEXT — dashboard /fragments/results, the workflow executions drawer, and MCP get_agent_details now confine to the caller’s management group / service scope (#1712, ADR-0017)

What changed. These three surfaces previously gated on a flat permission check with no per-agent filter: a management-group-confined operator holding Response:Read/Execution:Read saw the whole fleet’s response data and agent statuses through the dashboard/workflow surfaces (not just their own group), and a service-scoped API token calling MCP get_agent_details was denied outright (403) rather than shown anything. All three now migrate onto AuthRoutes::require_fleet_read, the same admit-then-filter primitive GET /api/v1/inventory/software/query_installed_software already use (#3290 note above) — a management-group-confined operator now sees a genuinely narrowed, correct result on the two dashboard/workflow surfaces; a correctly-confined service-scoped token now gets a filtered result from get_agent_details instead of a 403. get_agent_details on an out-of-scope agent returns the identical “not found” response a genuinely nonexistent agent would (closing a related existence-oracle, #3564) — the tool can no longer be used to discover which agents exist outside a caller’s scope.

Who this affects. Any management-group-confined operator using the dashboard results page or the workflow executions drawer will now see a narrower view than before — this is the intended fix, not a regression; an unconfined (global) operator’s view is byte-identical to before. A confined operator who newly reaches /fragments/results will also not see the “Create Group from N Agents” button that an unconfined operator sees — its downstream flow (/fragments/create-group-form, POST /api/dashboard/group-from-results) is not yet scope-safe, so the button is withheld entirely for a confined caller rather than risk a confined-N-vs-unscoped-M mismatch (tracked #3489). Any integration authenticating with a service-scoped API token that calls MCP get_agent_details: code that specifically branched on a 403 from this tool to mean “not available to this token” should be updated to expect a real (filtered) result instead, mirroring the query_installed_software guidance above.

What to do. Nothing required for the common case. If a confined operator reports dashboard/workflow results that look “smaller than before,” that is the fix working as intended — confirm their management-group membership matches the agents they expect to see. If a service-scoped integration’s error handling is keyed on a 403 from get_agent_details specifically, update it the same way as the query_installed_software note above.

vNEXT — GET /api/v1/dex/app and GET /api/v1/dex/overview (REST + MCP) stop narrowing an elevated administrator’s device list

What changed. GET /api/v1/dex/app/get_dex_app and GET /api/v1/dex/overview/get_dex_overview carried a per-caller visible-set resolver whose confinement was already dormant for the operator class it was meant to protect — a management-group-confined-only operator has no global grant, so the routes’ bare GuaranteedState:Read permission gate denied them (403) before that resolver ever ran — but the SAME resolver ran for a JIT-elevated administrator and wrongly narrowed their device list to the base identity’s own (usually empty) management-group grant instead of the unfiltered view elevation earns. The dormant, never-narrowing-anyone-real resolver has been retired outright; every admitted caller (a global grant, RBAC disabled, or an elevated administrator) now sees the same unfiltered devices[]/top_devices[] list. GET /api/v1/dex/signals/{obs_type}/get_dex_signal_detail never carried this resolver and is unaffected.

Who this affects. A JIT-elevated administrator who previously received a narrowed (often empty) result from GET /api/v1/dex/app or GET /api/v1/dex/overview will now correctly see the unfiltered fleet. No other caller class changes: a management-group-confined operator still receives 403 from the unchanged permission gate, and a service-scoped API token is still denied outright.

What to do. Nothing required.

v0.10.0 — API token revocation is owner-scoped

Starting with v0.10.0, non-admin users can no longer revoke API tokens they do not own. A caller holding the ApiToken:Delete permission may revoke only tokens whose principal_id matches the session’s username; the global admin role, or a session with an active just-in-time admin elevation, is the only bypass. Prior releases allowed any holder of ApiToken:Delete to revoke any token, which was an IDOR (tracked in GitHub issue #222).

If your deployment uses a non-admin service account to rotate tokens for other principals — for example, a shared ops role that recycles service-account tokens on a schedule — those rotations will begin receiving HTTP 404 token not found after upgrade. To restore the behavior, either:

1. Assign the rotation account the global admin role, or
2. Refactor the rotation so each principal owns and rotates its own token.

Option (2) is the recommended long-term posture because it aligns with least-privilege. The same ownership constraint applies to both the REST path DELETE /api/v1/tokens/{id} and the HTMX dashboard path DELETE /api/settings/api-tokens/{id}. Denied attempts are recorded in the audit log with action=api_token.revoke, result=denied, and detail=owner=<real owner> so operators can distinguish an enumeration probe from a legitimate self-revoke.

Both paths return HTTP 404 token not found on a cross-user revoke attempt — identical to the response for a truly-nonexistent token — to prevent the endpoint from being used as an enumeration oracle.

v0.12.0 — Echo dashboard page + mixed-version agent caveats

The new /echo dashboard page (issue #547) surfaces every device × source pair where an Echo collector has been disabled. The page is reachable from the Echo entry in the sidebar’s Posture group; viewing requires Infrastructure:Read and the per-source Re-enable / Scan-fleet actions require Execution:Execute.

Mixed-version rollout caveat. Agents running a build older than v0.12.0 do not emit the new per-source paused_at / live_rows / oldest_ts lines on echo.status. The dashboard renders an em-dash (—) for those columns when a pre-v0.12.0 agent appears in a scan result. This is not a server bug — it is an honest “we don’t know when this collector was disabled because the agent reporting it pre-dates the field.” Operators upgrading the server before the agent fleet should expect the em-dash for any pre-existing paused sources until the agent at the affected device is upgraded. See docs/user-manual/echo.md for the full Echo dashboard workflow.

Per-operator scan state caveat. Scan results are held in the server’s memory keyed by operator username, with a 30-second per-operator cooldown to defend against retry-storms. Restarting the server clears all scan state; operators will see “No scan data yet — click Scan fleet” after a restart and a fresh scan returns within seconds. Persistence across restarts and multi-server coordination land in Phase 15.G operational hardening.

Audit log additions. Two new audit actions emit on operator activity: echo.status.scan (every Scan-fleet click; result=success/denied/failure with detail) and echo.source.reenable (every Re-enable click; result=success/denied/failure). SIEM rules can distinguish forged form submissions from genuine connectivity failures via the detail=scope_violation vs detail=agent_not_connected distinction even though the HTTP response body is identical (Agent not reachable. 404) for both cases — the body identity is load-bearing for the no-enumeration-oracle property.

vNEXT — Fleet visualization (3D) (/viz/fleet)

The fleet-visualization feature ladder lands across the feat/viz-engine branch. PRs 1–12 are shipped: agent collector + server store + REST/fragment routes + page scaffold + camera controls + cube renderer + Sprite labels + hover tooltip + interior process nodes coloured by category + intra-cube localhost edges + per-cube listening-socket spheres + cross-machine connection edges + push-based topology ingestion + three-tier stacked layout, talking-socket dots, curved tube wires, loopback-bind filter (PR 12). Remaining ladder PRs add the vulnerability overlay and final polish (LOD, edge bundling, a11y, perf).

Operator-visible state today (PRs 1–12). Navigating to /viz/fleet shows one translucent cube per fleet machine, organized into three architectural tiers — frontend cubes on the top Y plane, applications in the middle plane, databases on the bottom plane. Within each tier, machines fall onto a deterministic per-tier grid (FNV-1a hash on agent_id). Live agents render at opacity 0.18; stale agents (no echo.fleet_snapshot push within the staleness threshold) drop to 0.08. Per-OS palette (on the default Slate scheme; the other schemes follow their own chart tokens): Linux #2dd4bf, macOS/Darwin #f5c542, Windows #4c9aff. Hostname Sprite labels render below each cube. Inside each cube, one SphereGeometry dot per process is laid out deterministically (hash(pid|ppid)-mod-bucket inside the cube’s interior) and coloured from a six-category palette (on the default Slate scheme; the other schemes follow their own chart tokens): system #9aa2ae, browser #4c9aff, database #f5c542, web #2dd4bf, runtime #a78bfa, other #9199a6. Each cube’s TOP face carries a ring of coral-coloured listening-socket spheres (one per listeners[] row) with :port labels; loopback-only listeners (127.x, ::1) are now hidden from the surface — they aren’t reachable from other instances. Each cube’s BOTTOM face carries a ring of cool-blue talking-socket dots (one per unique outbound (proto, dst_ip, dst_port)). Cross-machine connections render as thick curved tubes (THREE.TubeGeometry along a CubicBezierCurve3 with vertical end-tangents) that drop from the source’s talking dot down/across to the destination’s listener sphere. External (off-fleet) destinations render as short grey stub lines with ring markers. Hover order: listener sockets → talking sockets → process dots → edges → cubes. WASD pans, drag rotates, wheel zooms.

Tier classification heuristic. classifyTier reads listeners[] port hints (DB_PORTS = {3306, 5432, 6379, 27017, 1521, 1433, 9042, 9200, 5984, 8086, 11211} / WEB_PORTS = {80, 443, 8080, 8443, 8088}) plus process-category strings ('database', 'web'). Priority is db > web > app: a host with any DB signal lands on the database tier regardless of whether it also serves web traffic. A host with no DB or web signal defaults to the application tier. Known limitations: (a) databases on non-standard ports (e.g. Postgres on 5431 for a sharded cluster) fall through to the application tier unless one of the host’s processes is classified as database by process_category.hpp; (b) WEB_PORTS is intentionally narrow — it covers reverse proxies / load balancers / API gateways on ports 80/443/8080/8443/8088 but not dev-server defaults (3000/4200/5173/8000), because in a classic three-tier deployment a nodejs/django/vite server listening on those ports is application work, not the front edge. There is no operator-accessible override today; both limitations are tracked as a follow-up issue on function-aware tier classification.

Why empty cubes show no lines. Intra-cube edges appear only for processes with active loopback socket pairs (e.g. Prometheus scraping node_exporter, a client connected to a local Redis / Postgres). A fresh agent with no inter-process loopback traffic shows process dots but no lines — this is expected, not a bug. Lines appear as workloads generate loopback flows.

Browser requirements. The stylesheet every dashboard page loads, this one included, uses color-mix() and :has(), so the supported floor is Chrome 111+, Edge 111+, Firefox 121+, and Safari 16.2+. :has() sets the Firefox figure (color-mix() alone would allow Firefox 113) and color-mix() sets the others. Browsers below the floor still load the pages but lose some state styling, such as hover, selected-row and tinted-badge backgrounds. This page also uses ES module imports resolved through an <script type="importmap"> declaration, which needs Chrome 89+, Firefox 108+, Safari 16.4+, and Edge 89+; the stylesheet is therefore the stricter requirement everywhere except Safari, where importmap raises the floor to 16.4+ on this page (all browsers released since December 2023 meet both). A browser without importmap receives a visible error overlay on the page rather than a blank canvas — the page detects via HTMLScriptElement.supports('importmap') before attempting the module load. MeshPhysicalMaterial cubes additionally require WebGL 2.0, which is bundled with all browsers in the support floor but can be disabled by enterprise group policy on locked-down terminals. Browsers without WebGL 2 will see the grid but the cubes will fail to render with a black or missing material; verify WebGL 2 is enabled in browser policy before pilot rollout. Operators on enterprise-locked browser configurations that lag 2+ years should test on a representative deployment. Page weight: the /viz/fleet route pulls three.js (~685 KB), three-orbit (~32 KB), htmx (~51 KB), the yuzu-viz renderer (~84 KB), plus the design-system CSS. Approximate first-load total is ~1.5 MB of JS (uncompressed). For metered-link or low-bandwidth deployments, consider --viz-disable as a default and enabling it selectively for ops staff.

Reverse-proxy deployment constraint. The page hard-codes static asset paths (/static/three.module.min.js, /static/three-orbit-controls.js, /static/yuzu-viz.js) and the API path (/api/v1/viz/fleet/topology). Sub-path proxy deployments (e.g. nginx location /yuzu/) are NOT supported — the absolute paths would 404 against the rewritten origin. Mount Yuzu at the root path of its host or a fronting domain.

Cache posture. The /viz/fleet page response sets Cache-Control: no-cache, no-store, must-revalidate so a server upgrade cannot leave operators with a stale page that references the new vendored assets (which themselves cache for 24 hours via max-age=86400). The vendored asset bundles are content-addressed by server binary version — a fresh page revalidation after upgrade picks up any bundle changes immediately.

Kill switch behaviour. --viz-disable / YUZU_VIZ_DISABLE disables the whole feature: the REST/fragment endpoints, the per-host drill-down routes, and the /viz/fleet and /viz/host/<id> page shells all return 503 (a plain-text “fleet visualization is disabled by an administrator” body). An operator who sets the flag will not see a half-working page. The static asset routes (/static/yuzu-viz.js etc.) are not gated — they are inert without the page — so gating them at a reverse proxy is optional, not required. The kill switch is boot-time only (seeded from config at startup); there is no runtime toggle, so disabling viz under a live incident requires a server restart.

Sizing and capacity. Push-based ingestion holds one RawAgentSnapshot per agent in an in-memory pushed_ map. Per-agent footprint is bounded by the parser caps (4096 processes + 4096 connections per snapshot, ~1–2 MB worst case, typically 5–20 KB). The map is hard-capped at 100 000 agents (kPushedMapHardCap); at the cap, a new agent’s push evicts the least-recently-seen entry (LRU by server receipt time) and emits a topology.push.evicted_for_cap audit event. Watch yuzu_viz_pushed_map_size and alert before it approaches the cap. The store is in-memory only — on server restart it is empty until agents re-push (recovery window ≈ one agent push cycle, ~30–60 s); the pull-based dispatch fetcher serves as the cold-start fallback during that window.

Permission model. The page route is auth-gated only (require_auth); the data fetch at GET /api/v1/viz/fleet/topology enforces Response:Read. An operator with a session but without Response:Read can land on the page (sees the grid and the “Access denied” overlay when the JS fetch fires) but every JSON fetch returns 403. The viz.fleet_topology audit row is emitted on the API path, not the page path — auditors querying “who accessed fleet visualization?” should filter on the FleetTopology target_type (covered in detail in audit-log.md).

Browser-side errors are not server-logged. WebGL context loss, module-fetch 503, and importmap-resolution failures surface only in the operator’s browser console. A future polish PR will add a client-error beacon; today, support engineers diagnosing “viz is blank” reports should ask for a screenshot of the browser console.

Per-host drill-down (/viz/host/<agent_id>). Double-clicking a cube opens a new tab with a 2D bipartite IPC graph (processes + sockets, Cytoscape cose layout) above the existing Echo process tree, with cross-pane select-to-highlight and a resizable splitter. The page route is auth-gated only; the data fetch (GET /api/v1/viz/host/<agent_id>/topology) enforces Response:Read and honours the --viz-disable kill switch. Audit rows are emitted as viz.host_topology (target_type = HostTopology). The agent_id path segment is allow-listed to [A-Za-z0-9._-] before templating; any other character returns 400.

Connection window — fleet_snapshot_window_seconds (Echo plugin config). echo.fleet_snapshot reports not only the connections ESTABLISHED at the exact /proc sample instant but also connections Echo observed recently in its tcp_live warehouse, so short-lived flows still reach the viz. The look-back window is operator-tunable via the Echo plugin’s KV config key fleet_snapshot_window_seconds (default 3600). It is an Echo plugin config key, not a server CLI flag — set it through the Echo plugin’s configuration surface. The 60 s sampler still cannot see sub-interval connections; that needs kernel eventing (tracked separately).

vNEXT — Agent interval triggers now functional

yuzu_register_trigger / yuzu_unregister_trigger were previously no-op stubs — no TriggerEngine was instantiated on the agent, so interval triggers were silently discarded and never fired. As a result the Echo warehouse *_live tables (tcp_live and friends) sat permanently empty in the field, and any plugin that registered an interval trigger had it silently dropped.

Upgrade caveat. After upgrading the agent daemon, registered interval triggers begin firing for the first time. On the first collect_fast cycle (default 60 s) the Echo *_live tables start populating, and operators may see a small increase in per-agent CPU/IO proportional to the number of registered triggers. Any plugin that registered an interval trigger expecting the old no-op behaviour will now have that trigger fire — this is the fix, not a regression. Server-only upgrades are unaffected; the change is entirely agent-side.

vNEXT — OTA update binary signing (#416)

Endpoints that upgraded via OTA self-update have no trust-anchor directory. The installers create it, but agent self-update replaces the binary in place and runs no installer — so on Windows and macOS, where OTA is the normal upgrade route, neither the directory nor its ACL hardening exists after upgrading. Create it before you enable signing, or --update-trust-bundle points at nothing:

# Windows, elevated, in Windows PowerShell. Paste the WHOLE block. It runs only once its
# closing brace arrives, so a broken paste runs nothing, and it stops at the first check
# that fails. Nothing in it is recursive: the directory holds only files, and a recursive
# takeown or icacls would follow a junction planted in it out of the directory.
& {
    $ErrorActionPreference = 'Stop'
    $d = 'C:\ProgramData\Yuzu\agent-certs'
    # True when $p is a plain directory, owned by Administrators or SYSTEM, protected from
    # inheritance, and granting exactly those two full control. Accounts are compared by
    # SID, so this works on any language.
    function Test-Locked($p) {
        if (((Get-Item -LiteralPath $p -Force).Attributes -band 1024) -ne 0) { return $false }
        $s = (Get-Acl -LiteralPath $p).Sddl
        ($s -match '^O:(BA|SY)G:') -and ($s -match 'D:P[A-Z]*\(A;OICI;FA;;;(SY|BA)\)\(A;OICI;FA;;;(SY|BA)\)$') -and ($Matches[1] -ne $Matches[2])
    }

    if (Test-Path -LiteralPath $d) {
        # 1. An existing directory must ALREADY be secured, and is not re-locked. This
        #    procedure does not take over one someone else created: whatever is in it
        #    already decides which updates the agent trusts. If it stops here, check what
        #    the directory contains, move any update-trust-bundle.pem you placed there
        #    yourself somewhere safe, delete the directory -- removing any junction or
        #    link first with cmd /c rmdir (directory) or cmd /c del (file), which remove
        #    the link, never its target; never Remove-Item -Recurse, which in Windows
        #    PowerShell 5.1 deletes a junction target's contents -- run this again, then
        #    copy the bundle back in.
        if (-not (Test-Locked $d)) { throw "$d already exists and is not secured. See the step 1 comment." }

        # 2. Its contents: plain files only, each already owned by Administrators or
        #    SYSTEM and not a hard link -- nothing here takes ownership of a file, so it
        #    only ever repairs files an administrator placed. Then reset them so they
        #    inherit exactly Administrators and SYSTEM. This repairs the rc1..rc5
        #    lock-out (#5196).
        $c = @(Get-ChildItem -LiteralPath $d -Force)
        $bad = @($c | Where-Object { ($_.Attributes -band 1040) -ne 0 })
        if ($bad) { throw ("Only files may be in $d. Remove these first (a junction or directory link with cmd /c rmdir, a file link with cmd /c del): " + (($bad | ForEach-Object { $_.FullName }) -join ', ')) }
        foreach ($i in $c) {
            if ($i.LinkType -eq 'HardLink') { throw "$($i.FullName) is a hard link. Remove it and copy the file in again." }
            if ((Get-Acl -LiteralPath $i.FullName).Sddl -notmatch '^O:(BA|SY)G:') { throw "$($i.FullName) is not owned by Administrators or SYSTEM, so it cannot be confirmed you placed it. Inspect it; if it is yours, run: icacls `"$($i.FullName)`" /setowner *S-1-5-32-544 /L" }
        }
        if ($c) { icacls "$d\*" /reset /L /C /Q | Out-Null }
    } else {
        # 3. A new directory is built and locked in your own temporary folder, which no
        #    other account can reach, then moved into place, so it never exists unlocked.
        #    The lock: make Administrators the owner, drop every explicit entry (/grant:r
        #    alone replaces only the accounts it names), then break inheritance and grant
        #    exactly Administrators and SYSTEM; /L on each, so icacls acts on the path
        #    itself. Never /T: on a file the (OI)(CI) grant is invalid, and icacls then
        #    leaves every existing file with an EMPTY permission list -- SYSTEM, and so
        #    the agent, can no longer read it -- while still reporting success (#5196).
        #    If something takes the name first, the checks around the move stop it; see
        #    step 1 for how to clear it.
        $stage = Join-Path $env:TEMP ('yuzu-agent-certs-' + [guid]::NewGuid().ToString('N'))
        New-Item -ItemType Directory -Path $stage | Out-Null
        icacls $stage /setowner '*S-1-5-32-544' /L /C /Q | Out-Null
        icacls $stage /reset /L /C /Q | Out-Null
        icacls $stage /inheritance:r /grant:r '*S-1-5-32-544:(OI)(CI)F' '*S-1-5-18:(OI)(CI)F' /L /C /Q | Out-Null
        if (-not (Test-Locked $stage)) { throw "The new directory could not be secured." }
        if (-not (Test-Path -LiteralPath (Split-Path $d))) { New-Item -ItemType Directory -Path (Split-Path $d) | Out-Null }
        if (Test-Path -LiteralPath $d) { throw "$d appeared while it was being prepared. Check it and run this again." }
        Move-Item -LiteralPath $stage -Destination $d
        if (-not (Test-Locked $d) -or @(Get-ChildItem -LiteralPath $d -Force)) { throw "$d is not as prepared. Remove it and run this again." }
    }

    # 4. Verify -- the same check the installer runs. Every line must say OK: the directory
    #    protected and granting exactly Administrators and SYSTEM full control, each file
    #    inheriting exactly that, all owned by Administrators or SYSTEM. A BAD line means
    #    something else has access or SYSTEM cannot read the item.
    foreach ($i in @(Get-Item -LiteralPath $d -Force) + @(Get-ChildItem -LiteralPath $d -Force)) {
        $p = $i.FullName
        if ((($i.Attributes -band 1024) -ne 0) -or (($p -ne $d) -and (($i.Attributes -band 16) -ne 0))) { 'BAD  {0}  (link or subdirectory)' -f $p; continue }
        $s = (Get-Acl -LiteralPath $p).Sddl
        $e = if ($p -eq $d) { 'D:P[A-Z]*\(A;OICI;FA;;;(SY|BA)\)\(A;OICI;FA;;;(SY|BA)\)$' } else { 'D:[A-Z]*\(A;(?:OICI)?ID;FA;;;(SY|BA)\)\(A;(?:OICI)?ID;FA;;;(SY|BA)\)$' }
        $ok = ($s -match '^O:(BA|SY)G:') -and ($s -match $e) -and ($Matches[1] -ne $Matches[2])
        '{0}  {1}' -f $(if ($ok) { 'OK ' } else { 'BAD' }), $p
    }
}

Put the bundle in only once the directory exists and is secured, by the installer or by this block: copy it in as an administrator, then check that its owner is Administrators or SYSTEM ((Get-Acl <file>).Owner). Some admin accounts make themselves the owner of files they create; if so, inspect the file and make Administrators its owner with icacls "<file>" /setowner *S-1-5-32-544 /L, or the installer and this block refuse it. A directory created or pre-staged any other way is refused by the installer, which leaves the agent service as it was, and by this block.

Endpoints that ran an agent installer from 0.14.0-rc1 to rc5: those installers took over an agent-certs that already existed, including anything in it. If you did not place the update-trust-bundle.pem there yourself, delete it and copy in a fresh one before relying on signature checking.

What this does not cover: the parent directory C:\ProgramData\Yuzu is not itself locked down (#5257), and a process that opened the directory or a file in it before it was locked keeps the access it opened with (#5258).

# macOS, as root.
mkdir -p /etc/yuzu-agent/certs && chown root:wheel /etc/yuzu-agent/certs
chmod 755 /etc/yuzu-agent/certs

Linux endpoints upgraded through .deb/.rpm need nothing here — Debian’s post-install step re-runs on upgrade, and the RPM ships the directory in its payload. A Linux endpoint that upgraded by OTA self-update, however, is in the same position as Windows and macOS and needs install -d -m 0755 -o root -g root /etc/yuzu-agent/certs.

Default behaviour is unchanged. Agents that pass no --update-trust-bundle behave exactly as before: the downloaded binary is checked against the server-supplied SHA-256 and applied. Signature verification is entirely opt-in.

What signing does and does not close — read this before relying on it. With a trust bundle configured, a compromised or impersonated update server can no longer push a binary your signing key never signed. That is the substitution attack, and it is closed.

It does NOT close ROLLBACK. The signature covers the binary’s CONTENT only, while latest_version and the SHA-256 both come from the server. So a hostile server can still serve a genuinely signed OLD release — one you really did sign — while labelling it a newer version and quoting a hash computed over those same bytes. Every agent-side check passes and the endpoint installs a real but outdated, possibly known-vulnerable build, then reports itself up to date. Closing this needs the version bound into the signed material (a signed manifest, TUF-style), which is not in this release. Treat signing as confining an attacker to binaries you have signed, not to the binary you meant to ship, and keep retiring the signing key’s authority over withdrawn releases accordingly.

New on-disk artifact — the trust anchor. /etc/yuzu-agent/certs/ on Linux and macOS, C:\ProgramData\Yuzu\agent-certs\ on Windows, created by the installers. It is deliberately NOT the server’s /etc/yuzu/certs — that path is the server’s own CA directory, which the server owns and re-tightens on boot, so no single mode serves both. You place the PEM bundle there yourself; nothing is installed into it.

Second new artifact — the signature sidecar. An uploaded signature is stored as <binary>.sig beside the package in the update directory. Like the package itself it is node-local and not covered by the database backup: it is recreated by re-uploading the package with its signature, so it needs no separate backup step.

Two new agent flags, both off by default: --update-trust-bundle (YUZU_UPDATE_TRUST_BUNDLE) and --update-require-signature (YUZU_UPDATE_REQUIRE_SIGNATURE). Setting the second without the first refuses to start, rather than running with enforcement silently inert.

Nothing to do on upgrade unless you want signing. If you do, read Signing update binaries — in particular the two-stage rollout, because a fleet that enforces before every package is signed stops updating.

vNEXT — Plugin code signing (#80)

Plugin signature verification ships in two parts: an agent-side CMS verifier and a server-side Settings UI for managing the trust bundle. Default behaviour is unchanged — agents that do not pass --plugin-trust-bundle and operators that do not upload a bundle through the new Settings card see identical behaviour to prior releases (allowlist-only, sha256 hash check).

New on-disk artifact. plugin-trust-bundle.pem in the default cert directory (Linux, and macOS as root: /etc/yuzu/certs/plugin-trust-bundle.pem; Windows: C:\ProgramData\Yuzu\certs\plugin-trust-bundle.pem). The server always writes it there, even when --ca-dir points elsewhere. Server-managed via Settings → Plugin Code Signing. Back up the default cert directory (the path above), paired with the Postgres dump (see Configuration Files); that directory is --ca-dir only when --ca-dir is left at its default. A backup that captures the database but not that directory restores plugin_signing_required=true (in runtime_config) without the trust bundle file — agents fetching the policy will receive a 500 and require-mode agents will reject every plugin until the bundle is restored.

Cert-dir collision check. The server now treats this filename as authoritative. If a prior deployment placed an unrelated PEM at this exact path for a different purpose, it will be interpreted as the plugin trust bundle on first read. This is unlikely (the filename was unused before this release) but worth confirming before upgrade. Run ls <cert-dir>/plugin-trust-bundle.pem and rename the file if it pre-exists for any other purpose.

New runtime_config key. plugin_signing_required (string "true" or "false"). Set via the Settings card; reading the key directly via the runtime-config REST surface is supported, and writing it there needs an administrator account (an MCP-tier or service-scoped token is refused) but is not recommended — the Settings UI guarantees the disk-and-DB invariants (two-phase clear, file-presence-equals-enabled).

New audit actions. plugin_signing.bundle.uploaded, plugin_signing.bundle.cleared, plugin_signing.require.changed — see audit-log.md for the result and detail conventions. SIEM rules already filtering on success/failure/denied will pick these up unchanged; no new vocabulary tokens.

Operator distribution. The server hosts the bundle at GET /api/v2/agent/plugin-policy (PluginSigning:Read, Administrator-only; unreachable by any MCP token at any tier as of #4028). Agents are pointed at a local copy via --plugin-trust-bundle <path>; the manual workflow today is curl + jq + write the JSON’s data.trust_bundle_pem field to disk on each agent host. The old GET /api/v1/agent/plugin-policy (flat top-level body, no data envelope) is deprecated — see the vNEXT note below for the migration and removal window. Automatic agent-side fetch is a forthcoming change.

Fleet-suicide caveat. The Yuzu release pipeline does not yet sign the 44 in-tree plugins under agents/plugins/. Do NOT enable “Require signed plugins” until you have signed every plugin your fleet uses, including the in-tree ones. Use the transitional mode (bundle uploaded, Require off) during rollout. The Settings card surfaces this warning inline.

vNEXT — Response templates (#254, Phase 8.2)

Superseded (ADR-0058). instruction_definitions/instruction_sets moved from per-replica SQLite to the shared PostgreSQL substrate — the sqlite3 instructions.db commands and the SQLite schema_meta v1→v3 probe-and-stamp mechanism described below no longer apply; that mechanism doesn’t exist in the current codebase. For backup/restore and schema-version checks on this store today, use the PostgreSQL Substrate section’s pg_dump/ pg_restore procedure against the instruction_store schema, not the commands in this subsection. Left as-written below for historical reference (this is what the Phase 8.2 migration looked like on SQLite, in case an operator is diagnosing an upgrade that predates the Postgres cutover).

Phase 8.2 ships named response-view configurations attached to each InstructionDefinition: a column subset, sort order, and filter presets the dashboard’s filter-bar View dropdown surfaces. The feature is purely additive — operators who never author a template see a synthesised __default__ view that is byte-identical in behaviour to the prior “show all columns, sort by Agent” default.

Schema migration. instruction_definitions gains one column: response_templates_spec TEXT NOT NULL DEFAULT '[]'. The migration ledger advances from v2 to v3. ALTER TABLE ADD COLUMN with a constant default is O(1) in SQLite (metadata-only, no table rewrite); the migration is non-destructive.

Pre-upgrade snapshot (recommended). Take a backup of the InstructionStore database before upgrade:

cp /var/lib/yuzu/instructions.db /var/lib/yuzu/instructions.db.bak
# or, for a hot-running server, use SQLite's online backup:
sqlite3 /var/lib/yuzu/instructions.db ".backup /var/lib/yuzu/instructions.db.bak"

Post-upgrade validation.

sqlite3 /var/lib/yuzu/instructions.db \
  "SELECT version FROM schema_meta WHERE store='instruction_store';"
# expected output: 3

If the value is 2 instead of 3, the migration did not run — check the server logs for MigrationRunner: instruction_store migrated to v3 (or for a probe-and-stamp failed line in the InstructionStore section).

Boot wedge recovery. A corrupt schema_meta row or a pre-existing response_templates_spec column with a missing schema_meta v3 stamp will trip the probe-and-stamp guard, which fails closed (server logs InstructionStore: probe-and-stamp failed; closing database). To recover, restore the snapshot or apply the column manually:

# Stop the server first.
systemctl stop yuzu-server

sqlite3 /var/lib/yuzu/instructions.db \
  "ALTER TABLE instruction_definitions ADD COLUMN response_templates_spec TEXT NOT NULL DEFAULT '[]';"
sqlite3 /var/lib/yuzu/instructions.db \
  "INSERT OR REPLACE INTO schema_meta (store, version, upgraded_at) VALUES ('instruction_store', 3, strftime('%s','now'));"

systemctl start yuzu-server

New audit actions. response_template.create, response_template.update, response_template.delete — see audit-log.md for the failure-reason vocabulary. SIEM rules already filtering on success/denied will pick these up unchanged.

Authoring caveats. The dashboard YAML editor’s lightweight line-scanner does not extract spec.responseTemplates into the indexed column; author through POST /api/v1/definitions/import (JSON envelope) or the REST template endpoints. Imported templates with the reserved id: __default__ are silently dropped during normalisation.

vNEXT — webhook store moves to Postgres; secrets now encrypted at rest (ADR-0057) (breaking)

WebhookStore moves from SQLite (webhooks.db) to the PostgreSQL substrate, and the webhook HMAC signing secret is now envelope-encrypted at rest (SecretCodec, ADR-0010) instead of a plaintext column. No production fleet ever ran a pre-Postgres build of this store, so there was no real legacy data to migrate — the mandatory backfill this originally shipped with was retired shortly after (#3623, ADR-0057’s Update); a legacy webhooks.db file is never read into Postgres, only hardened to 0600 and counted for a diagnostic warning at boot. POST /api/webhooks now returns 400 for an invalid URL, distinct from a 503 for a genuine store/database error; both POST and DELETE return 503 (rather than a silently-empty/silently-failed result) on that latter case, previously ambiguous. A legacy webhooks.db file left on disk still holds every pre-cutover secret in plaintext and is restricted to the file owner where the platform supports it (POSIX only, see the ADR) — see rest-api.md for guidance on disposing of it securely. Full detail: docs/adr/0057-webhook-store-postgres-migration.md and the ## ⚠️ Behaviour change: webhook store moves to Postgres (ADR-0057) section in docs/user-manual/upgrading.md.

vNEXT — initialize can answer 503 during a graceful shutdown (#3042)

With MCP streaming on, initialize now returns HTTP 503 / JSON-RPC -32015 (“Server is shutting down”) for a narrow, transient window (seconds, not the deploy’s whole grace period) if it lands after ServerImpl::stop() has begun draining live sessions. Affected: any MCP client integration — the reference clients and most SDKs already treat a non-2xx initialize as a transient failure and retry/reconnect; a client that specifically asserted “initialize never 503s” needs updating. No retry_after_ms is given (this process has no visibility into when a replacement instance will be reachable); reconnect and re-initialize once it is. A session that was already live when shutdown began instead receives a clean notifications/yuzu.stream_closed close frame (reason: session_terminated) rather than a bare connection drop — see MCP — Troubleshooting for the full symptom/cause/fix.

vNEXT — Linux adapter names no longer carry the @peer suffix (breaking)

What changed. The network_config agent plugin’s adapters and ip_addresses actions were rewritten to read Linux interfaces via rtnetlink instead of parsing ip -o link show text. The previous parser copied iproute2’s display form for any interface with a parent link — a veth pair, a VLAN sub-interface, a tunnel device — which renders as <name>@<parent>, for example eth0@if74 or eth0.100@eth0. The new leg reads the kernel’s own interface name directly (IFLA_IFNAME), which never carries that suffix: the same two interfaces now report as eth0 and eth0.100. This is a deliberate choice, not an oversight — eth0@if74 is iproute2 presentation syntax that no other data source on the host recognises, and it was also silently breaking the adapter’s link-speed lookup (/sys/class/net/eth0@if74/speed never resolves; the un-suffixed path does), so restoring the old string would also restore that bug.

Who this affects. Any deployment with a Linux fleet segment where at least one host has an interface with a parent link — containerised hosts (veth is the default Docker/Kubernetes networking model), VLAN-tagged interfaces, or tunnel devices (WireGuard, OpenVPN, GRE) — and has a saved dashboard filter, a report, an external inventory join, or automation keyed on the adapter-name string containing @. A plain bare-metal or VM fleet with no such interfaces is unaffected; the field’s value changes only for interfaces that previously carried the suffix.

Before upgrading, check whether this affects you. On a representative Linux host, compare:

ip -o link show | awk -F': ' '{print $2}' | grep '@'

If this returns any interface names, those hosts will report a different (shorter) adapter name for those interfaces after upgrade — update any saved filter, dashboard, or join key that references the old @-suffixed string. If it returns nothing, this note does not affect you.

vNEXT — software_actions.installed_count’s reported number can change after upgrading (breaking)

What changed. The software_actions agent plugin’s installed_count action was migrated off popen/a powershell -Command shell-out onto native counting, and the counting rule changed on both platforms it touches:

• Linux. The dpkg presence filter now counts packages in the hi (installed, held) status alongside ii (installed), matching the filter installed_apps and vuln_scan already use. A host with any held packages reports a higher count than before, by exactly the number of held packages.
• Windows. installed_count now reads a raw subkey count of the registry Uninstall key via RegQueryInfoKeyW, replacing a powershell -Command Get-ItemProperty ... | Measure-Object pipeline. The two are not byte-identical: the registry read counts every subkey, the old script counted only subkeys carrying at least one property — a rare shape, but not one the new count can be relied on to match exactly. Both the old and new implementation read only the default (64-bit) registry view; 32-bit applications registered under WOW6432Node were never counted by either one and still are not.

Who this affects. Any deployment with automation, a dashboard trend line, or an alert threshold keyed on the literal count|N value from installed_count — particularly a Linux fleet with apt-mark holded packages, or any Windows fleet (the registry-count/property-count divergence is rare but not bounded to a specific host shape, so it isn’t pre-checkable the way the Linux change is).

Before upgrading, check whether this affects you. On a representative Linux host:

dpkg-query -W -f='${db:Status-Abbrev}\n' | grep -c '^hi'

A nonzero result means that host’s installed_count will report a higher number after upgrading, by exactly that many. If your automation only compares the count to a rough threshold or trend, no action is needed; if it asserts an exact expected value, re-baseline it after upgrading.

vNEXT — GET /api/v1/agent/plugin-policy is DEPRECATED; use GET /api/v2/agent/plugin-policy (#4028, #4144)

What changed, and why there are now two versions. This route already existed pre-#4028 (documented here as the trust-bundle-PEM distribution path for agent config management) but was off the REST-v1 API-parity ledger. #4028 originally hardened it onto the same conventions every other /api/v1/* route in this manual uses — but did so in place, reshaping GET /api/v1/agent/plugin-policy’s response envelope without a version bump. An external review (#4144) caught that this violates docs/api-versioning-policy.md’s own rule (“changing an error envelope’s shape” requires a /api/v2/ sibling and a deprecation cycle). Corrected: the hardened behavior now lives at GET /api/v2/agent/plugin-policy; GET /api/v1/agent/plugin-policy is restored to its exact pre-#4028 shape and formally deprecated.

GET /api/v2/agent/plugin-policy (the hardened route — migrate to this):

• Success body is enveloped, under data. {"data": {"enabled":..., "required":..., "cert_count":..., "sha256":..., "subjects": [...], "bundle_unreadable": <bool>, "bundle_error"?: <string>, "trust_bundle_pem":...}, "meta": {"api_version": "v1"}}. subjects and bundle_unreadable/bundle_error are new fields versus the v1 shape below.
• Error body is the standard A4 envelope — error.code/error.message/error.correlation_id/error.retry_after_ms.
• Authorization is the PluginSigning:Read RBAC permission (seeded Administrator-only). An MCP-tier token is denied this route regardless, at the tier chokepoint, before the permission check runs.
• Two 503 (retry) responses where v1 answers 200 with a value it cannot stand behind: a runtime_config_store outage backing the required flag (v1 silently reports required:false), and a concurrent trust-bundle upload/clear racing this request’s PEM re-read (v1 can pair a stale sha256 with the newly-uploaded trust_bundle_pem on this exact race — #4144 also closed this integrity gap by deriving every field from a single read of the bundle).

GET /api/v1/agent/plugin-policy (DEPRECATED — unchanged since before #4028): flat top-level body ({"enabled":..., "required":..., "trust_bundle_pem":..., "cert_count":..., "sha256":...}), require_admin gate (not RBAC), the old bespoke error shape, no audit row, and the pre-existing runtime_config_store/PEM-race behavior described above. Kept exactly as it always was — no new hardening is being back-ported to it.

Deprecation window (per docs/api-versioning-policy.md). Announced 2026-09-08. GET /api/v1/agent/plugin-policy keeps working for at least 90 days and at least one intervening feature release, whichever is longer (so no earlier than 2026-12-07, and not before the next feature release ships) — removal will carry its own CHANGELOG.md Breaking/Removed entry per the cycle’s Step 3, never a silent drop.

Who this affects, and what to do. Any script, admin tool, or manual curl pipeline reading this route directly. Point it at /api/v2/agent/plugin-policy and read response["data"]["trust_bundle_pem"] (was response["trust_bundle_pem"]) — no CLI flag or configuration change is needed, this is a URL and response-shape change only. No action is required before the removal window closes, but migrating now also picks up the TOCTOU integrity fix.

vNEXT - a poisoned inventory record now makes two result-set producer routes refuse instead of silently narrowing (#4496) (breaking)

What changed. POST /api/v1/result-sets/from-inventory-query and the MCP create_result_set_from_inventory_query tool now refuse (503/kInternalError) when a candidate inventory record’s stored data_json nests past the JSON depth guard (the #2437-class poisoned/over-nested record). Previously the poisoned record was silently skipped with no signal at all, and the call succeeded, materialising a result set that had quietly excluded that agent.

Who this affects. Any deployment with an existing stored inventory record (inventory_store.inventory_data) whose data_json nests deeper than the JSON depth guard allows - most likely a row predating the #2437-class write-side guard. A call to either route above that previously succeeded despite such a record now refuses outright instead of materialising a result set narrower than the true match set.

How to identify the affected record(s). There is no SQL-level detection query or purge endpoint for this today - the only current signal is a server log line at WARN level, evaluate_inventory: excluding agent=<agent_id> plugin=<plugin> - data_json nests too deeply (#2437-class), emitted once per excluded record on every call that reaches the guard. Watch the server log for this line following a 503 from either route above to identify which agent/plugin’s record needs re-collection at the source. Restart the affected agent to force a full resync; if the same WARN line (or a subsequent poison_excluded refusal) recurs afterward, the source data itself genuinely exceeds the depth guard and re-collection alone will not clear it - the source plugin needs a fix, or an operator can manually run DELETE FROM inventory_store.inventory_data WHERE agent_id=... AND plugin=... (the row repopulates on the next sync cycle if the source data is unchanged).

vNEXT - a malformed (unparseable) inventory record ALSO now makes the same two result-set producer routes refuse (#4496 follow-up) (breaking)

What changed. Same two routes as the entry above (POST /api/v1/result-sets/from-inventory-query and the MCP create_result_set_from_inventory_query tool), a DIFFERENT trigger: a candidate inventory record whose stored data_json fails to parse as JSON at all (a syntax defect, not over-nesting) now also refuses (503/kInternalError, reason=parse_error_excluded) rather than being silently skipped. Kept as a separate, distinctly-named cause from poison_excluded above so an operator can tell WHICH guard excluded a record.

Who this affects. Any deployment with an existing stored inventory record whose data_json is not valid JSON - the write-side depth guard (gateway_service_impl.cpp’s json_exceeds_depth) checks nesting depth only, not general JSON validity, so a malformed-but-shallow blob has always been able to reach storage. A call to either route above that previously succeeded despite such a record now refuses outright instead of materialising a result set narrower than the true match set. The read-only POST /api/v1/inventory/evaluate route instead surfaces this as a results_excluded_by_parse_error count field alongside results_excluded_by_poison (present only when non-zero), the same posture as the depth-guard entry above. POST /api/inventory/query is UNAFFECTED by this specific change: it never calls evaluate_inventory() (it lists records by agent/plugin/time metadata, not by evaluating conditions against parsed JSON) and already degrades gracefully on a parse failure, returning the record with its data field as a raw string rather than dropping it - there is no narrowed-match-set hazard on that route for this cause.

How to identify the affected record(s). Same mechanism as the entry above: a server log line at WARN level, evaluate_inventory: excluding agent=<agent_id> plugin=<plugin> - data_json failed to parse (malformed JSON), emitted once per excluded record. Watch the server log for this line following a 503 (reason=parse_error_excluded) from either producer route to identify which agent/plugin’s record needs re-collection at the source. Same remediation ladder as the entry above: restart the affected agent to force a full resync; if the same WARN line (or a subsequent parse_error_excluded refusal) recurs afterward, the source data itself is genuinely malformed and re-collection alone will not clear it - the source plugin needs a fix, or an operator can manually run DELETE FROM inventory_store.inventory_data WHERE agent_id=... AND plugin=... (the row repopulates on the next sync cycle if the source data is unchanged).

vNEXT — certificates delete on Windows fails closed instead of silently switching stores (#4377) (breaking)

What changed. The security.certificates.delete action on Windows opened the named store under the LOCAL_MACHINE hive; if that open failed, it silently retried under CURRENT_USER and deleted there if a same-named store existed and matched. delete now opens LOCAL_MACHINE only — a failure to open there reports error|<store> store could not be opened; nothing removed (non-zero exit, status|not_found never returned in this case) and removes nothing, rather than falling back. list/details are unaffected in kind — they keep the CURRENT_USER read fallback, and now disclose it more completely (an explicit output row and a CONSTRAINED/PARTIAL result whenever a store fell back, an earlier store couldn’t be opened, or its enumeration was incomplete, not only when the final match itself came from the fallback).

Who this affects. Any deployment where a delete automation’s target LOCAL_MACHINE store can become unopenable (a permissions misconfiguration, a corrupted store) and a same-named CURRENT_USER store exists on the same host. Before this fix, that combination made delete silently remove a certificate from a store the caller never named — which is the bug this closes, not a regression. Automation that only checked for a non-zero exit code was already correctly informed on every OTHER failure path; this is the one combination where the old behavior masked a wrong-target delete with an apparent success.

Before upgrading, check whether this affects you. If your delete automation for Windows hosts does not already treat a non-zero exit / an error|... result as a hard failure requiring investigation, add that check now. There is no way to pre-check for the specific LOCAL_MACHINE-unopenable-with-a-CURRENT_USER-fallback condition from outside the action itself; the fix is unconditional and has no opt-out.

vNEXT — service-scoped tokens can no longer create a result set from an inventory query (#4980) (breaking)

What changed. POST /api/v1/result-sets/from-inventory-query now denies a service-scoped API token outright (403), matching its 8 non-dispatch sibling result-set routes. Before this fix, the route gated only through the Inventory:Read fleet-read chokepoint (fleet_read_fn), which admits and confines a service-scoped caller rather than denying it — but the result set this route creates is owner-scoped to the minting principal’s identity, not the token’s own service tag, so a service-scoped token holding Inventory:Read could mint a set that its minting principal’s other tokens/session could then read back.

Who this affects. Any integration using a service-scoped API token to call this specific route. No legitimate use of a service-scoped token should have depended on this — the other 8 result-set routes have always denied this way, and the MCP twin (create_result_set_from_inventory_query) was never affected. There is no opt-out.

What to do. Nothing for a correctly-built integration. If a service-scoped workflow did depend on this route, it needs a non-service-scoped credential going forward, same as the flip notes above; there is no per-route or per-deployment allow.

vNEXT - agent logging is now asynchronous, with a new self-exit code 5 (#4666 PR-2) (NOT breaking)

What changed. The agent no longer writes log lines synchronously on the thread that produced them; main.cpp installs a bounded async hand-off logger as the process’s spdlog default (see “Agent logging is now asynchronous”, the paragraph immediately before Stopping a wedged agent under systemd Units, further down this page, for the queue-size/memory-cost/overrun details). Agent shutdown gains one more possible self-exit code, 5, distinct from the existing 1/3/4, fired from either of two causes at that step: tearing down that logger (flushing its queue, joining its worker thread) does not complete within an internal 2-second watchdog, or the teardown itself fails outright (an immediate exit with no wait at all — see Stopping a wedged agent under systemd Units, further down this page, for both causes in full).

Impact. Not a breaking change: no flag, wire format, API, or default behavior changes, and no operator action is required. Log output looks the same (same pattern/JSON formatting, same --log-file/rotation behavior) with one exception below. The externally-visible differences are: (1) under sustained log-sink overload, the agent can now silently drop older queued lines (overrun_oldest) rather than blocking, so a very bursty logger under a stuck sink may show gaps instead of a stall; (2) a shutdown wedge that used to hang or need SIGKILL before this and the related #2233 watchdogs landed can now self-exit with code 5 specifically, in addition to the pre-existing 1/3/4; (3) the “Received signal, shutting down…” line the agent prints on SIGINT/SIGTERM/Ctrl-C used to be a raw, fixed-format write straight to stderr — it is now routed through the same configured logger as everything else, so it picks up the configured pattern (or JSON structure under --log-format json) and now also lands in --log-file when one is configured, not stderr alone. A plain substring match against the message text (the default text pattern keeps the original words verbatim) is unaffected; a line-anchored or byte-exact matcher, or one that assumed this specific line was stderr-only, needs updating; (4) that same line is now an ordinary info-level call rather than an unconditional raw write, so at --log-level warn or above — a configuration this page itself recommends for agents to cut noise, see “If log volume matters” above — the line is silently absent entirely, whereas before it always printed regardless of level. If you rely on this line’s presence to confirm a clean/intentional stop, either keep --log-level at info or below, or switch to checking the process exit code (0 = clean) instead.

Who should check. Any operator running a supervisor script or monitoring rule that pattern-matches the agent’s process exit code against a fixed set ({0,1,3,4} or similar) should widen it to include 5. An unrecognised exit code there should not be interpreted as “impossible” or treated as a different failure class than the documented watchdog exits already are. See Stopping a wedged agent under systemd Units, further down this page, for what each code means and how they interact.

vNEXT — device_ids on result-set creation now requires Infrastructure:Read (#4983, breaking)

What changed. POST /api/v1/result-sets, MCP create_result_set, and the dashboard CSV-paste import (POST /fragments/result-sets/create) previously accepted an arbitrary caller-supplied device_ids array with only a type check and a size cap — no RBAC check of any kind ran when device_ids was supplied, beyond the ordinary session-authenticated/owner-scoped gate every call to these routes already passed. All three now additionally gate a non-empty device_ids through the admit-then-filter Infrastructure:Read chokepoint (fleet_read_fn, ADR-0017) and validate that every entry both exists and is visible to the caller’s own scope — a nonexistent or out-of-scope id now rejects the whole request (400 RESULT_SET_UNKNOWN_DEVICE_ID on REST, the JSON-RPC equivalent on MCP, an error toast on the dashboard fragment) instead of being silently accepted as a member. Each entry is also now capped at 256 bytes (400 RESULT_SET_DEVICE_ID_TOO_LONG on REST, the JSON-RPC equivalent on MCP, an error toast on the dashboard fragment) — a separate, additional new rejection path on all three surfaces.

Who this affects. Any RBAC-enabled deployment where a non-admin principal creates a result set by supplying device_ids directly. Checked against this repository’s actual seed grants (RbacStore::seed_defaults), of the 8 seeded built-in roles only Administrator and ITServiceOwner hold Infrastructure:Read — Viewer, Operator, PlatformEngineer, ApiTokenManager, Reviewer, and ForensicsAnalyst do not (ForensicsAnalyst was added later, so it is not a role that could have created result sets with device_ids before this release). A principal holding one of the five roles that existed at that release (Viewer, Operator, PlatformEngineer, ApiTokenManager, Reviewer) who could previously create a result set with device_ids (the RBAC-enabled default before this release carried no permission check on this field at all) will now receive a new 403 on all three surfaces. This is a genuine, confirmed regression path, not a hypothetical one. A caller who omits device_ids, or supplies an empty array, is completely unaffected on any surface — the gate is never consulted. RBAC-disabled deployments (the shipped default) are entirely unaffected — the legacy permission fallback admits every authenticated session regardless of this change.

Check your automation and your dashboard operators, not just your admins — this is the failure that hides. Before upgrading an RBAC-enabled deployment, identify every non-admin principal (API token, service account, or interactive session) that currently calls POST /api/v1/result-sets / create_result_set / the dashboard’s CSV-paste import (“New result set from a list of device ids”) with a non-empty device_ids. This includes a dashboard operator pasting a CSV of ids just as much as an automation calling the API — both hit the identical new gate. Audit your AuthDB/API-token principal list and your dashboard-role assignments for holders of Viewer, Operator, PlatformEngineer, ApiTokenManager, or Reviewer who use this capability; each will start receiving 403 the moment this release is live, with no advance warning from the caller’s own side.

What to do. Of the 7 fleet-wide-assignable built-in roles, only Administrator holds Infrastructure:Read — ITServiceOwner (the 8th, management-group-scoped-only) also holds it. A principal who needs to keep creating result sets with device_ids after this release needs one of those two roles; per rbac.md’s own current state, there is no REST API or Settings UI yet for creating a custom role or editing an existing role’s permission set (tracked as planned, not implemented) — the only way to grant Infrastructure:Read to a role that doesn’t already hold it is a direct RbacStore::set_permission() call against the shared Postgres rbac_store schema (see rbac.md’s “Custom roles” section for the exact mechanism). There is no per-route opt-out; a caller that only ever creates result sets via parent_id/source_payload (no device_ids) is unaffected and needs no change.

Settings Page

The Settings page is the primary administrative interface. It is accessible only to users with the admin role and is rendered server-side using HTMX.

URL: /settings (redirects to the sign-in page, /home, if unauthenticated; a signed-in non-admin is sent to /)

The Settings page is organized into sections, each loaded as an HTMX fragment. Changes take effect immediately without a server restart unless otherwise noted.

Sections

Plugins and updates (System page)

An admin-only page under Settings in the sidebar reads the same data as the Settings panels and REST routes it sits over (no new write route):

• Plugins and updates (/settings/plugins): stored plugin configuration (read-only; no agent reads these values yet, and secrets cannot be listed), a kill-switch lookup showing the all-OS layer and each OS layer and which row decided it (the server refuses to send commands to a switched-off plugin; it does not recall commands already sent or stop work an agent starts itself), product packs, and the Plugin Code Signing and Agent Updates panels.

The panels apply the page’s admin gate and the matching REST route’s own permission, and show a note (not an empty list) when denied or when the store could not be read. The stored-configuration panel decides from the one authorization check GET /api/v1/plugin-config and the MCP list_plugin_config tool use (the ADR-0017 list-read gate, which admits an active just-in-time elevation and refuses a management-group-confined grant), so the panel and those two routes make the same list decision for any principal who also passes the page’s administrator gate (the REST route and the MCP tool record the confined-grant refusal). A panel refused for a signed-in browser session leaves no denial row in the audit log (the panels load by themselves, so a refusal is not an attempt); an API-token or engine caller refused the same panel leaves the usual auth.admin_required or auth.permission_required row. Opening the page itself as a non-administrator is audited.

Certificates and secrets (System page)

An admin-only page under Settings in the sidebar reads the same data as the Settings panel and REST routes it sits over (no new write route):

• Certificates and secrets (/settings/pki): the certificate authority (trust mode, algorithm, validity, SHA-256, CA and CSR downloads, enterprise chain import), the revocation list (published time, next update, revoked count, and Out of date when a revocation is not on the published list), the issued-certificate inventory (the newest 500) with Active, Expired and Revoked states and a revoke confirmation worded for the certificate’s purpose, and a read-only key-encryption-key status panel (active key version, re-wrap state, live key versions, operation lock; Security:Read). Key rotation and re-wrap stay on POST /api/v1/secrets/kek/rotate and /rewrap.

The page itself requires an administrator. Each panel is loaded from its own route, which enforces the matching REST permission (Security:Read), and the page shows a note (not an empty list) when a panel is denied or cannot be read. A revocation list whose next-update time has passed shows Expired list rather than Current.

Revoking an agent certificate from the dashboard

When the server runs its built-in CA, Settings → Certificates and secrets (or Settings → Internal CA on the Settings page) lists the newest 500 issued certificates; for an older one, find its serial with GET /api/v1/ca/issued?offset= and revoke it with POST /api/v1/ca/revoke. A caller whose view of the fleet is confined (management-group confinement), or who holds no Inventory:Read grant, is refused that route with a 403, and the panel shows a note in place of the issued-certificate list while keeping the CA details, the root and CRL downloads and the import form. A confined Security:Delete holder can still revoke a serial it already knows through POST /api/v1/ca/revoke, but cannot list serials: use an unconfined account, or read revoked serials from the CRL (/api/v1/ca/crl). The panel’s reply to a revoke or an import is never blank for such a caller: it shows the panel with a note where the list would be, plus a confirmation banner and a success toast after a success; the banner adds that the issued-certificate inventory is not shown (see the note below). A caller without Security:Read gets the plain note instead of the panel, and after a success the same confirmation banner and toast, without that sentence about the inventory. To revoke one (e.g. a decommissioned or compromised agent):

1. Find the agent’s row in the inventory (match on Subject = agent_id or the Serial).
2. Optionally type a reason (e.g. key compromise, decommissioned) — it is stored on the revocation record and audited.
3. Click Revoke and confirm. The panel refreshes in place showing the cert as Revoked and the public CRL is republished automatically. If that publish fails (the Settings panel does not show this, while Certificates and secrets shows Out of date; the REST/MCP revoke response returns crl_republished:false and a ca.crl.published failure is audited), the server republishes it on its own: on the next 15-second freshness tick once the cause clears, or up to about 5 minutes later if that attempt fails too. You do not need to revoke again.

Revocation takes effect immediately server-side: the agent is refused on its next connection, and any already-open command stream is torn down by the revocation sweep within ~15s. A revoked agent cannot re-enroll its way back by deleting its key (re-issuance is refused while the revocation stands). The same operation is available over REST (POST /api/v1/ca/revoke) for automation. The dashboard action is CSRF-protected and requires Security:Delete.

Gateway-proxied agents: revocation is enforced on direct-connect agents only — an agent reaching the server through a gateway presents its cert to the gateway, not the server, so also disconnect it at the gateway. See docs/auth-architecture.md “Gateway-proxied agents: revocation scope”.

Issuing a code-signing certificate (gap-matrix #10)

The internal CA can issue a code-signing-only leaf for signing agent plugins (agents/plugins/*.so, see docs/user-manual/agent-plugins.md “Plugin Signing”), so operators don’t have to stand up a separate hand-rolled CA just to get a signer cert. This is not a key-minting service — the operator generates and keeps the private key; the server only sees and signs a CSR.

End-to-end workflow:

1. Generate a keypair and CSR locally (never sent to the server):

   openssl ecparam -genkey -name prime256v1 -out signer.key
   openssl req -new -key signer.key -out signer.csr -subj "/CN=Plugin Signer"

2. Submit the CSR to POST /api/v1/ca/issue-code-signing (Security:Write) with a label (becomes the leaf’s subject CN, ^[A-Za-z0-9._-]{1,64}$) and an optional validity_days (default 365, hard ceiling 730):

   curl -s -X POST -H "X-Yuzu-Token: $TOKEN" -H 'Content-Type: application/json' \
     -d "{\"csr_pem\": $(jq -Rs . < signer.csr), \"label\": \"plugin-signer-2026\"}" \
     https://localhost:8443/api/v1/ca/issue-code-signing > issuance.json
   jq -r .certificate_pem < issuance.json > signer.pem
   jq -r .chain_pem       < issuance.json > signer_chain.pem

   The equivalent MCP tool is issue_code_signing_cert (same schema, supervised-tier + approval-gated like every other Security:Write MCP tool). The issuance is also visible afterward in Settings → Internal CA’s issued-certificate inventory (purpose: code-signing).
3. Sign the plugin with the returned leaf, including the issuer chain:

   openssl cms -sign -binary \
     -signer signer.pem -inkey signer.key -certfile signer_chain.pem \
     -in chargen.so -outform pem -out chargen.so.sig

4. Distribute the trust bundle to agents. Since the leaf chains to the server’s own CA, the trust bundle agents need is simply the already-public CA root — GET /api/v1/ca/root — no separate CA-distribution step. Ship it as /etc/yuzu/plugin-trust-bundle.pem alongside the .sig files, same as any other plugin-signing setup.

Why this route is scoped to code-signing only. Usage is hard-pinned to codeSigning (never clientAuth/serverAuth), so a leaf issued here is rejected outright by the mTLS SSL_CLIENT purpose check and can never impersonate an agent at the #1118 identity gate regardless of its CN — which is what makes it safe to ship while the general POST /api/v1/ca/issue (operator-chosen CN, operator-chosen EKU) stays deferred. See docs/pki-architecture.md “Code-signing certificate issuance” for the full argument.

Revocation and expiry — operational caveats. Revoking a code-signing leaf (POST /api/v1/ca/revoke, same as any other issued cert) republishes the CRL and marks it revoked in the issued-certificate inventory — but the agent-side plugin-load verifier (docs/user-manual/agent-plugins.md “Plugin Code Signing”) does not consult the CRL, so revocation does not yet stop an agent from loading a plugin already signed with that leaf. Closing this gap is a tracked follow-up (#4234); until then, treat “revoked” as “will not be reissued/renewed”, not “immediately rejected fleet-wide”. Separately, the signer leaf’s own expiry already matters: once its not_after passes, CMS_verify rejects the signature at the agent’s next restart — track not_after from the issuance response and re-sign/re-issue before it lapses.

TLS Configuration

The Yuzu server has two independent TLS surfaces:

1. HTTPS — the dashboard and REST API (port 8443 by default). Configured via --https-cert / --https-key (the Settings page only displays it). Disabled with --no-https.
2. gRPC TLS — the agent listener (port 50051) and the management listener (port 50052). Configured via --cert / --key / --ca-cert (and optionally --management-cert / --management-key / --management-ca-cert for a separate management cert). Disabled entirely with --no-tls.

The two surfaces are configured separately and can be in different states (e.g., HTTPS enabled but gRPC TLS disabled for a local UAT against a remote dashboard).

Both surfaces pin the same TLS 1.2 cipher allow-list and version floor; see TLS policy for the exact list, what’s proven by CI, and what isn’t pinned yet.

HTTPS via CLI Flags

HTTPS is enabled by default. Pass --https-cert and --https-key at server startup. Use --no-https for development without TLS. See Server CLI Flags.

gRPC TLS via CLI Flags

The recommended posture is mutual TLS (mTLS) — the server presents a certificate and verifies a client certificate from each connecting agent:

./yuzu-server \
  --cert /etc/yuzu/grpc-server.crt \
  --key  /etc/yuzu/grpc-server.key \
  --ca-cert /etc/yuzu/agent-clients-ca.crt

If you have not yet stood up a CA for issuing agent client certificates, you have two fallback options:

Option 1 — One-way TLS (server cert is presented but client certs are not verified):

export YUZU_ALLOW_INSECURE_TLS=1   # required as a second confirmation
./yuzu-server \
  --cert /etc/yuzu/grpc-server.crt \
  --key  /etc/yuzu/grpc-server.key \
  --insecure-skip-client-verify

This applies to both the agent listener and the management listener. The server emits an ERROR-level startup banner and a 5-minute recurring reminder log line for the duration the listener runs in this mode. An audit event with action server.tls_degraded is also written every 5 minutes for SOC 2 evidence.

Option 2 — --no-tls (no encryption, no peer authentication, plaintext gRPC):

./yuzu-server --no-tls

This is the supported posture for local UAT, customer demos, and development. The administrative surface is ungated — anyone reachable on port 50052 can issue management RPCs. The server emits a multi-line ERROR-level startup banner and a 5-minute recurring reminder. Do not run --no-tls against any network you do not control end-to-end.

Upgrade note (v0.12.0)

The --allow-one-way-tls flag was renamed to --insecure-skip-client-verify AND now requires YUZU_ALLOW_INSECURE_TLS=1 in the environment as a second confirmation. Existing deployments that pass --allow-one-way-tls (or the new flag name) will refuse to start after upgrade until the env var is set. The old flag name remains accepted with a deprecation warning for one release.

For systemd-managed deployments, add the env var via a drop-in:

sudo systemctl edit yuzu-server   # creates /etc/systemd/system/yuzu-server.service.d/override.conf

[Service]
Environment="YUZU_ALLOW_INSECURE_TLS=1"

Via Settings Page

The Settings → Security and sign-in page has two certificate panels, and neither one turns HTTPS on:

1. Dashboard HTTPS is read-only. It shows whether HTTPS is on, the port, the certificate and key paths and the HTTP redirect. The dashboard certificate and key come from --https-cert / --https-key and reload from disk when the files change (not when the server runs with --no-cert-reload). There is no HTTPS toggle and no dashboard-certificate upload.
2. Agent connections (mTLS) uploads or pastes the three agent (gRPC) files: server certificate, private key and CA certificate. Each replace asks for confirmation, overwrites the file with no backup, and does not change the running listeners (gRPC certificate hot-reload is not supported). After a restart the server uses the uploaded file only if --cert, --key and --ca-cert name it; if they do and the material is wrong, the server will not start. The gateway command plane pins the server certificate, so changing it can stop command forwarding. The mTLS switch on this panel changes only the value this server reports; it is not saved.

Certificate Requirements

• Format: PEM-encoded.
• The certificate file may contain the full chain (leaf + intermediates).
• The private key must not be password-protected.
• On Unix, the private key file must not be readable by group or others. The server will refuse to start if permissions are too open. Fix with: chmod 600 /path/to/key.pem.
• A key uploaded or pasted in Settings is created owner-only (0600) and replaced atomically by rename. Mount the certificate directory, not the single server-key.pem file, into a container: a single-file bind mount cannot be replaced and the upload fails with an error in the server log.
• For production, use certificates signed by a trusted CA. Self-signed certificates work but require agents to trust the CA.

Certificate Hot-Reload

The server automatically detects when HTTPS certificate or key files change on disk and hot-swaps the SSL context without requiring a restart. This enables zero-downtime certificate rotation, including automated renewal via ACME/certbot.

How it works:

1. The server polls the cert and key file modification times at a configurable interval (default: 60 seconds).
2. When a change is detected, the new files are validated:
   • PEM format is parseable
   • Certificate and private key match
   • Key file permissions are secure (Unix: not group/others-readable)
   • Files are not empty and not larger than 1 MB
3. If validation passes, the SSL context is updated atomically. New connections use the new certificate; existing connections are unaffected.
4. If validation fails, the current certificate is preserved and an error is logged.

Configuration:

Best practice for atomic file replacement:

# Write to temp files first, then move atomically
cp new-cert.pem /etc/yuzu/certs/server.crt.tmp
cp new-key.pem /etc/yuzu/certs/server.key.tmp
chmod 600 /etc/yuzu/certs/server.key.tmp
mv /etc/yuzu/certs/server.crt.tmp /etc/yuzu/certs/server.crt
mv /etc/yuzu/certs/server.key.tmp /etc/yuzu/certs/server.key

Observability:

• Log messages: cert-reload: certificate hot-reloaded successfully or cert-reload: ... failed
• Audit events: action cert.reload with result success or failure
• Metrics: yuzu_server_cert_reloads_total (counter), yuzu_server_cert_reload_failures_total (counter)

Limitations:

• HTTPS only. gRPC mTLS certificate hot-reload is not supported. Rotating gRPC certificates still requires a server restart.
• The server logs a warning at startup if gRPC TLS is enabled with cert reload: “gRPC TLS certificate hot-reload is not yet supported.”

Gateway upstream peer authorization

The gateway-upstream service (--gateway-upstream) requires an authorized gateway peer: an SPKI public-key pin, or an explicit acknowledgement. Without the check the service would rely on transport CA membership and the network boundary alone (the decision record, Gateway peer authorization (minimal), 2026-10-07, states what that exposes and the residual risks). The decision is made once at boot, after certificate bootstrap and before any listener is built. A configuration that cannot be authorized makes the server exit non-zero before binding a port, with a message naming the flag to change.

What is checked on every call. Each of the five gateway-upstream RPCs (ProxyRegister, BatchHeartbeat, ProxyInventory, NotifyStreamStatus, ForwardGuardianMessage) is refused with gRPC UNAUTHENTICATED and one fixed message (gateway peer not authorized) unless the caller presented a client certificate on a TLS connection that the transport authenticated, and that certificate:

1. parses, and lists serverAuth in its extended key usage (the generated default-gateway leaf carries both serverAuth and clientAuth; agent leaves issued by the server’s CA are clientAuth-only, so they fail this step even if their key were pinned);
2. has a public key whose SHA-256 (over the full SubjectPublicKeyInfo DER) is in the pin set; and
3. is inside its validity window at the time of the call. The TLS handshake checks validity only once per connection, so this check runs on every call: a certificate that expires on a long-lived connection is refused from that moment (outside_validity). The window opens 5 minutes before the certificate’s notBefore (an allowance of this per-call check only; the TLS handshake applies none), and a call is denied when the current time, in whole seconds, is later than notAfter.

The pin is the identity. Membership of the install CA, the certificate subject, its issuer and the request contents never authorize a call. The reason for a refusal is never sent to the caller; it is the reason label of yuzu_server_gateway_peer_denied_total.

It applies on every port. The check wraps the service, and the server registers that one service on the single gRPC server that also listens for agents (--listen, default 50051) and for management (--management, default 50052). The check therefore runs wherever the service can be reached, including the agent listener on default certificates, which requests but does not require a client certificate (a caller with no certificate is refused as not_authenticated). Acknowledged mode (below) disables the check on all of those ports together, not only on --gateway-upstream.

Refusal reasons and remedies. reason is a closed set of eight values. The checks run in the order of this table, so an unpinned key is attributed not_pinned (or no_server_auth_eku) whatever its dates.

What decides at boot.

If this server runs no gateway at all, omit --gateway-upstream instead of configuring any of the above. The shipped Linux unit passes it unconditionally (see Upgrading).

Automatic pin. When the gRPC listener credentials are the server’s generated default files (the agent gRPC listener runs on the default server certificate and the default CA, and no --management-cert, --management-key or --management-ca-cert override is set) and no explicit pin is given, the server pins its own default-gateway.pem (in --ca-dir, or the platform default certificate directory when that is unset). The condition is about the gRPC credentials, not about defaults in general: HTTPS running on generated defaults while gRPC uses your own certificates does not trigger it, because the gateway you deployed is then not necessarily presenting that file. With your own certificates the automatic pin does not exist and the server refuses to start until you pin the gateway key.

Moving from the automatic pin to an explicit pin. Explicit pins replace the automatic pin; they are not added to it. Switching a running install to an explicit pin for a new gateway key therefore denies the gateway that still presents default-gateway.pem. Keep the old key admitted during the overlap by listing it as well, for example --gateway-peer-pin-file <ca-dir>/default-gateway.pem alongside the new pin, and remove it once the gateway presents the new key (see “Rotation”).

Pinning your own gateway.

# SHA-256 of the gateway certificate's public key (the pin), as bare hex
openssl x509 -pubkey -noout -in gw.pem | openssl pkey -pubin -outform DER | openssl dgst -sha256 | awk '{print $NF}'

yuzu-server ... --gateway-upstream 0.0.0.0:50055 --gateway-peer-pin <64-hex>
# or point at the certificate itself
yuzu-server ... --gateway-upstream 0.0.0.0:50055 --gateway-peer-pin-file /etc/yuzu/gateway-peer.pem

Without the awk step openssl dgst prints a label before the digest (for example SHA2-256(stdin)= <hex>, or (stdin)= <hex> on older OpenSSL), which is not a valid pin. The same values can be set in the environment (YUZU_GATEWAY_PEER_PINS, comma-separated, and YUZU_GATEWAY_PEER_PIN_FILE, one path), for example in /etc/yuzu/yuzu-server.env for the systemd unit. A value given on the command line makes the matching environment variable ignored entirely: the two are not merged, so a rotation that lists the old pin in the environment and adds the new one on the command line drops the old one. List every pin in the same source. A gateway leaf needs serverAuth for the server’s check and clientAuth for the TLS handshake to complete: a serverAuth-only leaf fails the handshake, which is a different symptom from a pin refusal. Pin files must be readable by the server’s user; under the packaged systemd unit keep them under /etc/yuzu (ProtectHome and PrivateTmp hide /home and /tmp). A pin file is integrity-critical, since whoever can write it decides which gateway key is admitted: make it writable only by the server administrator. The server does not check the file’s permissions or ownership, so that protection is yours to provide.

Pin file format. Either PEM certificates with nothing outside the -----BEGIN CERTIFICATE----- / -----END CERTIFICATE----- blocks (any other text, including openssl x509 -text output, makes the file malformed), each block contributing its own pin, or hex pins one per line (blank lines and # comment lines ignored). A file that mixes the two is malformed. A PEM block that is passphrase-protected (it carries a Proc-Type: 4,ENCRYPTED header) is refused as malformed and never prompts for a passphrase. ASCII or UTF-8 only (a leading UTF-8 byte order mark is accepted); a UTF-16 file, which Windows PowerShell 5.1 writes for > and Out-File, is refused. At most 64 KiB and 32 entries (certificate blocks or hex lines, counted before de-duplication) per file, and 64 distinct pins across all sources. The file must be a regular file (a symbolic link to one is followed); a FIFO, a directory, a device or a file the server cannot open for permission reasons is reported only as could not be read, with no cause printed.

Lockout diagnostic (not_pinned). Three values have to agree, and each can be read without guessing:

1. The gateway certificate’s SPKI hash. Compute it with the command under “Pinning your own gateway”; its first 16 hex characters are the prefix the logs show.
2. The denied call’s key. The warning gateway peer denied: rpc=<rpc> reason=not_pinned spki=<16 hex> (and the session.gateway_peer_denied audit row’s detail) carries the first 16 hex of the key the server saw. If it equals the prefix from step 1, the gateway presents the key you computed and the server’s pin set lacks it. If it differs, the gateway is presenting a different certificate from the one you hashed.
3. The server’s pin set. At boot with explicit pins the server logs, at info level, gateway peer authorization: enforcing N pin(s) from A --gateway-peer-pin value(s) and B pin file(s); pins are fixed until restart; pin prefixes (first 16 hex): p1,p2,.... On the generated default certificates, where the automatic pin applies, the line is instead gateway peer authorization: enforcing; the default gateway certificate (<path>) is pinned automatically (default gRPC certificates in use); pin prefixes (first 16 hex): <p>. The prefixes are sorted and at most 8 are listed, then ... and K more, so with more than 8 pins a prefix may be hidden: compare it through the spki= prefix in the denial log line, or recompute the pin from the certificate. Read the line on every replica: a prefix that one replica lacks is the divergence behind an intermittent not_pinned. This is the only way to compare pin sets across replicas.

Inputs the server refuses at boot (so a bad value cannot start the server, and the error names the first 8 characters of the value): the colon-separated form of openssl x509 -fingerprint -sha256 (AB:CD:...), a SHA256 Fingerprint= prefix, a SHA2-256(stdin)= or (stdin)= prefix, a sha256: prefix, and any value that is not exactly 64 hexadecimal characters (63 or 65 characters, for example). Surrounding whitespace is ignored.

The mistake that boots and then fails not_pinned: a 64-hex hash of the whole certificate (DER) instead of its public key (SPKI), for example openssl x509 -in gw.pem -noout -fingerprint -sha256 with the colons removed. It looks like a valid pin, so the server accepts it, and no call ever matches it. Recompute the pin with the SPKI command above, or give the certificate itself to --gateway-peer-pin-file, which computes the SPKI hash for you.

Acknowledged mode (--insecure-gateway-peer). Intended for plaintext development and demo rigs, which have no peer certificate to pin. The server logs GATEWAY PEER AUTHORIZATION IS DISABLED once at boot at error level, writes one server.gateway_peer_authz_disabled audit row (when the audit store is open) and sets yuzu_server_gateway_peer_authz_mode: insecure_ack when TLS is off or there is no client CA, insecure_ack_tls when TLS is on and a client CA is present, a production-shaped configuration in which the acknowledgement switched off a control that would have worked. The service then admits any caller that can reach any port the shared gRPC server listens on, which is the exposure this control exists to close. Treat the acknowledgement as a time-boxed risk acceptance and do not use it where those ports are reachable by anything you do not trust. The boot audit row records only the mode and whether TLS is on: it has no owner, no expiry and no record of where the flag was set. Keep that record yourself, for example:

Gateway peer authorization acknowledgement (--insecure-gateway-peer)
Owner:          <named person or team accountable for it>
Date accepted:  <date>
Expiry:         <date by which a pin replaces it, or by which this record is renewed>
Ticket:         <change or risk-register reference>
Set in:         <unit file, env file, compose file or installer parameters>
Reachable from: <networks that can reach --listen, --management and --gateway-upstream>

Only insecure_ack_tls (TLS on with a client CA, the production-shaped case) has a shipped alert. insecure_ack (a plaintext stack, or TLS with no client CA) has none, because it is the intended state of a development rig. For a production Prometheus add an operator-side rule such as yuzu_server_gateway_peer_authz_mode{mode="insecure_ack"} == 1. The shipped plaintext rigs carry it: the full, sanitizer and viz UAT composes (deploy/docker/) and the scripts scripts/start-stack.sh, scripts/start-UAT.sh, scripts/win-start-UAT.sh and scripts/integration-test.sh run the server from the build under test and pass the flag; the two compose files that pull published release images (docker-compose.uat.yml and deploy/docker/docker-compose.demo.yml) and the compose wizard’s plaintext gateway output set the environment variable YUZU_INSECURE_GATEWAY_PEER=1 instead, because an older release image ignores an unknown environment variable but exits on an unknown flag.

Rotation (restart-based). Pins are fixed at boot, so changing the gateway key is a coordinated restart. With one server: add the new pin (keep the old one), restart the server, switch the gateway to the new key, confirm calls are admitted, remove the old pin, restart again. Keep the old and the new pin in the same source: a --gateway-peer-pin on the command line makes YUZU_GATEWAY_PEER_PINS ignored entirely (they are not merged), and the same holds for the pin-file option and its variable. With several server replicas (HA), do the same with the union of old and new pins on every replica:

1. Add the new pin next to the old one on every replica, and roll the replicas one at a time. Verify: each replica’s boot log line gateway peer authorization: enforcing N pin(s) ... lists both pin prefixes (see “Lockout diagnostic”), and yuzu_server_gateway_peer_authz_mode{mode="enforce"} is 1 on every replica.
2. Switch the gateway to the new key: put the new certificate and key in the files the gateway is configured with. A running gateway presents the certificate it loaded when it dialled, so the switch takes effect when the gateway redials (a gateway redials after its connection drops and reads its certificate and key from the configured files when it does: measured with the gateway’s gun-based upstream client against a local TLS listener on OTP 28, see The upstream client). A redial can still present the previous certificate for up to about 2 minutes after the files change, because the Erlang ssl PEM cache cleans every 120 s (measured on OTP 28.4.2 with a stub by a reviewer, not on a Yuzu gateway), so allow about 2 minutes. Restart that gateway node, one at a time, only if its agents can take the disconnect, as in cause class B of the incident runbook: unless the agents run a build with the #5183 fix (shipped in 0.15.0), a gateway restart leaves released agents wedged until their service is restarted, and a restart also clears the certificate cache. Verify: the first 16 hex characters of the SPKI hash computed, with the command under “Pinning your own gateway”, from the certificate file the gateway is configured with are the new key’s prefix.
3. Confirm that yuzu_server_gateway_peer_denied_total{reason="not_pinned"} stays flat on every replica. A flat counter does not prove the gateway switched keys: both pins are admitted and the server records only denials, so no signal on either side names the key the gateway presented. Treat the gateway as switched only after a redial that followed the file change plus the 2-minute allowance in step 2; step 4 is then the test, because a gateway that still presents the old key is refused with not_pinned and an spki= equal to the old key’s prefix as soon as the old pin is removed (putting the old pin back and restarting the replicas restores admission). The alerts look at a 5-minute increase window and stay active for a few minutes after the last denial (about 6 minutes for YuzuGatewayPeerDenied): check that the counter is flat, not that the alert has cleared.
4. Remove the old pin from every replica and roll them again. Verify: the boot log lines list only the new prefix, and the counter is still flat.

On the generated default certificates every replica auto-pins its own default-gateway.pem, which is per --ca-dir. If you ever give replicas different certificate directories, supply the same explicit pin on all of them. During a mixed-version window an old replica (which has no peer authorization) admits calls that a new replica refuses, so upgrade every replica before relying on the control. Renaming default-marker.json (the way to apply new --cert-san values, see PKI architecture) mints a new gateway leaf key: the restarted server auto-pins the new key, and a gateway that dialled before the restart presents the old one until it redials. The server restart drops the connection and forces that redial, so the gateway may already present the new certificate. That is cause class B in the incident runbook: check first, and restart that gateway node, one at a time, only if denials continue and its agents can take the disconnect, because unless the agents run a build with the #5183 fix (shipped in 0.15.0) released agents stay wedged afterwards.

Withdrawing a gateway key. Remove the pin from every pin source and restart every serving replica. There is no reload and no revocation check on this path: revoking the certificate’s serial in the CA store does not remove a key pin, and a certificate reissued over the same key is still admitted. If the key itself is compromised, replace it as well as withdrawing the pin, since a copied credential stays valid until its pin is gone from every replica.

Incident runbook: gateway refused, then corrected. A pin mismatch is not a connection failure. The TLS session completes and the server answers each call with gRPC status 16 (UNAUTHENTICATED), which the gateway reports as upstream RPC errors rather than TLS-handshake failures. A refused NotifyStreamStatus is logged by the gateway as Failed to notify stream status for <agent>: .... What happens to the refused notice then depends on the gateway build:

• A gateway that includes #5695 keeps a refused notice and sends it again. Status 16 is not one of the statuses that blame the request (3, 5, 6, 9, 11), so the notice goes back to the head of the gateway’s held store. While the breaker is closed each refusal also counts as a breaker failure and the notice is sent again at once, so about five refusals (the default failure threshold, plus any sends already in flight) open the breaker. From then on the notice waits behind the open breaker and is sent again as the half open probe, after 10 s, 20 s and then every 30 s with the default settings, for as long as the refusal lasts: the resend is bounded in rate, not in time, and nothing drops it on a timer. Once the cause is fixed and the gateway’s calls are admitted, the next probe succeeds, the breaker closes, and the held notices, CONNECTED and DISCONNECTED alike, are sent first in first out. A held notice is still lost if the held store fills (a CONNECTED lost that way arms a full registration replay, which runs once the breaker has closed; see Gateway, “Stream Status Notification”), if a newer notice for the same session replaces it, or if the gateway or its upstream client process stops while it holds it. If the server then answers a delivered notice acknowledged=false (for example for a session it no longer knows), the gateway keeps it and sends it again with a backoff (Gateway, “When the server does not record a notification”). INFERRED from the code and unit tests (the unit tests use a transport error and status 14; status 16 takes the same code path), not run on a rig.
• An older gateway does not retry a notice refused on an ordinary notification worker, so a session whose CONNECTED notice was refused can stay without its placement after the cause is fixed when the server keeps its record of that session: heartbeats for a session the server already knows are acknowledged and trigger no replay. The exception is a gateway that includes #4632 while its circuit breaker is open: the head held notification is the half open probe, a refused probe goes back to the head of the held store and is sent again at the next probe, so it is delivered once the cause is fixed.

A server that restarted has no record of the gateway’s sessions, reports them unknown in its heartbeat replies, and a gateway that includes the #1197 heartbeat-verdict consumer (#5383) replays them; an older gateway does not act on the verdict until it is upgraded (see Upgrading, “BatchHeartbeatResponse gains unknown_session_ids”). Recovery needs an operator decision, and which one depends on where the cause is (the cause classes below). On a gateway that includes #5695, correcting a server-side cause is enough for the refused notices to be delivered, but a gateway-side cause still needs the gateway to redial (status 16 does not make it redial, so until then every probe is refused again), and the step 4 checks still decide whether placement converged.

What a refusal window costs. While the gateway is refused: sessions whose CONNECTED notice was refused stay unplaced, on a gateway that includes #5695 until the refusal stops and the held notice is sent, and on an older gateway possibly until the agent reconnects (above); ForwardGuardianMessage frames dropped in the window are never replayed; a refused DISCONNECTED notice leaves a stale placement (on a gateway that includes #5695 until the held notice is sent after the refusal stops); and the gateway retries heartbeats every second and logs a WARN each time (noise, not a fault). Agent Register calls through the gateway fail with INTERNAL (Upstream registration failed: ...) from the FIRST refusal (yuzu_gw_agent_service.erl), not only once the breaker has opened, and those refused registrations count toward the gateway’s circuit breaker. ProxyRegister (a replay registration included), ProxyInventory and, on a gateway that includes #4632, the half open probe (a held NotifyStreamStatus: a refused probe reopens the breaker, a successful one closes it) feed the breaker. On a gateway that includes #5695 a refused ordinary NotifyStreamStatus sent while the breaker is closed feeds it too, and because the refused notice is sent again at once, one refused notice is enough to open the breaker (above). A refused BatchHeartbeat or ForwardGuardianMessage does not feed it, and on an older gateway neither does a refused ordinary NotifyStreamStatus (yuzu_gw_upstream.erl; Gateway, “Replay failures feed the shared circuit breaker”). So the breaker can stay closed, and the gateway’s /readyz can answer 200, while every heartbeat is refused; that is what a window with no registering agents looks like (and, on a gateway that includes #5695, with no agent connecting or disconnecting either). When the breaker does open, /readyz answers 503 (its circuit_breaker check) and a gateway that includes #4632 holds stream-status notifications (yuzu_gw_upstream_notify_pending rises) and sends them first in first out once the half open probe succeeds; a notification is lost at admission only when the store is full, counted in yuzu_gw_upstream_notify_dropped_total{reason="at_capacity"}. A gateway without #4632 drops them and counts reason="circuit_open". There is no breaker state gauge, and /readyz alone does not tell you the state: circuit_breaker is true for both closed and half open, and 503 is also returned when any other readiness check fails. The signals are the gateway log lines Circuit breaker: OPEN (will probe in <ms>ms) and Circuit breaker: half_open -> closed (probe succeeded), and the counter yuzu_gw_upstream_circuit_transitions_total{state}. Status 16 is an RPC result and the gateway never redials by itself, so an open breaker whose cause is on the gateway side does not close by waiting (inferred from reading the gateway code, not tested).

Three cause classes decide what happens to the gateways. The circuit state alone never decides it.

• A. The cause was on the SERVER side (a wrong or missing pin, a missing acknowledgement, other server configuration; the gateway’s certificate is unchanged). Correct the server configuration and restart the serving replicas. Do not restart the gateways first, unless step 4 shows that placement was lost (class C): a gateway restart disconnects every agent the node holds, and on agents without the #5183 fix (shipped in 0.15.0) released agents stay wedged afterwards (see Gateway, “What happens when the server restarts”). On a gateway with the #1197 consumer (#5383) the gateway recovers through its existing replay path, after at most one open period of the breaker (at most 30 s with the default settings on a gateway that includes #5695, 300 s on an older one, and only if it opened), plus the agent heartbeat interval, plus the replay drip (the timing allowances in step 4). To roll the change back, revert the pin or acknowledgement on every replica and restart the replicas: the pin set is fixed at boot.
• B. The cause is on the GATEWAY side (the gateway’s certificate expired or was renewed, its key was rotated, renaming default-marker.json minted a new gateway leaf, or the gateway points at the wrong certificate). The gateway presents its current certificate and key only when it redials: see step 5, which first checks whether it already has. A renewal over the SAME key changes nothing on the server, so step 3 does not apply to it.
• C. Placement loss is suspected after a refusal window (notify errors or drops rose, or a sample agent is unreachable although the counts agree). Use it only when the verification in step 4 fails after the timing allowances there: see step 5.

1. Find the cause on the server: yuzu_server_gateway_peer_denied_total{reason} per replica, and the session.gateway_peer_denied audit rows for authenticated peers. Verify: you can name one reason, and whether the cause is on the server side (class A) or the gateway side (class B). For not_pinned, run “Lockout diagnostic” above.

2. Restore the pins or the credentials. Class A: put the correct pin (or the acknowledgement) on every replica. Class B: put the corrected certificate and key in the gateway’s configured files; a NEW key needs its pin on every replica first (see “Rotation”), while a renewal over the same key changes nothing on the server. Triggers include: pin sets that differ between replicas; a gateway certificate switched before the new pin was deployed; and a gateway certificate that expired while its connection was open (see “Certificate expiry” below). Verify: for class A, the pin or acknowledgement is present in the configuration of every replica (for example in the unit, the environment file or the compose file; it is not live until the restart in step 3). For class B, the first 16 hex characters of the SPKI hash computed from the certificate file the gateway is configured with (the command under “Pinning your own gateway”) are the prefix you expect.

3. Restart the serving replicas only if step 2 changed server configuration: class A, or class B with a NEW key whose pin you just added. A same-key renewal needs no server restart. When it applies, restart every serving replica that was refusing, so all replicas agree: this is the server restart that step 2’s change needs, made once after step 2 is in place on every replica, not as a second restart. Verify: each replica’s boot log line gateway peer authorization: enforcing ... lists the pin prefix of the key the gateway presents (see “Lockout diagnostic”), and yuzu_server_gateway_peer_authz_mode{mode="enforce"} is 1 on each. For class A, yuzu_server_gateway_peer_denied_total is also flat on every replica; for class B it keeps rising until the gateway presents the corrected certificate (step 5). Check that the counter is flat, not that the alert has cleared: the alert looks at a 5-minute increase and stays active for about 6 minutes after the last denial.

4. Check three separate things. None of them stands in for another, and agreeing counts alone do not prove placement.

   • Registration counts. Compare the totals, not the series (a per-series subtraction returns nothing, because the label sets differ):

     sum(yuzu_gw_agents_current) - sum(yuzu_agents_connected{job="yuzu-server"})

     Sum across all gateway nodes and all serving replicas, and use the job name your Prometheus gives the server. yuzu_gw_agents_current is a gauge sampled every 10 s. The server’s gauge counts agents that connect directly as well as those behind a gateway, so zero or below is what convergence looks like, and it can hide a gateway-fronted shortfall as large as the number of direct agents. A result that stays above zero means the gateways hold sessions the servers do not count. yuzu_agents_connected counts REGISTRATION (ProxyRegister).

   • Placement. Placement (the route to the gateway that holds an agent) is set by NotifyStreamStatus, separately from registration, and a refused or dropped CONNECTED notice leaves an agent counted but unreachable for commands (Gateway, “Dispatch reachability after the session is adopted”). Over the recovery window there should be no rise in the raw yuzu_gw_upstream_rpc_errors_total{rpc_name="notify_stream",code="16"} or yuzu_gw_upstream_notify_dropped_total (the code="16" series has no sample until its first event, so an absent series means none; a gateway with #4632 creates yuzu_gw_upstream_notify_dropped_total at 0 for reason="at_capacity" and reason="unknown", so there a flat 0 means none and an absent series means an older gateway, where the series likewise has no sample until its first drop). Then dispatch a read-only command to a sample agent and read its result. Dispatch with POST /api/command and a body such as {"plugin":"hardware","action":"manufacturer","agent_ids":["<agent-id>"]} (hardware.manufacturer is a read-only Inventory:Read action; the caller needs Execution:Execute). A 200 carrying "agents_reached":1 shows only that the server accepted the command: for a gateway-attached agent the server merely queues it for the gateway, so the proof is the agent’s own result, read with GET /api/v1/responses/<command_id>?agent_id=<agent-id> (the command_id is in the dispatch response; Response:Read), which must return a row with status 1 within a few seconds. An agent that is registered but has no placement is expected to be refused with 503 and failed to send command to any agent; 503 with no agent connected is the answer when the server holds no reachable agent at all (both read from command_routes.cpp and agent_registry.cpp, not run). No endpoint reports which gateway node holds an agent, so test several agents spread across the fleet and, where you know which agents dial which gateway node, behind every node: a pass on a few agents does not prove that all are placed.

   • Breaker. Read it from the log lines and the transitions counter described above, not from /readyz alone.

   Timing allowances (read from the gateway code and gateway.md, not measured here). The open-period cap applies only if the breaker opened: the open period doubles from 10 s up to 30 s with the default settings on a gateway that includes #5695 (circuit_breaker_max_reset_timeout_ms), and up to 300 s on an older gateway. The unknown-session verdict arrives only when the agent’s heartbeat appears in a gateway flush: up to one agent heartbeat interval (30 s by default) plus one flush (1 s by default). On a gateway that includes #5695 the replay then starts after a random delay of up to registration_replay_start_jitter_ms (1 s by default) and runs up to upstream_replay_concurrency (24 by default) registrations at once, so its length is the gateway’s agent count times the server’s ProxyRegister time divided by 24. On an older gateway it runs as a drip of registration_replay_spacing_ms (20 ms by default) per agent plus the RPC time of each, so 10,000 agents take at least 200 s. Allow for all of it before judging. On a gateway older than #5695 a first successful registration or inventory call after failures also starts a replay of the gateway’s registrations (yuzu_gw_upstream.erl, on_success); since #5695 a breaker recovery starts no replay; only the heartbeat verdict, or a CONNECTED lost at a full held store, does. Verify: the registration expression settles at zero or below within those allowances, the placement checks pass, and yuzu_server_gateway_peer_denied_total stays flat.

5. Make the gateway present its current certificate. Only class B, or class C when step 4 failed, reaches this step.

   • Class B. First check whether the gateway has already redialled and presents the new certificate: a server restart, or any dropped connection, forces a redial. It has when yuzu_server_gateway_peer_denied_total and yuzu_gw_upstream_rpc_errors_total{code="16"} have stopped rising, no new spki= denial lines appear in the server log, and the certificate file the gateway is configured with has the expected SPKI prefix (compute it with the command under “Pinning your own gateway”). Read the raw counters twice, a minute apart, rather than using increase(): the gateway series has no sample until its first event, and a series born at its first value is invisible to increase() (the server series is pre-seeded at 0). After the files are replaced, a redial can still present the previous certificate for up to about 2 minutes (the Erlang ssl PEM cache cleans every 120 s; measured on OTP 28.4.2 with a stub by a reviewer, not on a Yuzu gateway), so wait about 2 minutes before judging. Restart that gateway node, one node at a time, only if denials continue after that wait and only if its agents can take the disconnect: a restart clears the certificate cache and forces the redial, which status 16 does not do by itself, whatever the circuit state (an open breaker whose cause is on the gateway side does not close by waiting, which differs from the advice in Gateway for a server-side outage). Unless the agents behind the node run a build with the #5183 fix (shipped in 0.15.0), the restart disconnects every agent the node holds and released agents stay wedged and need an agent service restart (Gateway, “What happens when the server restarts” and “Agent dependency”); plan certificate renewals before notAfter, inside a maintenance window. Verify: the two raw counters are flat across reads a minute apart, no new spki= denial line appears, and, if you restarted the node, yuzu_gw_agents_current on it returns to its earlier level.
   • Class C. Placement loss is suspected only when the step 4 checks still fail after the allowances. The observed fix is to restart the agent service on the specific unreachable agents: Gateway, “Dispatch reachability after the session is adopted”, records that this recovers any agent (OBSERVED) and that the notify series carry no agent label, so find the unreachable agents by dispatching a command to them (the placement check in step 4). Fall back to restarting the gateway node only if many agents behind one node are affected, one node at a time and with the #5183 caveat above: unless the agents run a build with that fix, released agents stay wedged afterwards. That fallback is inferred from the code and untested (Gateway records that it was not tested whether the gateway alone recovers stranded agents). Verify: the step 4 checks pass.

If a mixed set of replicas was refusing and admitting at once (an old replica with no peer authorization beside a new one), a call can be admitted by one and refused by another, and a refused call can re-arm a replay: this is expected from reading the gateway code and was not measured. Roll every replica to the same version and pin set before judging recovery.

If the gateway was refused only at registration (for example the server started with the wrong pin and the gateway’s first calls failed), correcting the pin and restarting the server is expected to be enough, because a restarted server reports the gateway’s sessions unknown and a gateway with the #1197 consumer (#5383) replays them (step 4 will show it, within the allowances); an older gateway does not act on that verdict until it is upgraded.

Certificate expiry (the cliff). The guard re-checks the certificate’s validity on every call, but a TLS connection checks it once, at the handshake. A gateway whose long-lived HTTP/2 connection outlives its certificate’s notAfter is therefore refused with outside_validity on every call from that moment, while its connection stays up. Status 16 is an RPC result, so the gateway does not redial by itself: a renewed certificate, over the same key or not, is presented only after the gateway redials. A gateway redials after its connection drops (for example when the server restarts) and reads its certificate and key from the configured files when it does (measured with the gateway’s gun-based upstream client against a local TLS listener on OTP 28; see The upstream client). Renew before notAfter, inside a maintenance window; the gateway’s YuzuGatewayCertificateExpiringSoon / ExpiryImminent / ExpiryCritical alerts on yuzu_gw_cert_expiry_timestamp_seconds{role="upstream_client"} warn 30 days, 7 days and 24 hours ahead (see Certificate rotation and expiry). After replacing the files, check whether the gateway already redialled and allow about 2 minutes for the certificate cache (cause class B above); restart each gateway node, one at a time, only if denials continue and its agents can take the disconnect, because on agents without the #5183 fix (shipped in 0.15.0) released agents stay wedged afterwards. After notAfter the breaker can stay open and waiting does not help, because the cause is on the gateway side. A renewal over the same key needs no server change and no pin change; a new key needs its pin first (see “Rotation”). Neither a connection-age cap nor a gateway-side redial on status 16 exists today. The gateway’s upstream channel holds one connection per endpoint by default, so this is all or nothing, as with grpcbox. If pool_size is raised above 1, the connections redial on their own when they drop, so a certificate change can leave them presenting different certificates for a while and a share of the calls refused rather than all of them (INFERRED from the channel code, not run).

Gateway key custody. The gateway key file is the credential the pin authorizes, so who can read it decides who can act as a gateway. On the generated defaults it is default-gateway.key, group-readable (0640) by the shared --cert-group when that is set. Keep it readable only by the server and gateway processes: do not mount the key-bearing certificate volume into other long-running containers (the reference compose’s one-shot ca-export service below is the one deliberate exception), never widen --cert-group membership beyond those two, and when --cert-group is set while the generated default gateway key is in use (the automatic pin, or gRPC on the default credentials) the server logs a warning at boot that every process in the group holds gateway authority. The reference gateway compose mounts the key-bearing certificate volume into the server and the gateway, and a third time read-only into a one-shot ca-export service, which copies only the public CA certificate (default-ca.pem) into the ca-public volume (it runs with the shared group so it can read the directory, which means it could also read the group-readable gateway key; it runs one install command and exits). The agent container mounts ca-public and never the key-bearing volume. ca-export runs again on every docker compose up, so after a CA re-root prefer docker compose up -d, which re-runs it after the server is healthy, to refresh ca-public (restart also restarts the one-shot container but without that ordering); see PKI architecture. If anything else ever held read access to the gateway key (for example an agent container from an earlier compose file), replace the gateway certificate or move to a gateway-only certificate with a 0600 key pinned explicitly, and withdraw the old pin: changing mounts cannot invalidate a key that was already copied. Production installs should issue a gateway-only certificate and pin it explicitly.

Windows service. The installer rebuilds the service command line from its own state on every run, so a flag appended to the service’s command line by hand is lost at the next upgrade. The durable convention is a pin file named gateway-peer-pin in the installation’s certs folder (%ProgramData%\Yuzu Server\certs): the installer passes --gateway-peer-pin-file for it whenever gateway mode is selected, TLS is not skipped and the file exists.

• Lead with the certificate file: /GATEWAY_PEER_PIN_FILE=<cert.pem>. Give the leaf certificate only: a chain file pins every certificate block in it, not only the leaf. The server computes the SPKI hash itself, so there is no hash to get wrong. Windows operators should use this route; the OpenSSL byte pipeline in “Pinning your own gateway” is not documented for Windows PowerShell 5.1 (its text pipeline may corrupt the binary DER between the two openssl commands).
• Or give one hash: /GATEWAY_PEER_PIN=<64 hex> (the SPKI SHA-256, not a hash of the whole certificate: see “Lockout diagnostic”).
• The two parameters are mutually exclusive; give one. Either one replaces the stored certs\gateway-peer-pin, so rotating a pin with an overlap period needs a multi-entry pin file containing both the old and the new pin.
• The interactive wizard has no pin page. A pin can be given only on the command line; a wizard run that would start gateway mode on your own certificates with no pin is refused before anything is changed, and the message says to supply the parameter on the command line.
• What the installer checks: a /GATEWAY_PEER_PIN value is exactly 64 hexadecimal characters (either case), else it is refused before anything is changed; for /GATEWAY_PEER_PIN_FILE it applies only cheap checks (the file exists, is not empty and contains no NUL byte, as a UTF-16 file does) and does not parse it, so the server’s start-time validation remains the full one.
• /GATEWAY with /NOTLS adds --insecure-gateway-peer and does not pass the pin file, because the server refuses the acknowledgement together with a pin. Giving /GATEWAY /NOTLS together with /GATEWAY_PEER_PIN or /GATEWAY_PEER_PIN_FILE is refused by the installer, with nothing changed: a plaintext server has nothing to pin, and the contradiction would otherwise only surface as a server that refuses to start.
• /GATEWAY is re-read on every run. An upgrade that repeats /GATEWAY on an install carrying your own gRPC certificate and key, with no stored pin, is refused by the installer before anything is changed rather than leaving a service that cannot start. /NOTLS given on a silent install clears any /GRPC_CERT, /GRPC_KEY and /CA_CERT values, as the wizard does, so they are not copied into certs\ where the service arguments would ignore them.
• A refused silent install exits with code 7, and the reason appears only in the setup log (/LOG=<file>) on a line starting PrepareToInstall:. Nothing was changed, and the existing service is left as it was: these refusals run before the service is stopped. Deployment tools that report only the exit code do not show the reason, so deploy with /LOG= and read that file.
• A stale pin file outlives the configuration. If you later move an install back to the generated default certificates, the old certs\gateway-peer-pin still exists, is still passed to the server, and replaces the automatic pin, so the gateway’s generated certificate is refused until you delete the file (del "%ProgramData%\Yuzu Server\certs\gateway-peer-pin") and run the installer again.

Windows service limits. When the service fails to start, the Windows service manager reports error 1053, 7000 or 7009 whatever the cause, so the reason (including a gateway peer refusal at boot) is in <install>\logs\yuzu-server.log, not in the service manager’s message. The installer fixes the gateway-upstream listen address at 0.0.0.0:50055 and does not pass --ca-dir (the server then uses the platform default certificate directory), so two server services on one host collide; this is a limitation, not a parameter.

Proxies in front of the gateway-upstream port. The gateway’s client certificate is what identifies it, so the path from the gateway to the server must be L4 passthrough. A TLS-terminating proxy removes the client certificate, and every call is then refused as not_authenticated.

Observability.

• Metrics: yuzu_server_gateway_peer_denied_total{rpc,reason,event="security"} (every series of the five rpc values by eight reason values exists at 0 from boot; rpc is one of proxy_register, batch_heartbeat, proxy_inventory, notify_stream_status, forward_guardian_message), yuzu_server_gateway_peer_authz_mode{mode} (disabled, enforce, insecure_ack, insecure_ack_tls; 1 for the active mode) and yuzu_server_gateway_peer_denial_audit_suppressed_total; see Metrics.
• Alerts, on the server’s metrics: YuzuGatewayPeerDenied (warning; fires on bad_cert, no_server_auth_eku, not_pinned, outside_validity and internal_error, and not on the three reasons with no usable client certificate, so unauthenticated probes of the agent port do not fire it), YuzuGatewayPeerDeniedBatchHeartbeat (critical; the same reasons on rpc="batch_heartbeat" for 5 minutes, which means the gateway itself is refused: its registrations and stream-status notices fail, while established agent streams to the gateway stay up; a gateway sends BatchHeartbeat only while at least one agent heartbeats through it, because an empty-buffer flush is a no-op, so a gateway with no agents never fires it) and YuzuGatewayPeerAuthzDisabledWithTls (warning; the acknowledgement is active on a TLS server with a client CA). On the gateway’s metrics (scrape the gateway to get them): YuzuGatewayUpstreamAuthRefused (warning; increase(yuzu_gw_upstream_rpc_errors_total{code="16"}[10m]) > 0 for 2 minutes) and YuzuGatewayUpstreamTlsHandshakeFailures (warning; increase(yuzu_gw_upstream_tls_handshake_failures_total[10m]) > 0 for 5 minutes, the signal for a handshake-level lockout, which never reaches the server’s counter; it is a warning although the same cause can be a total outage of that gateway, so treat it as page-worthy where the gateway fronts your agents). insecure_ack has no shipped alert (see “Acknowledged mode”). YuzuGatewayPeerDenied and YuzuGatewayPeerDeniedBatchHeartbeat are active together by design. To page only on the critical one, add an Alertmanager inhibit rule such as the following (operator-side; not shipped):

inhibit_rules:
  - source_matchers: ['alertname="YuzuGatewayPeerDeniedBatchHeartbeat"']
    target_matchers: ['alertname="YuzuGatewayPeerDenied"']
    equal: ['instance', 'reason']

The gateway-side alerts are scraped from the gateway, so their instance is the gateway’s and differs from the server’s: they cannot be inhibited by this rule, and YuzuGatewayUpstreamAuthRefused will fire alongside the server-side pair. The out-of-memory fallback of the refusal path answers CANCELLED, not status 16 (gateway_peer_guard.cpp, denied_status), so YuzuGatewayUpstreamAuthRefused misses that one call: out-of-memory only.

• Audit rows (see Audit Log): session.gateway_peer_denied, written only when the peer’s key hash is known (a peer whose certificate parsed; a bounded sample), and server.gateway_peer_authz_disabled at boot in acknowledged mode. A refusal with no key to attribute (null_context, not_authenticated, no_cert, bad_cert, or an internal_error before the certificate was read) writes no audit row; it is in the counter and the server log only, the same split the OTA no_client_identity rejection makes. null_context, not_authenticated and no_cert are reached by unauthenticated or certificate-less callers, who can produce them in unbounded volume; bad_cert is reached only for a transport-authenticated peer whose certificate does not parse, so there is no key to attribute a row to.
• Denial audit budget: rows are limited per reason and peer key prefix (10 per key per 10-second window; beyond 64 distinct active keys, new keys are charged to one overflow bucket per audit-eligible reason). The refusal is decided first and unconditionally: a budget refusal or an audit write failure never changes it, and the denial counter is the complete count. Rows beyond the budget are counted in yuzu_server_gateway_peer_denial_audit_suppressed_total. A failed audit write has no counter: it is logged and nothing else (see Audit Log, session.gateway_peer_denied). Capacity: the audit write is synchronous on the gRPC handler thread and shares the audit store’s pool, so the budget is also the ceiling on that cost. The worst case is 680 rows per 10-second window per replica (64 tracked keys times 10 rows is 640, plus one <reason>|* overflow bucket of 10 rows for each of the 4 audit-eligible reasons, which is 40), up to 1,360 across a window boundary, and about 5.9 million rows per day per replica if sustained (it needs 64 or more distinct enrolled leaves refused at once). A single leaf produces 86,400 rows per day (10 rows per 10 seconds, 1 row per second). Size the audit retention and the pool for that ceiling, not for the normal rate of zero.
• Logs: a refusal is logged at warning level, rate-limited, as gateway peer denied: rpc=<rpc> reason=<reason> spki=<16 hex> when the peer’s key is known, and gateway peer denied: rpc=<rpc> reason=<reason> peer=<ip> when it is not. A refusal with a known key is logged only while its audit budget (see “Denial audit budget” above) still admits a row for that reason and key; one with no key is logged under a separate per-reason budget with the same limits. The counter is the complete count; the log is a sample. When a window that suppressed rows closes, the next denial also logs one warning, gateway peer: N denial audit rows suppressed in the last window (the denial counter is complete), so a burst is reported when the next denial arrives, not at the end of its window. peer=<ip> is peer=unknown when the transport reports no usable address; for an anonymous caller it is the only attribution there is.
• Not counted: a connection refused during the TLS handshake on a strict port (no certificate, or one the CA did not sign) never reaches the service, so it is in no counter.

Rolling back. See the “Rollback / temporary exception” section of Upgrading.

Known limits of the verification. (1) The guard’s wire tests produce not_authenticated, no_server_auth_eku, not_pinned and outside_validity over a real mutual-TLS connection and null_context by a direct call; no_cert and bad_cert could not be produced over a real gRPC connection in that harness and are covered at the policy-function level only (tests/unit/server/test_gateway_peer_policy.cpp). (2) The integration rig (scripts/integration-test.sh) keeps --insecure-gateway-peer in its --tls branch, so CI never exercises a pinned gateway hop end to end with the real gateway; the guard is exercised by unit tests over a real mutual-TLS harness, and the boot-refusal test presents the default gateway certificate (admitted) and the default server leaf (denied) to the real server binary on its three listeners. (3) The Windows installer change was compiled with Inno Setup 6.7.3 and silent-installed on a Windows 11 test host (34 of 34 silent-install cases at the final tip, including upgrade, refusal and uninstall cases, the refusal of /NOTLS with a pin and the wizard-path refusal wording, plus eight direct runs of the server executable). Starting the service under the Windows service manager could not be tested, because the server binary does not implement the service control protocol (pre-existing; open issue #1835, “Windows server binary has the identical SCM control-protocol defect as #1822 (agent)”, which prevents the installed service from starting under the Windows service manager). The interactive wizard path was not tested. (4) A pin file’s permissions and ownership are not checked by the server (see “Pinning your own gateway”). The decision record, Gateway peer authorization (minimal), 2026-10-07, states these and the TLS session-resumption question in full.

What this does not do. The pin establishes which gateway is calling; it does not restrict which agents a gateway may relay for. A compromised pinned gateway can still assert identities and events for the agents behind it (a documented residual; per-gateway scoping is tracked in #1292). The pin applies to the gateway-upstream service only; direct-connecting agents are not covered by it.

User Management

Yuzu supports two built-in roles for local users:

Creating a User

1. Navigate to Settings > User Management.
2. Enter a username and a password (12–1024 bytes). The form also shows a role selector, but the role is ignored — see below.
3. Click Add User.

The password is hashed with PBKDF2 before storage. Plaintext passwords are never written to disk.

Breaking change in v0.12.0 — the role field is ignored on create. New users are always created as user. To grant admin, call POST /api/settings/users/{username}/role (the Users table has no role button; use Access → Users and roles → Change role or see “Changing a User’s Role” below). This is a deliberate split (security finding C1): collapsing role assignment into the create endpoint allowed a 4xx-on-create + audit-as-success pattern that operators couldn’t audit cleanly. Each role transition now produces a single user.role_change audit event with old_role and new_role recorded in the detail field.

Changing a User’s Role

The Settings → User Management table does not render a role button (the Access → Users and roles page does); change a role with the Settings endpoint from an admin session whose own account role is admin (it is MFA step-up gated):

curl -s -X POST -H "Cookie: yuzu_session=$COOKIE" \
  -H "Origin: https://yuzu.example.com" \
  -H "Content-Type: application/json" -d '{"role":"admin"}' \
  "https://yuzu.example.com/api/settings/users/alice/role"

The server emits an audit event on every branch: