yuzu

Upgrading

This guide covers upgrading Yuzu components (server, agent, gateway) between versions.

Fresh deployment only. 0.15.0 supports initial deployment into a freshly provisioned database; it provides no upgrade path from an earlier build’s database or backups and no return to an earlier build. The server does not enforce this: migrations do not refuse a database written by another build, and an importer that runs at any boot where the database holds no import record for the file still reads leftover enrollment-tokens.cfg and pending-agents.cfg files without checking where they came from. Procedures below that describe upgrading, rolling back or restoring an earlier build’s data are historical and not supported for 0.15.0 (see “Deployment contract (hard cutover)” in the Forensics section of this page).

Behaviour change: the activity-record feature is renamed Echo (tar.db adopted as echo.db; stored references migrated)

What changed. The feature that records endpoint activity was called TAR and is now called Echo, with no aliases for the old names except the upgrade-window shim below. The agent plugin is echo (echo.dll, echo.so, echo.dylib; the Windows installer removes the old tar.dll). REST routes move from /api/v1/tar/... to /api/v1/echo/..., MCP tools from *_tar_* to *_echo_* (for example list_tar_retention_paused is now list_echo_retention_paused, and create_result_set_from_tar_query is now create_result_set_from_echo_query), metrics from yuzu_tar_* to yuzu_echo_*, audit actions from tar.* to echo.*, instruction ids from crossplatform.tar.* to crossplatform.echo.*, the agent heartbeat tags from yuzu.plugin.tar.* to yuzu.plugin.echo.*, and the result-set matcher kind from tar_rows_ge to echo_rows_ge (a request that still sends the old kind to POST /api/v1/result-sets/from-instruction-result or its MCP twin is now refused with an error naming the kind, as is any other unrecognised kind, instead of being accepted into a set that never matches anything). The OpenAPI operationId and tag changes are listed in the OpenAPI note further down this page.

Who is affected. Anything that calls the old routes or tool names, scrapes the old metric names (including the path_class label values tar_dashboard_sql and tar_result_set_sql on yuzu_body_cap_rejected_total and yuzu_server_csrf_denied_total, now echo_dashboard_sql and echo_result_set_sql), filters audit rows by tar.*, or references crossplatform.tar.* instruction ids outside the server’s own stored data. Update scripts, dashboards and alert rules to the new names (the metrics reference, docs/user-manual/metrics.md, lists the yuzu_echo_* series). Audit rows written before the upgrade keep their tar.* action spelling and are never rewritten: an echo.* filter matches both spellings, a tar.* filter matches only the old rows, so a SIEM rule that must see the whole history needs both.

Agents: tar.db becomes echo.db on first start. The first time an upgraded agent starts, before any plugin loads its data, it renames its existing tar.db (and the -journal, -wal and -shm files beside it) to echo.db and re-keys its plugin key-value namespace from tar to echo. Sidecar files move first, so an interrupted adoption simply resumes on the next start. The database keeps its contents and, on Windows, its owner-only DACL. On the next open the plugin renames the tables inside it (tar_state, tar_config and tar_cursor become echo_state, echo_config and echo_cursor) in one transaction and the file reads as schema version 9; the success lines are Adopted legacy ... as ... and EchoDatabase: migrated to schema version 9 (tables renamed). The agent never creates a new, empty echo.db beside an un-adopted tar.db: that would restart Echo with default settings and re-enable sources an operator had paused. Deliver this release’s agent with its package installer: an OTA update replaces the agent binary only, so the renamed plugin (echo.dll, echo.so or echo.dylib) may not be installed yet. While the plugin directory still holds the old tar plugin and no echo plugin, the agent does not load the old plugin (counted in yuzu_agent_plugin_rejected_total{reason="retired_name"}), logs legacy tar plugin still installed; adoption deferred until echo.<ext> is installed, and leaves tar.db and its key-value namespace as they are; the first start after echo.<ext> is installed adopts them. The same holds when the plugin allowlist or the signature check refuses the old plugin before it is opened: the file name tar.<ext> is enough to defer, and that rejection is counted under its own reason (load_failed, signature_missing, signature_untrusted_chain or signature_invalid) rather than retired_name.

Two cases the agent refuses, and what to do.

Server: stored references. This release is a fresh-database deployment (see “Deployment contract (hard cutover)” further down this page), so a new server database holds the renamed vocabulary from the start and the migrations below find nothing to rewrite. Run against a database carried forward from an earlier build, which is not a supported upgrade path, the server does rewrite the stored references to the old names in eleven stores on its first start: instruction definitions, schedules, executions, approvals, device tokens, policies, workflows, responses, plugin configuration (including kill switches such as tar@windows and the enablement rules), persisted result sets and the command outbox. Each of those migrations is idempotent, and there is no return to an earlier server build after that first start. They are serialised by a database advisory lock, so every serving replica must run the new build, as the deployment contract requires: a second replica that starts while the first is still migrating waits up to the connection pool’s 10 s lock timeout, then does not start; start one replica first, let its first start finish, then start the others, which find the work done. The response-store and execution-tracker updates are one sequential pass over the retained response history and the all-time execution history; budget minutes, not seconds, on a multi-million-row store, before starting the other replicas. Operator-authored definitions keep their prose and SQL literals as written; only the plugin and the plugin references in their YAML move (plugin:, requiredPlugins:, and a - tar list item under requiredPlugins: or tags:; a bare - tar under any other key, such as a list of archive formats, is left alone).

Upgrade-window shim. Upgrade the server before the agents. An agent that has not been upgraded still reports the plugin name tar: the server treats it as echo everywhere (presence checks, help, health tags) and writes tar on the wire to that agent, including through the gateway directory route used for high availability, until the agent is upgraded and reconnects. The shim runs in one direction only: an upgraded agent talking to a server that has not been upgraded has no fallback, and every Echo dispatch to it is refused with plugin_not_found.

Behaviour change: a refused or unusable RHEL entitlement directory pauses the licence sync for that host

What changed. This is an agent change on Linux. The license_scan plugin now reads /etc/pki/entitlement in-process instead of running openssl x509. entitlement_certs is an authoritative surface, so an error on it keeps the last good state and skips that day’s software_licensing sync for the whole host: its RPM/dpkg, FlexLM and entitlement rows all stay at their last synced values until the cause is cleared. Three causes newly pause the sync. A directory the agent may not list is now cert_permission_denied (result status PERMISSION_DENIED); before, it read as an empty surface and the daily sync wiped the host’s stored entitlement rows while the other rows kept syncing. A .pem larger than 64 KiB is now a parse failure for that file; the openssl runs parsed any size. A directory or file past the new scan limits also pauses it: more than 4,096 directory entries or 64 certificate files, or more than 16 certificates in one .pem; the scan also stops starting new files after 10 seconds, where it was 60. An unreadable, empty or unparseable .pem already paused the sync before this release.

Who is affected. RHEL-family hosts where the agent account cannot list /etc/pki/entitlement, or where a .pem in it is oversized or junk. A host whose certificates are all readable, parse and are under 64 KiB each sees no pause. Whether the unprivileged agent can read the directory on a real RHEL host has not been measured, so check a RHEL host after the upgrade rather than assuming either answer.

First sync after the upgrade. A Linux host with entitlement certificates sends its full rows once more, because its rows can change: they are now ordered by file name rather than directory order, and a .pem with several certificates yields one row per certificate (up to 16 per file) where the openssl x509 runs read only the first. On macOS a bundle with a binary Info.plist now reports its display name and version where it reported the bundle name and an empty version.

What to do. Run the license_scan plugin’s surfaces action on a RHEL host and read the probe_status|entitlement_certs| line; remove the cause its token names. See When the entitlement directory is refused.

Rollback. Rolling an agent back to an earlier build restores the silent wipe: a directory it cannot list reads as empty again and the next sync replaces the host’s stored entitlement rows with nothing.

New: leader-election metrics and alerts (HA WS-11, #4014)

Additive; no behaviour change and no action is needed to keep a deployment working. Every server with a Postgres-backed elector now exposes seven yuzu_server_leader_* series on /metrics (reference: docs/user-manual/metrics.md, “Leader election metrics”), and docs/prometheus/yuzu-alerts.yml gains the yuzu-ha-leader group: YuzuLeaderSplitBrain, YuzuLeaderNoLeader and YuzuLeaderCoordinationStale (critical) and YuzuLeaderFlapping (warning). Reload your Prometheus rule file to pick them up. The alerts aggregate sum by (job), so Prometheus must scrape each replica individually and unrelated Yuzu deployments sharing one Prometheus need separate jobs. On a single-replica deployment YuzuLeaderNoLeader is the alert that can fire: it means the leader-only background passes are paused while /readyz stays green (leadership is deliberately not a readiness input). Runbook: docs/user-manual/server-admin.md, “Leader election: metrics and alerts (HA WS-11)”. The thresholds are initial claims without fleet data. Two things to know: the leader_elector: acquired leadership ... at epoch N log line now ends with (holder_id=srv-…) (match on the prefix if you alert on that line), and no server up == 0 alert ships, so configure one (these alerts need a successful scrape; see the runbook). If you ever run mixed versions of a multi-replica deployment (unsupported today), reload the rule file only after the last replica is upgraded, or silence YuzuLeaderNoLeader for the window: an old binary exports no yuzu_server_leader_* series, so an upgraded follower’s 0 with an old leader’s absent series sums to zero and the alert fires after 5 minutes. Silence it during planned Postgres maintenance longer than 5 minutes as well, scoped to the job and ending shortly after the window (the silence must expire), then confirm a replica reads yuzu_server_leader_is_leader 1. This does not lift the safe-to-scale gate (docs/ha-delivery-matrix.md); running a second replica is still unsupported.

Behaviour change: device.filesystem.find_by_hash is bounded and reports what it skipped (#5624)

What changed. The agent hashes in-process and stops at the first of 20 seconds, 50,000 entries visited, 4 GiB hashed or 100 matches. The limits are fixed; scan a large tree one subdirectory at a time. When anything was skipped or the search stopped early, a truncated|<reason>|visited=N|... row precedes matches_found|N. Any symlink counts, so most scans have one; scripts must not read it as a match. The result is CONSTRAINED/PARTIAL when a limit was hit, an entry was unreadable, a network mount, UNC share or non-fixed Windows drive was skipped, or the mount table could not be read (a network_mount_guard|unavailable row says so). Symlinks are no longer followed, so a symlink to the target file is not reported; empty files are now hashed. Agents and server may upgrade in either order.

Stored definition. Upgrades keep definition 0.3.0, which declares the columns path, size and matches_found. The rows are unchanged, but the stored definition does not mark a row discriminator, so until it is replaced a column matcher on this instruction matches no device, as it did before (path contains X misses real hits). Check GET /api/v1/instructions/device.filesystem.find_by_hash for "version":"0.4.0". To adopt it, replace the definition after every agent is upgraded, because the 0.4.0 description promises a bounded walk that never follows symlinks, which an agent older than this change does not do (Replacing a stored parameter schema; edit result_schema, not parameter_schema). A replaced bundled definition never receives later corrections. A column matcher saved before the replacement is not rewritten: recreate it afterwards.

registry 2.0.0 changes three existing actions. set_value writes any of six value types (REG_SZ, REG_EXPAND_SZ, REG_BINARY, REG_DWORD, REG_MULTI_SZ, REG_QWORD) from value text or exact data_hex bytes, captures the previous value, reads the new one back on the same handle and fails with verify_mismatch when it differs. It validates its type and numeric text instead of storing every non-REG_DWORD type as REG_SZ: 1.0.0 parsed a REG_DWORD value with stoul, so 0x10 wrote 0, -1 wrote 4294967295 and 12abc wrote 12, abc or an empty value threw out of the plugin, and a type spelt in lower case (reg_dword) stored the text as REG_SZ; 2.0.0 refuses each of those (invalid_data, value_overflow, invalid_type). A creating write whose first key component is missing directly under the hive root is refused root_level_create. delete_value and delete_key fail with tokens (key_not_found, value_not_found, key_has_subkeys, access_denied, …) instead of sentences; delete_key still removes only a key with no subkeys, and on success appends four rows (counted_keys, counted_values, dry_run, capped) after status|ok. All three mutating actions now validate their input before anything is opened and refuse it as invalid_key_path or invalid_value_name: a key path that is empty, longer than 255 bytes or 32 components, has an empty, . or .. component (so a leading, trailing or doubled backslash), begins with a hive alias (HKLM\..., HKEY_LOCAL_MACHINE\...: the hive is its own parameter), or holds a control byte or invalid UTF-8; and a value name longer than 16,383 bytes or holding a control byte or invalid UTF-8 (control bytes are refused on the instruction and MCP routes; POST /api/command passes an embedded NUL to the plugin as a C string, so the value is cut at it). Every mutating action emits one leading audit| row before status|ok or error|<token>, so the first output line of these actions is now the audit row once the target resolves; a missing or unparseable hive or key is a bare error| line with no audit row. Registry links are refused, not followed: 1.0.0 followed them, so every set_value, delete_value and delete_key under HKLM\SYSTEM\CurrentControlSet (itself a link to the active control set) now answers error|key_is_link; reads still follow it. The read actions (get_value, enumerate_keys, enumerate_values, get_user_value) map |, CR and LF in registry data to _.

2.0.1: registry 2.0.1 bounds get_value: it reads at most 1 MiB (the bound get_user_value and the write path already use), and a larger value answers error|value exceeds 1 MiB limit with rc 1 instead of being read whole into the agent’s memory. A get_value whose value kept growing through the bounded re-reads now answers error|value changed while reading ... and any other failed read error|win32:<code>; only a missing value still answers error|value not found. 2.0.1 also widens the fold of the read actions and of every plugin row rendered through the shared sanitize_field: beside the |, CR and LF folded since 2.0.0, every other control character except TAB (ESC, VT, FF, FS, GS, RS and an interior NUL among them), DEL, the C1 controls U+0080-U+009F (NEL, CSI and OSC among them) and U+2028 and U+2029 each become _, and trailing NULs are dropped, so an embedded NUL in a name or in get_value data no longer cuts a row short (get_user_value and list_profiles read a string value up to its first NUL, as Windows does). The rows that gain the fold are the registry read actions (list_profiles and get_user_value included), the hive-access lines of installed_apps and autoruns, and the REG_MULTI_SZ and REG_LINK values that autoruns and the execution_artifacts AmCache walk read through read_reg_value. The rendered text is display text: the folded characters cannot be recovered from it. The registry.write audit tokens already folded the C0 controls and DEL and gain the C1 controls, U+2028 and U+2029. Affected: callers that read a value over 1 MiB, callers that match the error|value not found line for every failed get_value, and callers that compare a rendered name or value with the raw registry text. An agent with registry 2.0.0 keeps the unbounded read and the narrower fold until it is upgraded.

registry plugin 2.1.0: per-user writes, apply and recursive delete_key (additive, with a mixed-fleet hazard)

registry 2.1.0 adds three actions: set_user_value and delete_user_value write or delete one value in a user’s hive (the live HKEY_USERS\<SID> hive, or an offline NTUSER.DAT behind the hive-file guard; read back, and on the offline arm flushed before the unload), and apply runs a declarative plan of up to 256 HKLM set_value / delete_value / create_key / delete_key items with a dry run by default and an undo manifest in the reply (nothing is persisted on the endpoint). delete_key gains two optional parameters, recursive and dry_run; without them it behaves exactly as 2.0.1. The server’s audit binding now treats six registry actions as mutating, so the three new ones mint registry.write rows too: apply one per plan item it attempts, and a dry run mints success rows with dry_run=1 (a reader filtering result=success must also read dry_run). An apply of 256 items is up to 256 rows, so three of them inside one minute on one agent exceed the server’s 600 rows per agent per minute and the later rows are suppressed (counted and logged, as for any frame past the budget).

⚠️ Breaking: the runtime-configuration read shows a caller only the overrides it could write

What changed. GET /api/config (and the new GET /api/v1/config and MCP tool get_runtime_config) leave out the stored override of every key the caller could not write with PUT /api/config/{key}: the oidc_* keys, plugin_signing_required, the DEX alert keys and audit_retention_days, plus the effective audit_retention_days. An administrator account’s GET /api/config body is what it was, apart from the log_level spelling below, so the Settings pages are unaffected. An MCP-tier token, a non-administrator role holding Infrastructure:Read, and (for the oidc_* keys) an elevated-only session now get a smaller overrides object. Before, they received every override, including oidc_admin_group. The legacy route does not say what it left out; GET /api/v1/config returns withheld_keys.

Also in this release. heartbeat_timeout must now be 30 to 86400 seconds on every write route (the upper bound is new), a stored value outside that range is moved to the nearest bound at server start, and the read reports log_level as warn (the spelling the write accepts) instead of warning. See REST API: Runtime Configuration.

What to do. Read the configuration with an administrator account or an administrator API token that is not MCP-tier if a script needs the administrator-only overrides.

Behaviour change: a Guaranteed State delta push no longer drops the retry obligation of a rule it omits (#5547)

What changed. This is an agent change. The agent holds an acknowledgment application per policy push and reports a policy generation only once that application has no unresolved arm. Before this release every push replaced the open application, so a delta push (full_sync=false) that named only some rules forgot the pending arm, retained wedge or drained failure of every rule it left out. The generation could then be acknowledged while an omitted rule was still unarmed, with nothing left to retry it. Now a delta carries the unresolved outcome of each rule it does not name into the new application, and the following differ:

Who notices. Operators who push deltas through the REST or MCP endpoints, and anyone watching yuzu.guardian_arm_failed. A push to either endpoint without full_sync is a delta, because full_sync defaults to false. The dashboard’s Baseline deploy, the delete of a deployed Baseline, and the heartbeat reconcile of an agent that trails the server’s generation send full pushes and are not affected. The gauge used to reset on every replacement; it can now hold at a nonzero value across omitting deltas until a full push or a push naming the rule replaces it (a wedge whose hung call is still outstanding is re-observed and counted again). That includes an entry that no longer reflects the rule’s true state (a wedge adopted or settled before the agent retained it, the flip gate’s AC-15), so a sustained 1 can persist for a rule that has since armed. The wedge suppress budget has always restarted on every replacement; what is new is that a carried hold survives the replacement, so alternating deltas and full pushes leave a hold with no elapsed-time bound. See Guaranteed State and the Spark flip gate (AC-11, AC-15).

What you will see.

Restart and rollback. The carried state lives in agent memory only. Restarting the agent discards it, but does not by itself repair what caused it. The agent re-arms its cached rules at boot and a rule that still cannot arm is recorded again, in which case it reports generation 0 with the companion tag yuzu.guardian_boot_rearm_unresolved rather than its persisted generation. A rule that was never persisted (a persist failure on the push) is not among the cached rules, and no reconcile follows while the agent’s generation equals the server’s, so send a full push for it. If the store fault that stopped the persist is still present, that full push fails partway too (see “No automatic full push follows” above), so fix the store first. Rolling an agent back to an earlier build restores the behaviour this change removes: an omitting delta forgets the unresolved rule and the generation can be acknowledged while the rule is unarmed. An agent on the earlier build also resets its arm counts on every replacement, so a falling yuzu_fleet_guardian_arm_failed after a rollback is not evidence of repair; compare agent versions. No data change is involved.

Rolling upgrades. Agents that have not been upgraded keep the previous behaviour, and the fleet gauges sum agents that behave differently until all are upgraded. No server configuration or data migration is required; the server change is the metric HELP text.

⚠️ Breaking: seven runtime-config keys are administrator-only on PUT /api/config/{key}

What changed. PUT /api/config/{key} for plugin_signing_required, dex_alert_routing, dex_blast_min_devices, dex_blast_window_seconds, dex_blast_cooldown_seconds and dex_cohort_export_key now applies the same caller rule as the Settings routes that also write them, which are administrator-only. audit_retention_days joins them by design, for a different reason: it is the stored audit-retention setting (reported by GET /api/config, evidence-bearing, and the lever that will govern enforcement once #483 lands). Today a stored value is stored only and is not enforced (see “A stored audit_retention_days is not what the server enforces” below). It has no Settings route at all, so PUT /api/config/audit_retention_days is its only writer and it follows the same caller rule rather than mirroring a Settings route. It is administrator-class while response_retention_days and guardian_event_retention_days stay on Infrastructure:Write because it is audit evidence and the other two are operational data; all three are equally stored only today, and the choice should be revisited if any retention store gains a setter. A session with an MCP-tier token or a service-scoped token, or whose effective role is not admin (an active temporary elevation counts as admin), gets 403. The role is the account’s legacy role or a live temporary elevation, never an RBAC grant: an RBAC Administrator grant alone does not count. An engine principal is never admitted either. The refusal happens before the runtime-config store is opened and before the request body is parsed, nothing is stored, and the audit log records config.update with result denied, target_type RuntimeConfig, target_id the key and no preceding attempted row. detail is token_session_admin_setting_blocked (an MCP-tier token; in production only a supervised-tier one reaches it) or admin_role_required_setting_blocked (effective role not admin). The 403 body carries a remediation (“change this setting with an administrator API token that is not MCP-tier, or as an administrator from the Settings page where one exists”) and no permission field. Before this change the route admitted these keys under Infrastructure:Write alone.

The oidc_* rule is unchanged (your own role must be admin; an elevation and an MCP-tier token are refused; the handler also refuses a service-scoped token, as defence in depth, because the permission gate already does). Only the caller rule is shared with the Settings routes: the generic route still validates less than they do (they validate and clamp DEX values and tag keys and apply them live), so a value this route accepts may still be rejected or clamped when saved through Settings. The shape of GET /api/config is unchanged by this note (the read-scoping change is the first note above). Every other key stays writable under Infrastructure:Write, including response_retention_days, guardian_event_retention_days, log_level, heartbeat_timeout and auto_approve_enabled, and that includes a supervised-tier MCP token whose creator is an administrator. That is deliberate for now, pending a maintainer ruling.

Who is affected.

How to detect. After the upgrade, search the audit log for action config.update with result denied: GET /api/v1/audit?action=config.update&target_type=RuntimeConfig&target_id=<key> for one key (target_id requires target_type), or drop target_id and keep rows whose result is denied. The detail values above say which rule fired. The Settings routes audit their denials as auth.admin_required, so a SIEM rule keyed on auth.admin_required will not see a refusal on this route: add config.update with result denied.

Review stored values. Before this change a supervised-tier MCP token could write these seven keys. Stored values persist across the upgrade, and the dex_* ones are re-applied when the server starts; plugin_signing_required only records intent (see REST API, “records intent only”). To check what is stored now:

  1. GET /api/config with an administrator account (any other caller does not receive these keys) and read updated_by and updated_at for each of the seven keys under overrides. This includes audit_retention_days.
  2. Query the audit log for each key as above and read the success rows (detail is value=<new value>). For audit_retention_days use GET /api/v1/audit?action=config.update&target_type=RuntimeConfig&target_id=audit_retention_days.

Two limits. The stored audit events carry principal, principal_role, principal_class (agent for bearer tokens), session_id, source_ip and user_agent, but not the MCP tier, and mcp_tool is empty for REST calls, so a past MCP-tier write cannot be told apart from an administrator API-token write. GET /api/v1/audit returns only id, timestamp, principal, action, result, target_type, target_id and detail; principal_class and session_id are present only in the stored rows (the legacy GET /api/audit adds principal_role and source_ip). And rows older than your enforced audit retention window (--audit-retention-days, not a stored audit_retention_days) may have been pruned. If a value is suspect, set the key to the value you intend from the Settings page (DEX alerts, Plugin Code Signing) and read it back with GET /api/config. audit_retention_days has no Settings page: replace a suspect stored value with PUT /api/config/audit_retention_days using an administrator account (an administrator API token that is not MCP-tier, or an administrator session). Saving the DEX blast form writes all three dex_blast_* keys at once. For plugin_signing_required, toggle the plugin-signing checkbox: it records intent only and does no further validation. The updated_by of a Settings write is the username for the DEX forms and the literal ui for the plugin-signing checkbox (the clear-trust-bundle action records system).

A stored audit_retention_days is not what the server enforces. The audit store copies its retention window from --audit-retention-days (or YUZU_AUDIT_RETENTION_DAYS; the default is 365 and there is no enforced minimum, so 365 is a default, not a floor) when it is constructed, and the server applies stored runtime-config overrides afterwards, so at boot the stored value only updates the in-memory config that GET /api/config reports. As the code is ordered, a low stored value did not by itself make the server prune audit events earlier, and a restart does not make it take effect. Read the window actually enforced from enforced in GET /api/v1/settings/data-retention, and change it with the flag or environment variable and a restart. To find out whether audit events were in fact pruned, check the retention-pass log lines and the oldest audit row, not the stored value.

What to do. Change these seven settings with an administrator account (the account’s own role must be admin; an RBAC Administrator grant alone does not count): through the Settings pages for the six that have one, and for audit_retention_days (no Settings page) with PUT /api/config/audit_retention_days and an administrator API token that is not an MCP-tier token, or an administrator session. A cookie session is a browser write and needs a same-site Origin or Referer header (otherwise 403 and a csrf.denied audit row); an administrator API token that is not MCP-tier is exempt from that check. PUT only stores the audit_retention_days value: the window the server enforces is --audit-retention-days (or YUZU_AUDIT_RETENTION_DAYS) plus a restart, so check enforced in GET /api/v1/settings/data-retention. When #483 lands, every previously stored value becomes live at the next boot, so reviewing the stored values above is a precondition of that upgrade. response_retention_days, guardian_event_retention_days, log_level and heartbeat_timeout stay on Infrastructure:Write for now, so automation that writes only those needs no change.

Rolling upgrades and rollback. The check is enforced per server replica. During a mixed-version roll the same call can return 200 on a replica that has not been upgraded and 403 on one that has, so finish the roll before relying on the restriction. Rolling a replica back reopens it. No data changes in either direction.

Config-as-code replay. A caller that is not an administrator account (for example an MCP-tier token) no longer reads the administrator-only overrides at all: GET /api/config leaves them out of overrides, so a backup taken with such a token silently lacks them, and a reconcile client that treats a missing key as “unset” sees them as unset. Writing one of those keys back with that caller still stops with 403, so keys written before that point stay applied and the rest are not. GET /api/v1/config lists the withheld keys in withheld_keys: treat each as indeterminate, not unset. Take the backup, and replay, with an administrator API token that is not MCP-tier, or write the seven keys separately (the Settings pages for the six that have one, PUT /api/config/audit_retention_days with an administrator account for the seventh).

Behaviour change: the sessions table rejects an inconsistent principal class (#2343)

What changed. The durable sessions table (schema session_store) gains a CHECK that allows exactly two pairings: a human session (any source except engine_token) or an engine session (engine_token). The migration runs automatically at boot. Every decision, gate and audit-class site that reads a session now classifies an engine with the same Session::is_engine() rule.

Who is affected. Nobody in normal operation: the server writes only human sessions from a local, OIDC or SAML sign-in, and API, MCP and engine token sessions are never stored. Only a session row edited by hand into another pairing is affected: the migration deletes it, logs a WARNING in the Postgres server log (session_store v2 (#2343): deleted N inconsistent session row(s); with the default libpq notice handling it also appears, prefixed WARNING:, on the server process’s standard error, so check the journal there if the Postgres log is not available to you), and that principal signs in again. The migration also bumps the session write generation in that case, so other replicas drop their cached copy of the deleted rows within about a second. A clean upgrade deletes nothing and bumps nothing.

What to do. No action is needed, with one precondition. Do not start the upgrade while pg_dump or another long transaction is running: the migration takes a brief exclusive lock on sessions and waits at most 3 seconds to acquire it; once held, the migration scans the table once, and the table normally holds only a few dozen rows, so the hold is brief. If it cannot get the lock the migration rolls back completely and the server refuses to start, logging canceling statement due to lock timeout on the preceding PgMigrationRunner line; start it again once the transaction ends. While it waits, sign-ins and session checks on replicas that are still running pause for up to those 3 seconds, plus the time the migration then holds the lock. An older binary can still run against the upgraded database.

Behaviour change: engine principals are default-deny when RBAC is off

What changed. An engine principal’s only authority is an explicit RBAC grant. With RBAC off, the legacy-open fallback that admits every authenticated human no longer applies to an engine credential: GET /api/v1/upload-grants answers it 403 (503 when the RBAC store is unavailable), and the shared per-agent list reads, device visibility, scheduled arming and dispatch-permission checks return no agents or refuse it instead of treating it as legacy-open.

Who is affected. Only deployments whose engine credentials ran with RBAC disabled (the default) and relied on that fallback. Human sessions are unchanged.

What to do. Enable RBAC and assign the engine principal the grants it needs (for example UploadGrant:Read to list upload grants). Enabling RBAC applies to every operator, so assign the human roles first and follow the prerequisites in Role-Based Access Control.

⚠️ Breaking: a management-group-confined operator, or one with no Inventory grant, is refused the unfiltered issued-certificate list

What changed. GET /api/v1/ca/issued without a filter parameter, the MCP tool list_issued_certs without filter arguments, and the issued-certificate list in the Settings certificate panel now apply the confinement check the filtered form already used. A caller whose Inventory:Read is granted only for management groups (a confined view), or who holds no Inventory:Read grant at all, gets 403 (MCP: permission denied) with no permission named. In the Settings panel only the issued-certificate list is withheld, with a note in its place; the CA details (trust mode, algorithm, validity, root fingerprint), the root, signing-request and CRL downloads and the import form are still shown, and a revoke or an import answers the panel with that note (and, after a success, a confirmation banner and toast) instead of a blank body. If the check reports itself unavailable the answer is 503 with retry_after_ms: 5000. Both forms of the route and the tool are now audited crypto.ca.certificates.view (set-and-proceed).

Who is affected. An operator or agentic worker whose role grants global Security:Read but no global Inventory:Read (whether Inventory:Read is granted only through management groups or not at all, as for a custom role that holds Security:Read alone), and who listed the CA inventory; and anyone who relied on that call being unaudited. The seeded Viewer and Administrator roles hold a global Inventory:Read, so they are not confined and are not affected. To find out whether anyone is: after upgrading, query GET /api/v1/audit?action=crypto.ca.certificates.view (the route has no filter on result, so read the result field of each row) and look for denied rows; their principal is each refused caller. An agentic worker that calls list_issued_certs and already treats a permission-denied error as an error needs no change: the tool’s input schema is unchanged, and an unfiltered call can now also carry audit_persisted: false (a marker the filtered form already had) when its audit row could not be written.

What did not change, stated plainly.

What to check.

Behaviour change: every fleet-gated read answers 503, not 403, when its authorization read fails

What changed. The shared list-read check (RbacStore::authorize_list_read) used to answer the same “deny” for a caller who holds no grant and for a caller whose permission, permission-group or management-group read FAILED, and every fleet-gated route turned that into a 403. It now reports a failed read separately, and the fleet-read gate and the list-read gate answer a retryable 503 (retry_after_ms: 5000, no permission named) with an audit row result=failure (auth.fleet_read_required or auth.permission_required, detail ... authorization read failed ...) instead. An engine principal whose grant read fails gets the same 503, and so does one whose RBAC enforcement state cannot be confirmed (for example a replica whose cached “disabled” view is stale because its refresh keeps failing past the 5 second bound, so another replica may have enabled RBAC), on these two gates only; it used to get a terminal 403 there. A fresh “disabled” view is still 403, and the plain require_permission gates keep their 403 (#6437). A management-group store that is not wired or not open now also reads as unavailable (503) on require_list_read routes, as it already did on the fleet-read gate. Both answers still deny.

Who is affected. A client of any fleet-gated or list-gated read (status and drill-down views, plugin configuration lists, inventory, executions, responses, devices, hardware, enrollment, result sets and others) that treats 403 as terminal will now see 503 on a transient database fault and should retry after the interval. Dashboard fragments already handle 503 as “unavailable”. The two upload-grants list resolvers do not carry the new answer: a failed read there is still a plain deny.

What to check. Alerts or dashboards that count 403s on these routes see fewer during an outage; the root-cause signals are the RBAC and management-group read-degrade counters and their alerts (YuzuRbacReadDegraded, YuzuRbacBreakerOpen, YuzuMgmtGroupReadDegraded). A SIEM rule that counts auth.fleet_read_required rows with result=denied as authorization denials is unaffected; the new rows are failure.

Breaker and counter behaviour. A failed role lookup inside any cache-missing permission check (check_permission, which every require_permission, fleet and list-read gate uses, not only the CA reads) used to read as “no roles” and was recorded as a breaker success (it reset the failure streak) with no degrade count. It now counts as a failed read: it increments yuzu_server_rbac_read_degrade_total{reason="query_error"} and moves the RBAC fail-fast breaker (yuzu_server_rbac_breaker_open, opens after 2 consecutive failures), so YuzuRbacReadDegraded and YuzuRbacBreakerOpen can fire for an outage they were silent about before, and two consecutive failures on a replica open its breaker, which sheds that replica’s cache-missing authorization reads for the cooldown. A caller with no roles at all counts as a success: it adds no degrade count and does not count toward opening the breaker.

Scope. The 503 applies to the list-read and fleet-read checks. The front Security:Read gate that precedes them on the CA routes, and the other require_permission call sites, still answer 403 when their own first read fails (their failed reads now count toward the breaker and the degrade counter as described above) (deny-on-degrade, ADR-0041); that is unchanged and is tracked as #6437.

No action on upgrade. There is no schema change.

Behaviour change: engine credential mint, rotate and the no-admin auditor fail closed on a degraded read

What changed. The engine credential routes POST /api/v1/engine-principals/{id}/credentials and .../credentials/rotate, and the MCP tools mint_engine_credential and rotate_engine_credential, now read the token store through checked reads. If the check for an existing credential cannot be answered, nothing is minted and the caller gets 503 (REST, Retry-After: 2) or a retryable error (MCP, retry_after_ms). If a mint or rotation commits but the new credential cannot be read back, the secret is withheld and the audit row is partial; retry a rotation within about two minutes against the same replica to receive the same secret, or recover as described in Engine Principals. The no-admin auditor (GET /api/v1/engine-principals/audit/no-admin, audit_engine_no_admin) answers 503 / a retryable error instead of ok:true when a read it needs fails. The auditor also now reads grants by principal id, so it reports grants held by user-typed rows for engine: ids (invisible before), reports role by name (a single role that covers the whole grid was reported with an empty role before; only a union-only finding still has ""), and records scope=all|active in its success audit detail; a script that gates on ok or parses role should be checked. On the MCP tool, include_revoked=false skips only principals the roster lists as revoked and also reports a grant whose principal has no roster row as orphan_grant (the default run does not), so it is not a subset of the default run. A mint whose read-back answers but finds no active credential is now the same 503 with the secret withheld and a partial audit row, and a rotation whose read-back finds no linked successor answers 503 with no retry hint.

No action on upgrade. There is no schema change. A client that treated a 200 with an empty token_id as success will now see the 503, and should retry as described above.

Behaviour change: engine principal list and get answer 503 instead of an empty result on a degraded read

What changed. GET /api/v1/engine-principals, GET /api/v1/engine-principals/{id}, the MCP tools list_engine_principals and get_engine_principal, and the Settings Engine Principals panel used to turn a database connection pool timeout or query failure into “no engine principals”, active_credential_count: 0 or an empty active_credentials list. They now answer 503 with Retry-After: 2 and retry_after_ms: 2000 (MCP: a retryable internal error with retry_after_ms), write an engine_principal.list or engine_principal.get audit row with result=failure, and the Settings panel shows “unavailable” with a Retry button.

Who is affected. Clients that parsed an empty 200 as “nothing exists” during a database incident, and alerting that counts result=failure rows for these two verbs. A healthy server answers exactly as before.

No action on upgrade. There is no schema change. Retry after the Retry-After interval.

Behaviour change: every Echo capture source is on by default; stored opt-outs reset once (echo.db schema v8)

What changed. All 16 Echo capture sources now ship enabled on every OS. Eight of them were off by default before: procperf, netqual, module, software, arp, dns, netconn and mapdrive. A fresh agent now reports <source>_enabled|true in echo.status for all of them.

Before you upgrade, if you paused any of the eight on purpose (for example to preserve evidence): run echo.status on that agent and record its config|<source>_enabled|false lines (and the matching config|<source>_paused_at|<epoch> lines), and export what you need with echo.sql, which reaches every table and every tier (echo.query and echo.export read only the live tables, and of these eight only software, arp, dns and mapdrive have one). The reset re-enables the source, and retention skips only disabled sources, so the exemption ends with it: from the first rollup pass after the upgrade (rollups run every 15 minutes) rows older than their tier cutoff start being pruned, and that cannot be undone. After the upgrade, re-apply <source>_enabled=false with echo.configure for each source you recorded to pause it again: that stops collection and exempts the rows that remain from pruning, but rows already pruned do not come back.

One-time reset on upgrade. The first time an upgraded agent starts, it migrates its echo.db to schema version 8. For each of the eight sources above whose stored <source>_enabled value is false (including a value an operator set deliberately), the migration sets it to true and clears the source’s <source>_paused_at marker, in one transaction. A value of errored (any stored value other than exactly true or false) is left alone, as are the stored values of the eight sources that were already on (process, tcp, service, user, perf, power, removable, usage). The success line in the agent log is EchoDatabase: migrated to schema version 8; an opt-out written after it sticks. If any step fails, the agent logs EchoDatabase: v8 migration failed at followed by the failing step, rolls the whole migration back and retries it on the next start; until a start succeeds the old stored values stay in force, and a disable written for one of the eight in that window is reset by the retry, so look for the success line before relying on one. On Windows the first open after this upgrade also re-secures echo.db and its sidecars (-wal, -shm, -journal) to a protected owner-only DACL, and refuses to open an echo.db owned by an account other than SYSTEM, Administrators or the agent’s, or carrying more than one hard link (the owner test needs a volume with ACLs; where the owner cannot be read, as on FAT or exFAT, nothing is refused on that ground). For a foreign-owned or hard-linked file the log names the file and why. For a foreign-owned echo.db, move it and its sidecars aside or remove them and restart (the history in them is lost); if the owner SID the log names is a former agent account’s and all four files carry that same owner, an administrator can instead take ownership of the four files and grant the agent account full access (this sequence has not yet been verified on a Windows host). If the agent’s account changed from LocalSystem or an administrator account to a non-administrator one, no owner is named: the log shows could not open <file> to secure it (error 5) and then failed to open echo.db, and an administrator must grant the new account full access to the four files or move them aside. For a hard link, remove the other link. The same checks apply to the echo.db this upgrade adopts or finds beside an un-adopted tar.db (see “Both tar.db and echo.db exist” in the Echo rename section at the top of this page).

Opt-outs after the upgrade stick. The reset runs once. A <source>_enabled=false written through echo.configure after the migration has succeeded (look for the success line) is kept across restarts and further upgrades.

What to expect. New collection begins without operator action, on each OS where the source has a collector (OS compatibility matrix in docs/user-manual/echo.md), for procperf (and the daily per-application performance sync it feeds), netqual, module (Windows ETW), software, arp, dns, netconn and mapdrive (netconn and mapdrive also run history backfills; netconn_lookback_seconds bounds the first). A source whose <source>_enabled row cannot be read is treated as disabled for that tick and its stored rows are kept.

⚠️ Breaking: MFA step-up is checked once, before the route’s own checks, and two more routes need a fresh proof

A fresh MFA proof is required on 37 named high-risk routes (the list is in REST API, which also names the three JIT-elevation routes) for MFA-enrolled local accounts and OIDC sessions per --mfa-enforcement; un-enrolled accounts and API-token and MCP-token callers pass, and engine principals are denied. “High-risk” names these routes; it is not a claim that every high-risk operation is covered. Other mutating routes and all MCP tools are not MFA-step-up gated in this release. The check on those 37 routes now runs in one place, in the server’s request pipeline, instead of inside each handler, and two of the 37 are existing routes that are newly gated; six more are routes that are new in this release (see below). The proof each caller needs and the step-up window are unchanged. The three JIT-elevation routes keep their own handler gate. What an operator or script can notice:

The same test suite also classifies every non-Read MCP tool in a second table and fails on a missing or stale row; that table is data only and changes no MCP behaviour.

Action. None required for a stock deployment. Check any script that signs in with a session cookie and calls one of the 37 routes: it may now see 401 where it saw 403. Measured from v0.14.1, six existing routes that needed no fresh proof now need one inside the window: the two Settings MFA routes (gated by this change) and the four existing routes listed under “What is newly gated” above (gated earlier in this release cycle). The six routes that are new in this release need one too. --mfa-step-up-window-secs 0 (or less) disables the stale-proof requirement on the 37 routes and still logs a startup WARN. While it is in force no stale proof is refused (so none is recorded), and the SAML refusal and the authentication-store read are skipped too, so SAML sessions also pass; engine denials and unwired-store refusals still apply and are still recorded. The JIT-elevation routes keep a 300 s window (the global window when it is positive, else 300 s). The new alert YuzuStepUpWindowDisabled fires while the window is 0 or less. This release has no supported return to v0.14.1 (see “Deployment contract (hard cutover)”). v0.14.1 does not gate the two Settings routes or the four existing routes listed under “What is newly gated” above in their handlers, and it does not have the six routes that are new in this release. A server whose /metrics lacks yuzu_auth_mfa_step_up_gate_total and yuzu_auth_mfa_step_up_window_seconds has no step-up gate on these routes at all, and YuzuStepUpGateUnavailable and YuzuStepUpWindowDisabled cannot fire there: their silence is not evidence that the gate is on.

Behaviour change: /health and /api/health per-source budget

/health and /api/health now apply a per-source budget to requests that carry a credential header (Cookie, Authorization or X-Yuzu-Token):

⚠️ Breaking: the OpenAPI document is larger and typed, every operation has an operationId, and there is a /swagger page

What changed. GET /api/v1/openapi.json now gives 262 of its 264 operations a success response with a content block (the previous release’s document listed 241 operations and did so for 14), and every operation carries an operationId (the earlier document had none). The document also lists 23 operations that the previous release’s document did not, among them GET /config, PUT /config/{key} and GET, PUT and DELETE /dex/score-policy; all 23 are typed. The default form of the document, which is also what the MCP resource yuzu://openapi serves, grew from about 314 KB to about 670 KB as served. A second form, GET /api/v1/openapi.json?guides=true (the value must be exactly true or 1), adds a prose guide to each of the 47 tag groups. The MCP resource and the discover tools never serve the guide form, short of a build defect. A new dashboard page, /swagger, renders the guide form in the browser. Both the route and the page need Infrastructure:Read, and the page has no navigation entry. The default form is re-serialised by the server, so its key order and whitespace differ from the guide form.

Who is affected. Anyone who generates a client from the document (typed return types, renamed methods and a changed security-scheme model can break generated code), anyone who compares or hash-pins the document byte for byte, and any agentic worker that reads yuzu://openapi into its context.

What to check.

Correction: webhook subscriptions to command.completed and policy.violation have never delivered

What changed. Nothing in the server: the REST API reference listed command.completed and policy.violation as webhook events, and the server has never sent either. It sends only agent.registered, execution.completed, dex.signal and dex.blast_radius (offload targets match the same events). The reference now says so.

Who is affected. Anyone who subscribed a webhook to command.completed or policy.violation. A subscription that lists only those types has never delivered anything, so anyone who relied on a policy.violation webhook for compliance alerting has had no alert from it since the subscription was created.

What to check. execution.completed is the sent event that stands in for command.completed. There is no policy-violation event to subscribe to: poll GET /api/v1/compliance (needs Policy:Read; the MCP twin is get_fleet_compliance) for the fleet’s compliant and non-compliant counts, or GET /api/v1/compliance/{id} for one policy, and review any subscription that lists only the two unsent types.

⚠️ Breaking: the guardian. definition-id prefix is reserved; approvals gain frozen-request columns; a new audit action and metric (#6062)

This release lands the storage and redemption half of the four-eyes bound approval primitive (ADR-1009, epic #6060). Nothing in this release creates a bound approval ticket and there is no Guardian deploy gate yet; those arrive in later changes (#6063 onwards). What an operator can see today is the three items below, plus a fourth refusal that normal operation never triggers.

1. The guardian. definition-id prefix is reserved (breaking for operator-authored ids). A new instruction definition whose id starts guardian. is refused on create, import and the dashboard YAML editor, and such a definition is skipped at boot auto-import. No shipped content uses the prefix. Definitions that already carry such an id keep executing, but cannot be saved from the YAML editor if the document declares metadata.id, and cannot be re-imported. Before upgrading, run the read-only check and rename anything it lists: see the guardian. entry in Server Administration.

2. Approvals schema migration v3 (store approval_manager). It adds three text columns to approvals (request_json, request_digest, excluded_roots), each NOT NULL DEFAULT '', so every existing row reads as “binds no request, excludes nobody” and behaves as before. The migration is additive and needs no operator step, but it takes a brief ACCESS EXCLUSIVE lock on approvals. If a long-running or idle-in-transaction session holds that table, the upgraded replica cannot finish the migration and refuses to start (Refusing to start: approval manager migration/open failed) rather than serve with a half-migrated schema; end the session and start it again. It follows the data-only v2 step of the scheduled-approval change (#6096, below), which claimed v2 first, so a database upgraded from a build with neither change runs v2 and then v3 in two separate transactions, and one already at v2 runs only v3. The v3 step is idempotent (ADD COLUMN IF NOT EXISTS), so a development or UAT database that ran an earlier build of this branch, which recorded the same columns as version 2, upgrades cleanly instead of failing on a column that already exists.

3. A new audit action and metric: approval.precondition_denied and yuzu_approval_precondition_denied_total{origin}. When an approved ticket is refused at redemption because the state its effect depends on has changed, the approval store now counts the denial and writes an approval.precondition_denied audit row (result=denied; see the audit log reference). The only redemptions that supply such a check today are MCP recalls (every approval-gated tool checks that the ticket still matches the recalled tool and arguments; confirm_engine_rotation also checks the rotation’s state), so for now the new series moves only with origin="mcp"; the other three origin values are present at 0 until their producers exist. For SIEM and alert owners: add approval.precondition_denied to any audit-action allow-list. An MCP recall refused this way still writes its own mcp.* denied row as before, so one MCP event now leaves two rows: do not sum the two when counting denials. The older yuzu_mcp_approval_precondition_denied_total{tool} series keeps its meaning but is now pre-seeded, and can move, for every approval-gated tool rather than only confirm_engine_rotation.

4. A redemption integrity re-check, and a row-moved refusal. Redeeming an approved, unconsumed ticket now re-checks the stored row. This applies to every MCP recall of an approval-gated tool (the ticket flow that runs before the tool for a supervised-tier token, for example execute_instruction, quarantine_device, revoke_certificate and confirm_engine_rotation), not only to the one that supplies a precondition. A ticket is refused if its reviewer is the submitter or an excluded root, it records no reviewer, its excluded roots cannot be read, or its stored request does not match its digest. Through MCP it surfaces as the same “approval already used (one-time ticket)” refusal a replay gets, and the audit row reads refused: foreign_origin; the cause is only in the server log. Approving a ticket always records a distinct reviewer, so ordinary operation never triggers this: it indicates a row edited by hand, restored from a backup or written by an older build, and such a ticket stays unredeemable until its seven-day lifetime ends. Separately, a ticket whose row was rewritten by another transaction between the check and the consuming statement (and, rarely, VACUUM FULL or CLUSTER running at that moment, which move the row) is also refused, audited as refused: not_consumable. Both read like a replay in the counter and the audit trail. The second is a false refusal only when the cause was a no-op rewrite or one of those maintenance commands: a retry then succeeds. When the other transaction was a concurrent recall that spent the ticket, or a real edit, the refusal is correct and a retry is refused again (“approval already used”). See MCP approvals and metrics.

Behaviour change: the Windows agent installer passes the enrollment token by file

What changed. The Windows agent installer no longer puts the enrollment token on the registered service command line. It writes the token to %ProgramData%\Yuzu\enrollment-token (restricted to Administrators and SYSTEM, owned by Administrators), configures the service with --enrollment-token-file pointing at that file, and the agent removes the file after its first accepted registration or when the server refuses the token. /TOKEN=<token> still works; /TOKEN_FILE=<file> is new and keeps the token out of Setup’s own log too (Setup’s command line, and therefore its /LOG= log, still records a /TOKEN= value verbatim if that is the form given). A service registered by this installer shows --enrollment-token-file and no token value in sc.exe qc YuzuAgent.

Who is affected. Anyone installing or upgrading the Windows agent with /TOKEN=, and every Windows agent registered by an earlier installer.

How to fix, and what the installer and agent now refuse.

Behaviour change: stored responses of Forensics-tier plugins need Forensics:Read (#4990, fresh deployment only)

This section describes how the new build behaves in a fresh deployment. It is not a procedure for upgrading an existing database: see “Deployment contract (hard cutover)” below.

What changed. Rows written by Forensics-tier plugins (app_usage, browser_inventory, execution_artifacts, privacy_permissions, remote_access_audit) are now served only to a caller who holds Forensics:Read in addition to Response:Read. The set of plugins is derived from the capability catalogue (every plugin with an action on the Forensics securable). This applies on every path that reads stored responses: the legacy /api/responses/{id} routes, the /api/v1 response, execution-response and visualization routes, query_responses and aggregate_responses on MCP, the dashboard results view and its filter options, the executions drawer and page detail, the Hardware Actions-lens result poll, and the agent counts in the management-group previews. To any other caller those rows are absent: counts, truncation flags and filter options are computed over the rows the caller may read, and no error is returned. An empty or short result therefore no longer implies that no data exists. A read that serves such rows also writes a response.forensics.read audit row.

The same tier is applied wherever the server reads those rows for a reason other than a direct response read:

What the caller sees. A caller without Forensics:Read sees fewer rows, or none, where those plugins ran. If the server cannot complete the permission check (the authorization store is unavailable, or a dependency of the check was never wired), the REST and MCP routes answer a retryable error (503 with retry_after_ms 5000 and no permission field on REST, since granting a permission does not fix an outage; an internal error carrying a retry hint on MCP) instead of a partial result, and the dashboard shows a notice instead of an empty list. This holds for sessions, API tokens, MCP tokens and engine principals (an engine principal is included once its cached RBAC view says RBAC is off and is older than the 5 second stale-serve bound: whether RBAC is on can no longer be confirmed, so no grant is read and the engine is refused with the retryable error, whatever grant it holds; a cached view that says RBAC is on still reads the grant, and a failed read is the retryable error; inside the bound a stale view that says RBAC is off still answers a definitive 403; the authorization model reference, docs/authz-model.md, has the detail), and a failed role query counts as unavailable as well (yuzu_server_rbac_read_degrade_total{reason="query_error"}). A service-scoped token is the exception: while the authorization store is degraded it is default-denied with a definitive 403 (no retry hint; such a token never holds the grant). The 503 shape above (retry_after_ms 5000, no permission field) is the Forensics check’s own answer. Every surface first runs its own admission gate (Response:Read; Inventory:Read and Execution:Execute on the Hardware poll), and when a whole-store outage fails that gate first, the gate’s own answer is returned instead, in its own shape (it can name a permission), and yuzu_server_forensics_tier_withheld_total does not move. Retry, and see the alerts below if it persists. “Create group from results” creates nothing and shows an error toast (“The permission check for this action could not be completed. Nothing was created. Retry shortly.”) when the permission check cannot be completed (HTTP 200; the toast arrives in an HX-Trigger showToast header because the form swaps no body into the page); its store-degraded case answers 503 with its own error toast and also creates nothing. The Hardware Actions-lens result poll tells a caller who may not read a Forensics-tier reply that the reply needs Forensics:Read, instead of reporting a timeout; that notice, and the notice shown when the permission could not be confirmed, leave no audit row of their own. The poll is not re-armed after the “could not be completed” notice: do not press Run again for that reason, because Run starts a new collection although the first reply may already be stored; open the execution’s results (the Executions page) instead. A call made with an API token, an MCP token or an engine principal that does not hold a global Forensics:Read also leaves one auth.permission_required denied audit row per read of these responses, so expect new denial rows from automation that polls stored responses; an interactive browser sign-in (local, OIDC or SAML) does not leave one. The volume is one row per read attempt per such caller: a token that polls once per second leaves about 86,400 denial rows a day (arithmetic from the one-row-per-attempt rule, not a measurement), so size audit retention and SIEM ingest before pointing such automation at a tier it does not hold.

What an operator or an agentic worker will notice. A succeeded execution of one of these plugins shows zero rows to a caller without Forensics:Read (the fan-out counts come from the execution record, not from the rows); that is the tier, not a lost result. A retryable 503 / -32603 with retry_after_ms means the authority could not be determined, not that access was refused. A bundle containing a Forensics step never completes for a caller who cannot read it (the step stays pending; the MCP result keeps advertising retry_after_ms, and each poll refreshes the manifest window, so a worker must bound its polling with its own deadline). The dashboard Console stays on “Waiting for results…” for a Forensics action, because nothing is published to it; open the execution results. A policy whose check runs one of these plugins shows unknown for every agent although the check is still dispatched.

Who is affected. Callers who held only Response:Read: the Viewer, Operator, PlatformEngineer and ITServiceOwner roles, and, while RBAC enforcement is off, every non-administrator session (Forensics:Read is administrator-only until enforcement is on). They no longer see the stored rows of these plugins, including in the executions drawer. A caller with the Execution:Read permission but not Response:Read no longer sees the responses section of the executions drawer at all; it names the missing permission instead. Administrators, and holders of the new ForensicsAnalyst role (below) or of a role delegated Forensics:Read, see no change. Automation that polls these plugins’ results with an API token is affected the same way as its owner: every one of them needs a global Forensics:Read; a service-scoped token never proves it. A plain API token carries its owner’s authority, and so does an MCP-tier token created by a user holding a global Forensics:Read.

How to fix. Decide who needs these rows, then:

The filter works per plugin because only the plugin name is stored with a row; no plugin with both Forensics and ordinary actions exists today. The tier is per plugin, not per kind of data: other plugins can return comparable information under their own securables (for example file, registry, WMI or event-log reads), and their stored rows remain governed by Response:Read as before.

Deployment contract (hard cutover). This release supports initial deployment into a fresh database with no data imported from earlier builds. Existing databases and backups from earlier builds are unsupported. Every server writer and serving replica must run the new build. Earlier binaries must have no access to the new deployment’s database. Restarts, additional replicas and recovery may reuse data created entirely within this deployment contract. Recovery means rolling forward on the new build, or restoring this build’s own backups into a fresh deployment; a database or backup written by an earlier build is unsupported. The executable does not enforce this: it does not verify the provenance of the database it opens and it cannot keep an earlier binary away from it. Provisioning a fresh database and excluding earlier binaries from it are deployment responsibilities. Data written by earlier builds is outside the supported population: this section describes no upgrade path, no operation alongside earlier builds and no return to an earlier build, and makes no statement about what an earlier build stored. An earlier binary has no Forensics tier, so pointing one at this deployment’s database would let every holder of Response:Read read these rows again, and the executable cannot prevent it. Notification history is served as stored (there is no read-time scrub and no retention bound), so a database carried forward from an earlier build would expose earlier failure text through the notification list to a holder of Infrastructure:Read, and nothing in the executable detects such a database. Test environments follow the same rule: because this release is a fresh-deployment cutover, any pre-release upgrade test (the /test skill’s Phase 2, which upgrades the previous release’s database to HEAD) must run against a freshly provisioned database, and standing rigs (the 3-env stack, a home server, a shared test PostgreSQL, long-lived agent hosts) must be reprovisioned fresh before they are pointed at this build. The contract is a scope decision of the project owner, 2026-10-24 (no upgrade path is provided); it is not an acceptance of any residual risk, and it is procedural, not enforced by the executable.

Leftover enrollment files (added 2026-10-30 as a hazard disclosure; not part of the owner’s decision above). The statement above that no data is imported from earlier builds has an exception for two files. The server keeps one import record per file kind (tokens, pending agents), holding a SHA-256 of the file. At a boot where there is no record for that kind (a fresh database, or one restored from a dump taken before the first import), it imports enrollment-tokens.cfg and pending-agents.cfg from --data-dir, or from the directory holding yuzu-server.cfg when --data-dir has no copy, without checking where they came from; a copy in the other directory is never imported but stays in place, so remove it too. Where a record exists, an identical file is skipped and a different file (for example one restored with an older data directory) is refused, not merged: the server logs [enrollment-import] ... DIFFERS from the file that was already imported at CRITICAL, writes an enrollment.import audit row with result=warning, counts yuzu_server_enrollment_import_total{outcome="fingerprint_mismatch"} and leaves the file in place; remove or archive it. A file that exists but cannot be read stops the server from starting. Imported tokens (with their max_uses, use_count, expires and revoked state) and approved pending agents are live credentials. Before the first boot of a fresh deployment, confirm both files are absent from both directories; a fresh database is not enough if the data directory is reused. Also look for their .imported renames: the server never reads those again, but they show that an import happened. The directories are: for the Docker image and the reference compose files, /var/lib/yuzu (the <project>_server-data volume); for the packaged Linux service, /var/lib/yuzu and /etc/yuzu; for the Windows installer, %ProgramData%\Yuzu Server and %ProgramData%\Yuzu Server\data. To find out whether an import already happened, look for the log line [enrollment-import] ONE-TIME import of legacy ..., an enrollment.import audit row (source=<path> in the detail), the counter yuzu_server_enrollment_import_total{outcome="imported"} (per process, so it resets on restart), and rows whose created_by or status_changed_by is legacy-import. The audit row, the legacy-import rows and the import record are stored in the database, so a database restored from a dump taken before the import contains none of them; the .imported renames and the server log are on the host and are not rolled back by a restore. With several server replicas on one database, check the directories on every replica. To undo an import, revoke the imported tokens with DELETE /api/settings/enrollment-tokens/{id} and deny any imported agent that is still pending (POST /api/settings/pending-agents/{id}/deny). An imported agent that was already approved cannot be denied or removed through the dashboard or the REST API (both answer 409), is listed nowhere, and registers without an enrollment token; there is no supported way to remove one. Treat a deployment that imported approved agents as one whose enrollment state you cannot trust, and redeploy from a fresh database and empty directories. Removing the rows in SQL (auth.pending_agents, status_changed_by = 'legacy-import') is outside the supported surface and unaudited.

After deployment.

  1. Assign the ForensicsAnalyst role (or Administrator) fleet-wide to the people and automation that need to read these rows (see “How to fix”).
  2. Watch yuzu_server_forensics_tier_withheld_total (it should stay flat), yuzu_server_response_provenance_total{outcome} (the unrecorded and lookup_degraded outcomes should stay flat) and the counters that name the cause: yuzu_exec_correlation_write_degrade_total (dispatch-record writes that failed or, in the operator Guardian rule push after its first failure, were skipped; one outage push adds one count per target), yuzu_exec_correlation_read_degrade_total{reason} (record reads that failed) and the yuzu_pg_* pool and lock metrics. The alerts YuzuDispatchRecordWriteFailing (write failures), YuzuResponseProvenanceLookupDegraded (record reads that fail twice) and YuzuExecCorrelationReapSaturated (the retention sweep running at its cap) watch the record (see “Alert” below). If one fires, check PostgreSQL health and locks first, then the matching counter. Also watch the server log warning “Forensics execution lookup failed” (a second signal, log only with no counter: the execution lookup that decides whether stored failure text is shown failed, and the text was withheld; throttled to about one line a minute), and the response.forensics.read audit verb (it is the evidence of who read these rows, with limits: it is written only by reads that return at least one such row; aggregate, facet-option and count reads, the dashboard filter bar and the create-group-from-results action write no such row; an interactive cookie session that lacks the grant leaves no denial row; and on the dashboard and MCP surfaces a failure to persist it does not stop the data being served, unlike REST; the audit log guide lists the reads it does not count). Expect auth.permission_required denial rows from token automation that polls these responses without the grant (about 86,400 a day for a once-per-second poller; see “What the caller sees”).
  3. Verify. Run a Forensics-tier action against one device, then read its responses with query_responses as an Administrator (the rows are returned and a response.forensics.read row is written) and as an Operator (no rows and no error). If the Administrator gets none, first check that the action completed on the device and that the Administrator holds Forensics:Read. Then run an ordinary plugin (one that is not Forensics-tier) against the same device and confirm that the Operator sees its rows and that yuzu_server_response_provenance_total{outcome="verified"} moved. This second check matters: if every dispatch-record write is failing, every response is stored ~unverified, which is Forensics-tier, so the first check gives the same observations as a healthy tier while every ordinary plugin’s results are hidden from every caller without Forensics:Read.

Known limits. The filter covers reads of stored responses and the server-side consumers listed above. It does not change who may dispatch a Forensics-tier plugin; a separate follow-up (#6065) tracks the dispatch-side authorization of policy checks. The provenance label proves that the server issued the command id for that plugin; it does not prove who sent the frame. An endpoint that was legitimately sent a command (or was inside a broadcast of it) can replay that id on a frame of its own and so label that frame with the command’s plugin, and the content of a frame is not validated. A single execution that runs longer than the 24 hour record window finds no record when it answers and is stored as ~unverified; a delayed or staggered command does not reach that window, because delay and stagger are each limited to 300 seconds (600 seconds in total) and only POST /api/command accepts them. A dispatch whose record write failed has its responses stored as ~unverified, so a reader that excludes the Forensics set (for example the Hardware Actions-lens result poll) never finds them, and a registry write command with no record mints no registry.write audit rows, because those rows are minted only for a response labelled registry.

Role name collision. The ForensicsAnalyst role is created by the idempotent seed pass at server start and is assigned to nobody. If a role with that name that is not a built-in role already exists (it can only have been created by hand in PostgreSQL), it is left as it is, the four read grants are NOT added to it (so its existing holders gain no Forensics:Read), and the server logs one error naming the role at start-up; the built-in role is then unavailable until the hand-made role is removed and the server restarted. To find it, list the roles (GET /api/v1/rbac/roles) and look for ForensicsAnalyst with is_system false, and its holders with GET /api/v1/rbac/roles/assignments. While the collision exists, assigning ForensicsAnalyst through POST /api/v1/rbac/roles/ForensicsAnalyst/assignments or assign_rbac_role is refused as a name collision (409 on REST, an invalid-params error on MCP, audited). There is no product route to remove a role, so the remedy is direct SQL on the rbac_store schema, which is unsupported and not audited: create a replacement role under another name first if its holders need one, then DELETE FROM rbac_store.roles WHERE name = 'ForensicsAnalyst' AND NOT is_system, which also removes the role’s grants and assignments (they reference the role by name with ON DELETE CASCADE, and a role with grants cannot be renamed); a hand DELETE does not bump the RBAC generation, so restart every serving replica afterwards. If the seed’s role check cannot be completed at start-up, the four grants are withheld for that boot and added at the next one (an error is logged). Grants an earlier build already merged into such a role are not removed. Because it is not a built-in role, the engine-principal bar does not apply to it: review who holds it and what it grants.

Alert. YuzuForensicsTierUnresolved fires when reads are being refused because the caller’s authority could not be determined, repeatedly over about 10 minutes (it is rate(yuzu_server_forensics_tier_withheld_total[5m]) > 0 for 10 minutes, so a single refusal or a burst shorter than about 5 minutes does not fire it, and neither does a refusal that recurs only every 5 minutes or longer, by design, because the 5 minute rate window is then false between events). It points at the authorization store (rbac_unavailable), as the metric’s reason label names; unwired is not a store problem but a wiring defect in the build, which a running server should not show. For rbac_unavailable, check the authorization store’s health and correlate with yuzu_server_rbac_read_degrade_total, the pool metrics and the server log line “Forensics-tier visibility could not be resolved”; the reads recover on their own once the store answers.

Three more alerts (four in all, with YuzuForensicsTierUnresolved above) watch the dispatch record, which decides the label (and so the tier) of every stored response. YuzuDispatchRecordWriteFailing (warning) fires when rate(yuzu_exec_correlation_write_degrade_total[5m]) > 0 for 10 minutes: dispatch is unaffected, but each response to a command whose record was not written is stored ~unverified. YuzuResponseProvenanceLookupDegraded (warning) fires when yuzu_server_response_provenance_total{outcome="lookup_degraded"} has a non-zero rate for 10 minutes: the record read failed twice for incoming responses, with the same consequence. For both, check PostgreSQL health and locks and the yuzu_pg_* pool and lock metrics first, then the counter; responses already stored ~unverified are not repaired when the store recovers, so re-run the affected commands. A dispatch rate below about one every 5 minutes is not detected by the rate window (the counter is the complete evidence). YuzuExecCorrelationReapSaturated (warning) fires when the hourly sweep of the dispatch-record table has run at or near its 50,000 rows per pass cap for 24 hours (increase(yuzu_exec_correlation_reap_total[6h]) >= 200000): records are created faster than the sweep reclaims them, which is a capacity signal and leaves nothing unverified. There is deliberately no alert on the unrecorded outcome: it also counts an id the server never issued, a record reaped after 24 hours and an empty id, so it is non-zero on a healthy fleet.

Start-up refusal. If the log shows [Forensics] Refusing to start: <reason> (#4990) - fail-closed at ERROR level and the server exits, the capability catalogue compiled into this binary yields no usable Forensics plugin set, and the server will not start rather than serve a tier that excludes nothing. It is a defect in the build, not in the configuration or the database: install a correct build of the same release.

Behaviour change: new enrollment tokens default to 72 hours and 100 uses

What changed. A limit left blank when minting an enrollment token now defaults to 72 hours and 100 uses, on every create surface: the dashboard form, POST /api/settings/enrollment-tokens, POST /api/settings/enrollment-tokens/batch (ttl_hours only — its max_uses default stays 1), and the server CLI --generate-tokens --token-ttl-hours (its --token-max-uses default also stays 1). An explicit 0 still means unlimited uses / no expiry, exactly as before. The CLI now refuses a negative --token-ttl-hours or --token-max-uses (a negative --token-ttl-hours used to mint a token with no expiry, and a negative --token-max-uses was refused by the store rather than by the option parser), and its enrollment.bulk_token_create audit row now records ttl_s= and label=.

Who is affected. A script or operator that relies on leaving a limit blank to mean unlimited/never-expiring now gets a 72-hour/100-use token instead. A rollout that stages tokens in advance, or enrolls more machines from one token than its uses, is refused once the token is spent or expired.

How to fix. Pass an explicit 0 for max_uses and/or ttl_hours to keep the old unlimited/no-expiry behaviour, or size them to the rollout window and the number of machines.

Behaviour change: a new Postgres schema for the fleet certificate inventory, and an eighth decommission store

The server now creates the Postgres schema certificate_inventory_store at boot (seven tables, no data until a certificate-collecting agent reports). The server’s database role needs CREATE on the database, as for every other store; if the schema cannot be created the server refuses to start (fail-closed, ADR-0007). /readyz includes certificate_inventory_store once the server is up, so a degraded store after boot shows there.

DELETE /api/v1/sle/agents/{id} now fans out across eight stores: the response’s stores map gains a certificate_inventory key and deleted reaches 8 when every store is configured. During a rolling upgrade a replica that has not been upgraded runs the old cascade and returns 200 without that key; it did not erase certificate rows. Re-issue the DELETE once every replica is upgraded (the same advice as for the presence row, #5682). Erasure is not durable yet: a still-running agent repopulates its rows on its next report until the decommission tombstone ships in a later change.

No agent sends certificate_inventory yet; agents start only once the server advertises that it accepts the source, in a separate later change. The server’s gateway path (ProxyInventory) now stores such a blob in this typed store, exactly as the direct ReportInventory path does. The previous release discarded it there; releases before the typed-source read filter (see “Behaviour change: generic inventory reads never return a typed source’s rows” below) stored it in the generic store, which that filter and the boot purge now exclude. It never reaches the generic inventory store.

Rolling back leaves the certificate_inventory_store schema in the database; it is harmless to the older binary and is not read. Plan capacity before agents start reporting: DER is optional and bounded at 1 MiB per device, so a large fleet is storage- and WAL-heavy (see Server administration).

Behaviour change: generic inventory reads never return a typed source’s rows

What changed. The generic inventory store’s reads (query_inventory, get_agent_inventory, list_inventory_tables and their REST twins, POST /api/v1/inventory/evaluate, and the inventory-query result-set source) no longer return rows for the typed daily-sync sources: installed_software, app_perf, device_ci, software_licensing, app_usage, and the reserved certificate_inventory. Those sources have their own stores and gates. Before this change, a typed source’s blob that reached the generic store through an older server’s gateway path could be read there under Infrastructure:Read or Inventory:Read until the next boot purge removed it. That window is now closed on every server at this release or later (see the per-server note below). A certificate_inventory blob, a source no agent sends yet, arriving through a gateway is stored only in the certificate inventory’s own typed store (see the certificate inventory note above), never generically.

Who notices. Only a caller that read one of those six source names from the generic store. Their data was never meant to be there. Use the typed surface instead, which is unchanged:

SourceRESTMCP
installed_softwareGET /api/v1/inventory/softwarequery_installed_software
device_ciGET /api/v1/hardware/{id}get_hardware_ci, list_hardware_ci
app_perfGET /api/v1/dex/perf/apps and the other /api/v1/dex/perf/app* routeslist_dex_perf_apps, get_dex_app_perf
software_licensingGET /api/v1/sle/agents/{agent_id}query_software_licenses
app_usageGET /api/v1/forensics/agents/{agent_id}/app-usageget_agent_app_usage
certificate_inventoryreserved; its routes arrive with its store—

Deploy order, and what is per server. The filter protects reads served by a server at this release or later, and nothing else. Every server older than this release, or rolled back below it, keeps storing typed-source blobs that arrive through a gateway attached to it, and keeps serving them under Infrastructure:Read or Inventory:Read. In a multi-replica deployment that means the exposure is not closed until every replica runs this release: during a rolling upgrade, reads routed to an older replica still return those rows, and rows an older replica writes after an upgraded replica’s boot purge stay in the table (unreadable through upgraded replicas) until a purge runs again. Upgrade servers before agents, finish the rolling upgrade promptly, and never roll a server back below this release once agents send certificate_inventory (none does yet).

No action on upgrade. There is no schema change. The server still removes any stray typed rows at every boot, but it now does so in the background in bounded batches, so startup no longer waits on the purge and a large backlog cannot delay serving. Stray rows were already unreadable from the generic store, so nothing depends on the purge finishing. Completion or failure is logged, and removed rows and failures are counted in yuzu_inventory_typed_purge_rows_total and yuzu_inventory_typed_purge_failures_total. If a pass fails, a restart re-runs it.

Behaviour change: the DEX score policy document carries a version (#6085, #6086)

What changed. The canonical text of the dex_score_policy document now includes "version": 1, and version is a recognised key. A document without the field reads as version 1. A server that meets a version above the one it supports scores under the shipped defaults, reports fault: unsupported_version at version: N above the supported 1 on every Apps response, and logs it once. It never applies part of a document, and the revision stays the hash of the stored text. PUT /api/v1/dex/score-policy writes the document from this release. A build that predates the score policy reader ignores it. A development build that has the reader but predates this change refuses a version key as an unknown key and serves the shipped defaults with a fault for any document that carries it.

The rule, from this release. PUT and DELETE /api/v1/dex/score-policy are live, so the rule applies: do not change the score policy during a rolling upgrade or rollback: a newer server may write a document that an older server in the same fleet refuses or, if it predates the score policy reader, ignores (on a rollback, a build that predates the version key refuses it as an unknown key), and until every instance runs the same release the instances can score under different policies.

If a stored document is refused. A later release that tightens validation can refuse a document an earlier release accepted. The server then serves the shipped defaults with a fault naming the field. Save the policy again through PUT /api/v1/dex/score-policy (the response to a GET of the policy shows the revision to send as If-Match). Nothing is migrated automatically.

No action on upgrade. There is no schema change. No policy written by an earlier release exists to migrate (a row placed by a direct database write is read as any other document).

Behaviour change: the DEX score policy document gains model parameters and version 2

What changed. The score policy can now set the per-device event cap, the regression full-weight and start ratios (each globally and per application) and the interval confidence level (globally). The canonical text stays version 1 and byte-identical, with the same revision, until one of them differs from its default (5, 3.0, 1.0, 0.95) or a per-application override sets one, even at the global value, because that pins the application. The writer then stores version 2, with a model object. A version 1 document that carries model is refused. With the defaults every score, interval and band is exactly what the previous release produced.

The rule. A server on the previous release meets a version 2 document as unsupported_version at version: 2 above the supported 1. It scores under the shipped defaults with that fault on every Apps response and never applies part of the document. So do not set a model parameter during a rolling upgrade or rollback.

While a version 2 document is stored, new and old replicas return different Apps figures. An older replica’s GET /api/v1/dex/score-policy shows the shipped defaults, not the stored document, and a PUT or DELETE from it replaces the whole document. The audit row records the new document and the prior revision, not the replaced text, so keep a copy of the version 2 document (GET it from a current replica) before any such write. After a rollback, save the policy again without the model block first; DELETE only if resetting tiers and weights too is intended. The unsupported_version fault is described in REST API.

No action on upgrade. There is no schema change. A stored version 1 document stays valid and is read as before.

⚠️ Breaking: POST and DELETE /api/v1/quarantine now apply the dispatch approval check (#6065)

What changed. POST /api/v1/quarantine (and the dashboard’s Quarantine device button, which calls it) now runs the same dispatch approval check as the quarantine plugin’s quarantine action, after the scoped Security:Execute gate and the whitelist validation and before any record is written. DELETE /api/v1/quarantine/{agent_id} (the Release from quarantine button) now runs the same check for the plugin’s unquarantine action, after the scoped gate and before the attempt audit row and the release. Neither depends on any approval-workflow setting.

“Administrator” here means an account whose account role is admin (a live temporary JIT elevation counts as that role). The RBAC Administrator role, or a global Security:Execute grant, on an account whose role is user does not make it one.

A refused request writes no quarantine record and releases none. The 403 bodies carry message and remediation and no permission field.

Release. The same answers apply to DELETE: a non-administrator who could end a record before now gets 403 approval_required (RBAC on, global grant) or 403 forbidden (group-only, or an unreadable RBAC store), the record stays active, and the refusal is audited as quarantine.disable with result=denied and no attempt row. Only an administrator (with RBAC on, one who also holds Security:Execute globally) can release a record through REST or the dashboard. A JIT elevation counts as the administrator role only if the account also holds Security:Execute globally; otherwise the release is refused forbidden. The remediation text names the MCP supervised flow, which can create a record but has no release tool, so for a release the only path is an administrator. Release still ends the server record only; it does not remove the endpoint firewall rules.

Rolling upgrades and rollback. The check is enforced per server replica. During a rolling upgrade, or after rolling a replica back, a replica that has not been upgraded still accepts POST /api/v1/quarantine from a non-administrator who holds Security:Execute on the device (201, and the leader then isolates the device) and accepts the same caller’s DELETE release (200). The check applies across the cluster only once every replica runs this version, so do not treat a 201 from a non-administrator during the rollout as a failure of the new check. Rolling back restores the old behaviour; no data changes in either direction.

What to check. Automation that creates or releases quarantine records with a non-administrator API token or cookie session must move to a durable administrator account (not a JIT elevation; with RBAC on, one that also holds Security:Execute globally in its own right). For creation only, MCP quarantine_device on the supervised tier is the alternative, where an administrator approves the ticket in an interactive session with MFA step-up (see the #6061 note below); release has no such path and needs an administrator. To find callers that were relying on the old behaviour, search the audit log for action quarantine.enable (create) or quarantine.disable (release) with result denied: the principal and the device are on the row. yuzu_server_dispatch_denied_total{reason} shows only the fleet-wide trend; it is shared with MCP and /api/command and cannot name this route. For a least-privilege containment operator (Security:Execute, not an administrator) there is no REST or dashboard approval path yet; adding one is planned follow-up work, and until then creation can go through the MCP supervised flow or the quarantine plugin’s quarantine action on the Instructions page (role-gated approval). Releasing through the plugin’s unquarantine action alone clears the endpoint firewall rules but leaves the active record, so while the record is active the reconciler re-applies the stored isolation when the device next reconnects; an administrator must end the record. No configuration or data migration is required.

Before you upgrade. Do these on the version you are running now:

  1. If a reverse proxy fronts the dashboard, check that it forwards the client’s Host unchanged, port included. nginx rewrites Host by default (it sends $proxy_host), so add proxy_set_header Host $http_host; in every location that proxies to Yuzu, or at server level. $host drops the port, so use $http_host unless the public port is 80 or 443. Behind a TLS-terminating proxy, also declare the public origin with its scheme (for example --csrf-trusted-origin https://yuzu.example.com) even when Host is forwarded: the server cannot see the public scheme, and the scheme-qualified entry makes it refuse a plain-http:// Origin for that host. If the proxy cannot forward Host, set --csrf-trusted-origin https://<public-origin> (or the environment variable YUZU_CSRF_TRUSTED_ORIGIN) on every server replica first, restart, and confirm the start-up line described under “Confirming it took” in Server Administration.
  2. Create at least one API token for your automation before upgrading. Creating a token from the dashboard is itself a cookie-session write, so it is refused on a misconfigured proxy after the upgrade.
  3. A script that reuses a login cookie jar should switch to the token.

Symptom and recovery. On a deployment that skipped step 1, every dashboard write answers 403 cross-site request refused; sign-in and ordinary reads still work, API tokens cannot be created, and the step-up elevation request is refused. The check cannot be disabled: the remedies are the flag, an API token or rolling back. Recovery is to restart with the flag or environment variable, or to reach the server directly (for example an SSH tunnel to the web port, so Host matches Origin), sign in and create a token. A script that reaches the server directly must send an Origin naming the address it connects to (for example http://localhost:8080, or https://localhost:8443 when the server terminates TLS itself, where an http:// origin is refused). Direct access needs a local sign-in: a deployment that offers only OIDC or SAML sign-in redirects to the public URL, so for those only restarting with the flag works. There is no reload: the value is read at start. Every replica needs the same value, or the same page flaps between 200 and 403 depending on which replica answers. An Origin: null header, which some browser and proxy configurations send, cannot be fixed by the flag; do not set Referrer-Policy: no-referrer at the proxy (it removes the Referer fallback, and the Fetch standard makes browsers serialise the Origin of a non-CORS form submission as null under that policy; script-driven requests are unaffected), and the remedy is to remove it or roll back. /livez and /readyz stay green throughout, so a probe does not detect this. The upgrade makes no schema change, so downgrading is safe.

What changed. Every request other than GET, HEAD or OPTIONS that is authorised by a session cookie must carry an Origin or Referer header naming this site (or an entry from --csrf-trusted-origin). Otherwise it answers 403 cross-site request refused (the standard error envelope, with a remediation) and is audited as csrf.denied. One check, applied centrally, decides this after authentication and before step-up evaluation, challenge emission and route-handler effects, and before the request body is parsed; a client that sends Expect: 100-continue has already been told to send the body, and a large upload may still be in flight when the 403 arrives. A request that presents a valid session cookie is treated as a cookie-session request even if it also carries a token. A request authenticated only by an API, MCP or engine token is not subject to this check, and neither is a request answered before a session is resolved. The per-route checks described elsewhere in this guide stay in place as defence in depth; a route that keeps a copy of its own that is not token-exempt, so a token caller of one of those can still be refused by it.

Guarded reads. The same rule also covers cookie-session requests that dispatch work, persist state or force a refresh of server state, whichever method they use (the guarded reads). The exact path, never a query flag, selects the gate. The mechanism for them is the browser’s fetch metadata, and a Sec-Fetch-Site header that is present decides alone: exactly one header with value same-origin or none is admitted, any other value (or a repeated Sec-Fetch-Site header) is refused with origin_mismatch, and Origin and Referer are not consulted. Only when Sec-Fetch-Site is absent does the Origin or Referer rule above apply. Browsers send Sec-Fetch-* headers only to potentially trustworthy origins (HTTPS and loopback), and Safari before 16.4 sends none, so on a plain-HTTP deployment at a non-loopback address these requests depend on the Origin/Referer rule; a same-origin GET carries no Origin, so it depends on Referer, which the server’s default Referrer-Policy (strict-origin-when-cross-origin) supplies. A proxy or page policy of no-referrer would break those dashboard panels on such a deployment; a TLS-terminating proxy is fine, because the browser talks to the proxy over HTTPS. These requests are audited as csrf.denied with a target_id such as GET /path. Token callers are not gated.

Who is affected. A script or tool that signs in with a cookie and writes without an Origin or Referer header now gets 403. A deployment behind a reverse proxy that rewrites Host and has no --csrf-trusted-origin entry has every cookie-session write refused until the flag is set. At start-up the server prints one line to standard output: the number of origins accepted and their values when at least one is accepted, otherwise the rule and a note that none was accepted (it is not written to --log-file, and --log-level does not hide it). The first refusal of each kind logs a warning naming the flag. The dashboard and other browsers send these headers on every write and need no change. A script or tool that sends the guarded reads (cookie-session requests that dispatch work, persist state or force a refresh of server state) with a login cookie and no Sec-Fetch-Site header must now send a same-site Origin or Referer; one that sends Sec-Fetch-Site must send it once with value same-origin or none. Otherwise it gets 403. The audit row for such a refusal still carries the fixed detail text “browser write refused”. Browsers need no change. On most dashboard pages a refused write shows no error message (the page does not swap the 403 response in), so a proxy misconfiguration can look like buttons that do nothing; check the audit log and yuzu_server_csrf_denied_total.

How to fix. Send Origin matching the console origin (for example -H "Origin: https://yuzu.example.com"), or Referer for an endpoint that validates Origin against its own allowlist (see MCP), or use an API token. Behind a reverse proxy, declare the public origin with --csrf-trusted-origin, including the scheme (for example https://yuzu.example.com); this is required if the proxy rewrites Host, and with a scheme it also makes the server refuse a plain-http:// Origin for that host. A server that terminates TLS itself needs no declaration: it requires an https:// Origin or Referer. For the guarded reads, either send no Sec-Fetch-Site header and a same-site Origin or Referer, or send Sec-Fetch-Site once with value same-origin or none (a Sec-Fetch-Site header that is present decides alone, so a script that sets it to any other value, or sets it twice, is refused even with a good Origin); or authenticate with an API token. Do not configure the proxy to inject or rewrite Origin or Referer, which would turn the control off for that deployment. No data migration is required.

What to check. Since this release, a csrf.denied row for a cookie-session caller has result denied, target_id set to the sanitised request method and path (for example POST /path; the request path is truncated at 200 bytes) and a detail of fixed text followed by reason=<token>. Rows keyed on the earlier per-route target_id values or on result error now appear only for token callers of routes that keep their own non-token-exempt copy, so a SIEM rule keyed on those values goes quiet for cookie callers: re-key it on action csrf.denied (and result denied), and treat the date of this release as the boundary in exported evidence. The counter yuzu_server_csrf_denied_total{path_class,reason} records every refusal (reason is no_origin_no_referer, origin_mismatch, referer_mismatch or gate_error) and the shipped yuzu-web-writes alert rules watch it (Metrics); a rise after the upgrade is usually a script without the header or a proxy without the flag.

What changed. A call to a privileged REST write route (the routes whose reference entry carries a “Cross-site writes” note in the REST API guide) authorised by a session cookie must carry an Origin or Referer header naming this site (or an entry from --csrf-trusted-origin). Otherwise it answers 403 cross-site request refused and is audited as csrf.denied. The central same-site check (see the section above) answers first, after authentication and before authorization; each of these routes keeps its own copy after its authorization gates and MFA step-up as defence in depth, and before the request body is read. API, MCP and engine token callers are not affected by these routes.

Who is affected. A script or tool that signs in with a cookie and calls one of these routes without an Origin or Referer header now gets 403. No dashboard page calls these routes.

How to fix. Call the route with an API token, or send an Origin header matching the dashboard origin (for example -H "Origin: https://yuzu.example.com"). Behind a reverse proxy that rewrites Host, declare the public address with --csrf-trusted-origin. No configuration or data migration is required.

⚠️ Breaking: approving or rejecting a request now needs an interactive session and an enrolled MFA factor with a fresh proof; MCP cannot approve (#6061, #6079)

Before you upgrade (checklist).

  1. List your administrators and everyone who holds Approval:Approve (see “Find affected automation before you upgrade” below). With RBAC enforcement off (the shipped default) the approvers are exactly the accounts with role = 'admin'; the step-0 query in step 2 returns the local ones that are not yet enrolled.
  2. Check who is enrolled: run the step-0 SQL, the first block of Path A in the recovery runbook. On a Linux package install the psql "$YUZU_POSTGRES_DSN" blocks in this section fail in an interactive shell, because the DSN lives only in the env file, which is readable only by root and the yuzu account; read the value into a shell variable (do not source the file, because a DSN containing & or spaces breaks as shell) with YUZU_POSTGRES_DSN=$(sudo sed -n 's/^YUZU_POSTGRES_DSN=//p' /etc/yuzu/yuzu-server.env) and then run each block as written.
  3. Enrol each administrator in Settings → Security and sign-in → Two-factor sign-in; another administrator cannot do it for them.
  4. Plan each non-administrator approver (possible only once RBAC enforcement is on): it needs the runbook path, not Settings.
  5. A SAML deployment has no approver path (#6098); give those approvers a local or OIDC account first.
  6. Have at least two enrolled approvers: a submitter cannot review their own request, so a single enrolled approver cannot decide that approver’s own requests.
  7. After the upgrade, run the approvals-refused query (step 6 of “After upgrading, verify approvals on a test request” below) to see whether approvals are being refused and why.
  8. Owners of scripts that run yuzu-server --mfa-reset, and of SIEM rules on mfa.disabled or mfa.reset.breakglass: --mfa-reset now exits 1 when ending the sessions fails (result=partial), when the audit store refuses its intent row, and when the clear itself fails; every clear writes two audit rows; a failed session delete is result=partial, not success (details in the --mfa-reset notice below).

What changed. Approve and reject (POST /api/approvals/{id}/approve, POST /api/approvals/{id}/reject, the dashboard Approvals page, and the MCP tools approve_request / reject_request) now require all of: an interactive cookie session (local or OIDC), an enrolled MFA factor with a fresh proof (#6079, described below), a human principal and not an engine principal. The existing rule that the reviewer cannot be the submitter is unchanged.

Who this affects. Any automation that approved or rejected over MCP or with an API token. It will now fail. A supervised-tier agentic worker still requests approval and recalls with the approval_id, but a person must approve in the dashboard or over REST on a signed-in session. SAML-only approvers must sign in with a local account or OIDC. Also affected: every approver whose account has no enrolled MFA factor, which on a default install means every approver. With RBAC enforcement off and --mfa-enforcement=optional (the shipped defaults) the only approvers are administrators, and none is enrolled until it enrols itself in Settings, so each is refused until it does (another administrator cannot enrol it). Once RBAC enforcement is on, every non-admin approver under optional or admin-only is in the same position and needs the runbook path. Also affected is every OIDC approver whose provider does not assert an MFA amr; both are refused until fixed. Scripts that approve with a session cookie must complete the step-up (POST /login/mfa/stepup) when they receive 401 with meta.mfa_step_up_required.

If you set --mfa-step-up-window-secs=0, approval no longer loses its MFA requirement: approve and reject use a 300-second window instead, and still require an enrolled factor. The setting still switches the step-up gate off for the other step-up endpoints. Decisions made before the upgrade, and in particular any made with the setting at 0, gain no retroactive assurance: review every approval.approve audit row with result=success from that period, and check every attempted row that has no outcome row (compare counts per target_id). A build that has #6061 also writes a result=attempted row first, so do not count both, and a decision whose outcome row failed to write leaves only the attempted row (see Audit Log). A build that predates #6061 writes no attempted row, only the outcome row.

Find affected automation before you upgrade. Nothing is listed for you, so search the audit log for callers that approved or rejected without a signed-in browser session.

The search only sees what the audit retention window still holds, so an empty result is not proof that nothing approves by token. Before upgrading, also check which service accounts and API tokens hold Approval:Approve and move each approval step to a person on a signed-in session. No audit query lists a token or service account that holds Approval:Approve but has never approved: after this change it is refused anyway, so the check is about finding automation to move, not about finding exposure. To list what is granted, read the RBAC grants (GET /api/v1/rbac/roles/assignments, or the Access Reviews export GET /api/v1/access-reviews/export; both need AccessReview:Read) and match each principal’s roles against the roles that carry Approval:Approve. GET /api/v1/tokens lists only the calling user’s own tokens, so check each token owner’s grants.

What that search cannot see. The audit query finds token and MCP approvers only. The approvers most likely to hit the new behaviour are human accounts, which carry principal_class = 'human' and so look the same before and after the upgrade: SAML users (refused), OIDC users whose identity provider asserts no MFA-method amr (now refused mfa_proof_missing; before this change they passed unless --mfa-enforcement=required), and local accounts with no enrolled factor (now refused mfa_not_enrolled; before this change they passed without a second factor). To list the SAML and amr-less OIDC operators who have signed in, query the login rows in the audit store (the retention window applies):

SELECT principal, action, to_timestamp(max(timestamp)) AS last_login
FROM audit_store.audit_events
WHERE action = 'auth.saml_login'
   OR (action = 'auth.oidc_login' AND detail LIKE '%amr_mfa_asserted=false%')
GROUP BY principal, action
ORDER BY last_login DESC;

timestamp is epoch seconds, hence to_timestamp. That lists who signed in by those routes, not who holds Approval:Approve; compare it with the role assignments by hand. For an OIDC row the detail column carries the display name and e-mail (personal data), which helps match a principal to a person. To list the local accounts that have no enrolled MFA factor (read-only; SSO identities are rows with another identity_source), query the auth schema:

SELECT username, role, last_login_at
FROM auth.users
WHERE is_active
  AND identity_source = 'local'
  AND mfa_enrolled_at IS NULL
ORDER BY username;

role is the account’s legacy role column, not its RBAC grants, so compare the result with who holds Approval:Approve. An account listed here is refused as an approver (mfa_not_enrolled) after the upgrade, so enrol it first if it is meant to approve; a non-admin needs the enrolment sign-in window described above. With RBAC enforcement off (the shipped default), a non-admin account is refused 403 “admin role required” before the MFA check, so mfa_not_enrolled is what an administrator sees, and a non-admin sees it only once enforcement is on.

After upgrading, verify approvals on a test request. Use a test device and a harmless, read-only instruction, and do not skip the safety notes in step 1: an execute call that is not held for approval really dispatches.

  1. Create a pending test request. Pick an instruction definition whose approval_mode is always (it needs a ticket whoever submits it) and that only reads state, such as a status or inventory query. No shipped always definition is read-only (all are mutating server.* actions), so with one of those the 202 pending_approval before dispatch is the safety property, and anything else is a finding. Do not use a role-gated definition from an administrator account: under role-gated an administrator, including one with an active JIT elevation, gets no ticket and the instruction is dispatched. As a signed-in dashboard session, or any session you can send a same-site Origin with, call the execute route with a JSON object body that names ONE test device explicitly:

    curl -b cookies.txt -X POST "https://yuzu.example.com/api/instructions/<definition-id>/execute" \
     -H "Origin: https://yuzu.example.com" -H "Content-Type: application/json" \
     -d '{"agent_ids": ["<test-device-agent-id>"]}'

    Never omit both agent_ids and scope on this call: an omitted target is the whole fleet, and only destructive and forensic actions refuse it. The body must be a JSON object (a bodiless POST answers 400 “invalid request body”; {} is the minimum), and a session-cookie POST without a same-site Origin is refused with 403 and a csrf.denied row. Expect 202 with "status":"pending_approval" and an approval_id before you go on. Only a 200 (the instruction was dispatched) means it ran, so stop, treat it as a finding, and check what was dispatched; a 400, 403, 404 or 503 means nothing was dispatched, so fix the cause and retry. A 500 with message dispatch failed may already have queued the command, so check the executions list before retrying. You need two distinct accounts: the submitter cannot approve their own request (that attempt answers 400 and leaves an attempted and a denied row), so approve it as a different human account that holds Approval:Approve. Approving a ticket created this way does not run the instruction: an instruction-origin ticket has no redemption path yet, so the approval is recorded and nothing is dispatched (tracked in #6064). The test proves the approval gate, not execution.

  2. As that human approver on a signed-in browser session, approve the pending test request on the Approvals page. The approver must be an enrolled local account (or an OIDC login your provider sent an MFA amr for). Expect the step-up prompt if the proof is older than the approver window (min(--mfa-step-up-window-secs, 300), or 300 if that is 0), then success. The audit store should show an approval.approve row with result=attempted followed by one with result=success, both with detail=approver_assurance:enrolled_fresh_mfa, and the success row should carry principal_class = 'human'. If you script this step with a session cookie, send an Origin header naming the server (for example -H "Origin: https://yuzu.example.com"), or the POST is refused with 403 before anything is recorded other than a csrf.denied row.

  3. With an API token that holds Approval:Approve, send POST /api/approvals/{id}/approve for a pending request (create a second test request if step 2 already decided the first). Expect 403 with message approver_assurance_required:non_interactive_session, and one approval.approve row with result=denied and detail=approver_assurance:non_interactive_session. The same attempt through MCP approve_request returns -32003 with the same message. (A token without Approval:Approve gets the plain permission 403 first, so it does not prove the new check. Over MCP, an operator-tier or readonly-tier token is refused earlier by the tier check with -32004; use a supervised-tier or untiered token for this step.)

  4. Repeat step 2 with one account of each kind you run and check the outcome: a SAML user (expect 403 MFA step-up is not available for SAML sessions in this release from the central step-up check, with an mfa.step_up.required row reason=saml_session_no_mfa_this_release and no approval.* row; the typed approver_assurance_required:saml_mfa_unsupported appears only with --mfa-step-up-window-secs at 0), an OIDC user your identity provider does not send an MFA amr for (expect 403 approver_assurance_required:mfa_proof_missing; under --mfa-enforcement=required, or admin-only for an administrator, the central check answers first with a 401 that sends the user back to the identity provider, so a provider that never asserts amr loops instead of showing mfa_proof_missing), a local account with no enrolled factor (expect 403 approver_assurance_required:mfa_not_enrolled, with a denied row whose detail is approver_assurance:mfa_not_enrolled), and an enrolled local account whose last MFA proof is older than the window (expect the step-up 401, then success after the code). Then enrol a non-admin approver through the runbook and confirm it succeeds. Each repeat needs its own pending test request, submitted by someone other than the account being tried, and a decided request cannot be decided again. With RBAC enforcement off (the shipped default), only administrator accounts reach the MFA check (any other account is refused 403 “admin role required” first), so this step can only be run with administrator accounts, and the non-admin enrolment outcome cannot be observed until enforcement is on.

  5. Clean up. Reject every test request that is still pending, from the dashboard Approvals page or with POST /api/approvals/{id}/reject (same gates as approve: a human session with a fresh step-up, a same-site Origin, and not the submitter). A ticket that is already approved cannot be rejected; it stays listed as approved, is refused at use 7 days after its review, flips to expired on the first accepted approval submission after that (#6098), is never deleted, and, for an instruction-origin ticket, dispatches nothing. Record the ids of the approved test tickets in your change record so nobody mistakes them for a real decision later.

  6. To see that approvals are currently being refused, and why, query the audit store (read-only; the approver gate’s refusals carry approver_assurance:<reason> in detail; a refusal by the central step-up check is an mfa.step_up.required row instead):

    SELECT detail, result, count(*) FROM audit_store.audit_events
 WHERE action IN ('approval.approve','approval.reject')
   AND detail LIKE 'approver_assurance:%' AND result IN ('denied','error')
   AND timestamp >= extract(epoch FROM now() - interval '1 hour')
 GROUP BY detail, result ORDER BY 3 DESC;

    A run of approver_assurance:mfa_not_enrolled means approvers still have to enrol; the runbook section Enrolling a non-admin approver has the same query and the enrolment paths. An approver_assurance:mfa_proof_clock_skew row with result=error means OIDC approvers are being refused 503 because their proof is dated after the database clock; the fix is clock synchronisation (see OIDC approvers refused with a clock-skew 503).

Rolling back, or running mixed versions. The approver-assurance change itself adds no schema; the separate scheduled-approval change (#6096, below) adds a data-only approvals step and a command-outbox index, both of which an older binary ignores, so a rollback starts cleanly. What changes is which checks run.

Warning: mixed versions silently weaken the approver control. The enrolled-factor and fresh-proof requirement (#6079) is in force only while every replica that serves /api/approvals/* runs a build that has it. Behind one load balancer, an old replica keeps the weaker gate: it passes an approver who has no enrolled factor, and a build that predates #6061 also accepts bearer and MCP approvals. The same person can therefore get 403 mfa_not_enrolled from one replica and succeed on the next request, and nothing in the response says which replica answered. Rolling those replicas first shortens the exposure window; the gap closes only when approval traffic reaches upgraded replicas only (pin or drain approvals), and stop-all, upgrade, start-all is the default recommendation. Running more than one server replica is unsupported today (the safe-to-scale gate, docs/ha-delivery-matrix.md), so this describes a rolling upgrade and is not a statement that a multi-replica deployment is supported. Rolling back to an older build is a security-policy downgrade, not only a convenience: it re-opens the weaker gate, and tickets decided before the upgrade gain no retroactive assurance.

If you are about to roll, write down each replica’s upgrade time (in UTC) in the change ticket as you go; see “Mixed-version admission” below for why it matters and for the queries that use it.

Behaviour change: plugin-enablement tables and the continuity anchor (plugin_config_store schema v3, ADR-3006)

The plugin_config_store schema moves to v3. It adds the tables of the plugin-enablement store (enablement_rules, enablement_endpoints, enablement_meta) and records a new store id. Nothing flips on upgrade: there is no operator surface yet and nothing issues desired state, so no endpoint’s behaviour changes.

Behaviour change: the macOS agent package installs transactionally and can refuse

This applies to every macOS .pkg, including the release package.

What changed.

What to check.

The opt-in signed-app development lane (Endpoint Security) additionally moves the executable into /Library/Application Support/YuzuAgent/YuzuAgent.app and always runs with --no-auto-update, which preserves the bundle signature. Update that lane by installing a new package (for example through MDM), not over the air. The development .pkg is intentionally unsigned; it is not a customer-distribution lane. Before switching lanes, read the packaging and recovery runbook and the macOS development foundation.

Deprecated: --max-agents / YUZU_MAX_AGENTS has no effect and will be removed (#6165)

What changed. --max-agents (env YUZU_MAX_AGENTS) was never enforced: the server has never capped how many agents connect. It is now marked deprecated. Setting it explicitly logs one startup warning; the default logs nothing. GET /api/v1/settings/server-config still returns max_agents for compatibility.

What to check. Remove --max-agents and YUZU_MAX_AGENTS from your service units, compose files and environment. When the flag is removed in a later release, an explicit --max-agents will stop the server from starting (unknown option), while a leftover YUZU_MAX_AGENTS will be ignored. A malformed leftover value (for example YUZU_MAX_AGENTS=unlimited) still stops the server from starting today, because the value is still parsed as a number. There is no replacement: the server does not cap how many agents connect; spread a large fleet across gateways. Do not use --grpc-max-threads to limit agents: it is a fleet-size ceiling that must stay above the connected-agent count (size it at about 1.5x that count); set below it, every RPC fails.

Version Compatibility

Server VersionMin Agent VersionMin Gateway VersionNotes
0.1.x0.1.00.1.0Initial release family
0.5.x0.5.00.5.0Compiler hardening flags (-fstack-protector-strong, _FORTIFY_SOURCE=2, full RELRO), config file permission enforcement (0600 on Unix), SRI integrity attributes on CDN scripts, configurable trigger limit (default 2000), git-derived version strings, chargen instruction definitions.
0.6.x – 0.9.xsame as 0.5.xsame as 0.5.xNo on-disk format changes from 0.5.x; upgrade directly to 0.10.x.
0.10.x0.10.00.10.0Server-side schema migration runner wired into every SQLite store. Upgrading from 0.9.x or earlier is data-preserving: the first 0.10.x startup stamps each database at schema v1 and runs a one-time legacy compatibility shim for stores that historically added columns via silent ALTER TABLE (api_token_store, instruction_store, patch_manager, policy_store, product_pack_store, response_store). Failed migrations close the affected store’s DB handle and are reported via /readyz with the failed store name — check /readyz, not /livez, to confirm upgrade success.
0.15.x (next)0.12.00.12.0The audit log is admin-only while RBAC is off, and audit reads are bounded (breaking; #5771). Non-admin sessions and tokens get 403 on every AuditLog:Read surface until RBAC is enabled with a role holding it; legacy GET /api/audit clamps limit to 1000 and refuses offset above 10000; MCP query_audit_log rejects wrong-typed filters. Server restart only; see the section below.
0.15.x (next)0.12.00.12.0Gateway listeners and the upstream channel move out of {grpcbox, [...]} (breaking for every gateway config; one edit covers both). The listeners move from {grpcbox, [{servers, ...}]} to {yuzu_gw, [{grpc_listeners, ...}]}, and the channel to the server moves from {grpcbox, [{client, ...}]} to {yuzu_gw, [{upstream_client, ...}]} with the value unchanged. The agent (:50051) and management (:50063) listeners are served by the gateway’s own gRPC layer over Cowboy and the gateway-to-server hop is its own gRPC client over gun; the gateway no longer contains grpcbox. A config that still has a {grpcbox, [{servers, ...}]} or a {grpcbox, [{client, ...}]} block refuses to start, and an old gateway image given the new config starts, reports healthy and binds nothing, so the image and its sys.config are an atomic pair. The gateway now also decides the client address it reports to the server (gateway_observed_peer) and discards what the agent sent: upgrade every gateway before you rely on audit source_ip. No agent or server change. See “Gateway listeners move to grpc_listeners” and “The upstream channel moves to upstream_client” below, and Client addresses behind a proxy.
0.15.x (next)0.12.00.12.0Fleet visualization three-tier layout + talking sockets + curved tube wires (PR 12). /viz/fleet no longer renders machines on a single flat grid. Cubes now stack into three architectural tiers: frontend on the top Y plane, applications in the middle, databases on the bottom. Classification is heuristic — classifyTier reads listener-port hints (DB/web port sets) and process category, priority db > web > app. Behavioural break for automation consumers: if you scripted SIEM rules or dashboards that filter by “where a cube falls in the canvas”, expect tier reassignments after upgrade. The wire change is additive — schema_minor bumps 3 → 4 with a new optional local_addr field on ListenerSocket carrying the kernel-reported bind address (server-side bounded at 64 bytes per field). Strict-validating consumers pinned to schema_minor == 3 should relax their validator to minimum: 3. Loopback-only listeners (127.x, ::1, [::1], ::ffff:127.x) no longer appear on cube surfaces — they’re not reachable from other instances. New talking-socket primitive: each cube grows a ring of cool-blue dots on its BOTTOM face, one per unique outbound (proto, dst_ip, dst_port); hover surfaces talking: tcp → ip:port. Wire geometry changed: cross-machine connections render as THREE.TubeGeometry along a CubicBezierCurve3 with vertical end-tangents instead of 1px THREE.Line — wires drop straight down out of the source cube floor, run mostly-straight through space, and dock straight up into the destination’s listener sphere. Screen-scrapers that parsed wire colour or geometry need updating. Origin RGB AxesHelper removed from the empty-scene scaffold — the three tier planes replace it as the orientation cue. Default camera reframed to (45, 60, 45) looking at the middle tier (was (35, 30, 35) looking at origin); bookmarked URLs will land on the new framing. Bundle size ~70 → ~84 KB. Known limitation: databases on non-standard ports (Postgres on 5431, etc.) misclassify as app tier unless their process is identified as database by the agent’s process classifier. Rolling-upgrade behaviour: during a staged agent rollout, agents on a build older than the echo.fleet_snapshot action have no topology to push and appear in /viz/fleet as dimmed stale cubes until their agent is upgraded — this is expected, not a regression (previously such agents vanished from the visualization entirely once any agent pushed). Kill-switch change: --viz-disable now also 503s the /viz/fleet and /viz/host/<id> page shells, not just the REST endpoints — an operator who sets the flag no longer sees a half-working page; it also writes a server.viz_disabled audit event at boot. Governance Gate 7 hardening (no operator action required): parser field caps on all agent-controlled strings, an IP-claim reclaim window so a crashed agent no longer strands its IPs forever, CAP-1 eviction keyed on the server clock, per-entry isolation in gateway BatchHeartbeat ingest, and a fix for a registration-replay storm under upstream flapping. Scope-walking YAML fromResultSet: DSL (PR-E). Policies whose spec.scope: used a selector: mapping block previously stored an empty scope (matched all devices — the selector was silently ignored). Existing rows are not migrated, but re-creating or re-importing such a policy after upgrade applies the selector as a real predicate and may narrow targeting — review the intended scope before re-import. Inline flow-mapping scope (scope: {fromResultSet: x}) is now rejected; use the block form. Result-set aliases referenced from fromResultSet: must be drawn from the [A-Za-z0-9_.:*-] charset (no spaces/quotes). Inventory freshness gauge now server-clock-stamped (#1685). yuzu_inventory_stale_agents keys on the server’s receipt time, not the agent-supplied collected_at. A one-time migration (v3) at first 0.15.x startup clamps any inventory_state row whose last_seen/first_seen was stamped from a future-skewed agent clock back down to now. Operator-visible: if any agents had future-skewed clocks, the gauge may show a one-time increase post-upgrade as previously-hidden endpoints re-enter the staleness window with a fresh ~48h grace — genuinely active agents fall back out within two daily sync cycles; this is the intended security correction, not an incident. No operator action required (the YuzuInventoryStaleAgents alert ships disabled). Rollback note: downgrading the server below 0.15.x after v3 has run is data-safe (schema unchanged) but new inventory syncs revert to stamping last_seen from agent time, silently re-opening the clock-skew gap for those rows. DEX application performance over time. New per-app, per-version CPU/working-set trend views (DEX → Performance → “Application performance over time”; REST /api/v1/dex/perf/{apps,app,group} + the list_dex_perf_apps/get_dex_app_perf/get_dex_group_app_perf MCP tools; per-device drill GET /api/v1/dex/devices/{id}/app-perf). No action is required to upgrade. Per-application sampling (procperf_enabled) is now on by default on every OS (see “Behaviour change: every Echo capture source is on by default” at the top of this guide), so the views fill on their own while the daily-sync master switch --inventory-disable is unset; a device where sampling was switched off with procperf_enabled=false stays empty until it is re-enabled via echo.configure. Data appears after the first completed UTC midnight on each sampling device (the agent ships a daily summary, not immediately), and the trends lengthen as days accumulate (fleet ≤180 days, group ≤31 days). A freshly-enrolled device, or one with sampling switched off, shows an honest empty state (“no application performance history yet”), not a bug. The per-device drill is also reachable from a dashboard panel on the /device DEX lens (“Application performance over time” — same retained data, no live query, no Execute permission; no upgrade action needed). The per-device drill is behavioural PII — scoped + audited (dex.device.app_perf.view, fail-closed); the fleet and group aggregates suppress any app/version on fewer than 10 devices to a count only (no singling-out). Response/execution reads fail closed on a corrupt RBAC store (#1634, partial). The response readers (query_responses + aggregate_responses MCP tools, GET /api/v1/executions/{id}/visualization, GET /api/responses/{id} / /aggregate / /export) route through a per-agent management-group filter. The only operator-visible change in this release is fail-closed behavior under a corrupt/load-failed rbac.db: these surfaces now return zero rows (the legacy /api/responses/{id}/aggregate returns 503) instead of exposing the whole fleet via the legacy read fallback. Not yet changed: under normal RBAC operation these reads are not management-group-scoped — a holder of global Response:Read still sees all agents’ responses (the filter is inert under the current global gate; the gate change that makes scoping effective is tracked under #1634). RBAC explicitly disabled is unchanged. No operator action required. MCP agentic write surface + A2 discovery + A4 error-shape (R2, #289 / #1794). Five MCP write tools (set_tag/delete_tag/approve_request/reject_request/quarantine_device) ship with a ticket-then-recall approval flow, plus the /api/v1/discover/* discovery family. The approvals store gains additive consumed_at + consumed_by columns (auto-migrated at first startup; no operator action). Breaking wire-shape change: many /api/v1 error bodies that were previously {"error":"<string>"} are now the nested A4 object {"error":{"code","message","correlation_id",…}} — a REST client that read error as a string must migrate to error.code / error.message (see rest-api.md §Error envelope). The MCP write surface is gated behind the existing tier model + a maker-checker approval workflow; audit mcp.<tool> covers every write. NVD CVE-store schema migration (v1→v2) + full CPE version-range matching. The server-side NVD store is reshaped on first startup (flat cve → normalized cve + cve_match); /api/nvd/match now evaluates real CPE version ranges. Operator-visible on upgrade: (a) the migration drops and rebuilds the local CVE mirror — vulnerability-matching coverage is reduced until the next NVD sync completes (rate-limited; up to a few hours without an API key), self-healing and logged with a warning at migration time; (b) GET /api/nvd/status total_cves now counts distinct CVEs (was one row per affected product) so it reads lower after upgrade even once fully synced, and near-zero during the rebuild window — expected, not data loss. Any SIEM/dashboard alerting on the total_cves magnitude should be re-baselined. No config change or action required. NVD CVE sync now actually runs (was dormant). A rate_limit() integer overflow meant the server-side NVD sync slept ~292 years before its first request, so it never populated on any prior deployment (/api/nvd/status total_cves stayed at the built-in seed). It now runs on startup. Operator-visible: (a) the server makes new outbound HTTPS requests to services.nvd.nist.gov — restricted-egress / air-gapped deployments that silently never reached it before may now log connection failures (set --no-nvd-sync to disable, or --nvd-proxy); (b) total_cves grows from the seed as the (keyword-scoped) sync populates. No config change required to benefit. OIDC JWT signature verification now enforced on the Windows server (#1856/#1782, CRITICAL). A Windows server build previously skipped OIDC id_token signature verification entirely — accepting forged RS256/384/512 tokens (session minting / account takeover). It now verifies on every platform via the same OpenSSL path. Operator-visible only if you run a Windows server with OIDC SSO: verification now genuinely depends on the IdP’s JWKS being reachable, so a misconfigured or unreachable jwks_uri that the old Windows build silently tolerated (by accepting everything) will now cause all OIDC logins to fail closed. Before upgrading a Windows + OIDC deployment, confirm the server can reach the IdP jwks_uri and that system clock skew is within tolerance; if OIDC logins start failing fleet-wide post-upgrade, check JWKS reachability first. Local-password and API-token auth are unaffected; Linux/macOS servers already verified and are unchanged. MCP execute_instruction’s zero-agents-reached response schema gained three new fields and a widened status enum (#3424/#3511). The published outputSchema’s oneOf grew from 2 branches to 5: the success branch is unchanged, but the single {status:"no_agents_reached", ...} zero-reach branch is now four mutually-exclusive branches — no_agents_reached (same trigger condition, but now REQUIRES retry_after_ms where it was previously absent/unspecified), plus new quarantined, containment_unreadable, and plugin_not_found — each REQUIRING three new fields that were not previously present: retry_after_ms (a per-branch const — 5000 for the two retryable branches containment_unreadable and no_agents_reached, null for the two permanent branches quarantined and plugin_not_found), agents_quarantined, and agents_unknown_plugin (integer counts, present on every zero-reach branch regardless of which status matched). Strict-validating consumers pinned to the old 2-branch oneOf or the old no_agents_reached-only zero-reach shape will reject a response using one of the three new statuses, or reject the new required fields on the shape they already recognized. Relax any pinned oneOf/const validation to accept the widened schema (GET /mcp/v1/ tools/list always reflects the current, authoritative shape) before upgrading an agentic client with strict response validation. A client that only branches on status == "no_agents_reached" string-equality (rather than treating an unrecognized status as “some non-retryable or retryable failure, check message”) should add explicit handling for the three new values — see docs/user-manual/mcp.md’s execute_instruction row for the full semantics and retry guidance. Alert-rule change (action may be required if you maintain a fork). YuzuDispatchTargetRejected’s exclusion widens from reason!="quarantined" alone to reason!~"quarantined|unknown_plugin" — same reasoning as the #881 exclusion above: a misconfigured automation retrying against an absent plugin is a caller mistake, not a fleet outage, and without the exclusion it clears the alert’s >3/15m threshold on its own. If you maintain a fork of docs/prometheus/yuzu-alerts.yml, apply the same widened exclusion (!~, not =~ — the fork’s rule excludes both reasons, it does not match only them).
0.15.x (next)0.12.00.12.0Fleet visualization three-tier layout + talking sockets + curved tube wires (PR 12). /viz/fleet no longer renders machines on a single flat grid. Cubes now stack into three architectural tiers: frontend on the top Y plane, applications in the middle, databases on the bottom. Classification is heuristic — classifyTier reads listener-port hints (DB/web port sets) and process category, priority db > web > app. Behavioural break for automation consumers: if you scripted SIEM rules or dashboards that filter by “where a cube falls in the canvas”, expect tier reassignments after upgrade. The wire change is additive — schema_minor bumps 3 → 4 with a new optional local_addr field on ListenerSocket carrying the kernel-reported bind address (server-side bounded at 64 bytes per field). Strict-validating consumers pinned to schema_minor == 3 should relax their validator to minimum: 3. Loopback-only listeners (127.x, ::1, [::1], ::ffff:127.x) no longer appear on cube surfaces — they’re not reachable from other instances. New talking-socket primitive: each cube grows a ring of cool-blue dots on its BOTTOM face, one per unique outbound (proto, dst_ip, dst_port); hover surfaces talking: tcp → ip:port. Wire geometry changed: cross-machine connections render as THREE.TubeGeometry along a CubicBezierCurve3 with vertical end-tangents instead of 1px THREE.Line — wires drop straight down out of the source cube floor, run mostly-straight through space, and dock straight up into the destination’s listener sphere. Screen-scrapers that parsed wire colour or geometry need updating. Origin RGB AxesHelper removed from the empty-scene scaffold — the three tier planes replace it as the orientation cue. Default camera reframed to (45, 60, 45) looking at the middle tier (was (35, 30, 35) looking at origin); bookmarked URLs will land on the new framing. Bundle size ~70 → ~84 KB. Known limitation: databases on non-standard ports (Postgres on 5431, etc.) misclassify as app tier unless their process is identified as database by the agent’s process classifier. Rolling-upgrade behaviour: during a staged agent rollout, agents on a build older than the echo.fleet_snapshot action have no topology to push and appear in /viz/fleet as dimmed stale cubes until their agent is upgraded — this is expected, not a regression (previously such agents vanished from the visualization entirely once any agent pushed). Kill-switch change: --viz-disable now also 503s the /viz/fleet and /viz/host/<id> page shells, not just the REST endpoints — an operator who sets the flag no longer sees a half-working page; it also writes a server.viz_disabled audit event at boot. Governance Gate 7 hardening (no operator action required): parser field caps on all agent-controlled strings, an IP-claim reclaim window so a crashed agent no longer strands its IPs forever, CAP-1 eviction keyed on the server clock, per-entry isolation in gateway BatchHeartbeat ingest, and a fix for a registration-replay storm under upstream flapping. Scope-walking YAML fromResultSet: DSL (PR-E). Policies whose spec.scope: used a selector: mapping block previously stored an empty scope (matched all devices — the selector was silently ignored). Existing rows are not migrated, but re-creating or re-importing such a policy after upgrade applies the selector as a real predicate and may narrow targeting — review the intended scope before re-import. Inline flow-mapping scope (scope: {fromResultSet: x}) is now rejected; use the block form. Result-set aliases referenced from fromResultSet: must be drawn from the [A-Za-z0-9_.:*-] charset (no spaces/quotes). Inventory freshness gauge now server-clock-stamped (#1685). yuzu_inventory_stale_agents keys on the server’s receipt time, not the agent-supplied collected_at. A one-time migration (v3) at first 0.15.x startup clamps any inventory_state row whose last_seen/first_seen was stamped from a future-skewed agent clock back down to now. Operator-visible: if any agents had future-skewed clocks, the gauge may show a one-time increase post-upgrade as previously-hidden endpoints re-enter the staleness window with a fresh ~48h grace — genuinely active agents fall back out within two daily sync cycles; this is the intended security correction, not an incident. No operator action required (the YuzuInventoryStaleAgents alert ships disabled). Rollback note: downgrading the server below 0.15.x after v3 has run is data-safe (schema unchanged) but new inventory syncs revert to stamping last_seen from agent time, silently re-opening the clock-skew gap for those rows. DEX application performance over time. New per-app, per-version CPU/working-set trend views (DEX → Performance → “Application performance over time”; REST /api/v1/dex/perf/{apps,app,group} + the list_dex_perf_apps/get_dex_app_perf/get_dex_group_app_perf MCP tools; per-device drill GET /api/v1/dex/devices/{id}/app-perf). No action is required to upgrade. Per-application sampling (procperf_enabled) is now on by default on every OS (see “Behaviour change: every Echo capture source is on by default” at the top of this guide), so the views fill on their own while the daily-sync master switch --inventory-disable is unset; a device where sampling was switched off with procperf_enabled=false stays empty until it is re-enabled via echo.configure. Data appears after the first completed UTC midnight on each sampling device (the agent ships a daily summary, not immediately), and the trends lengthen as days accumulate (fleet ≤180 days, group ≤31 days). A freshly-enrolled device, or one with sampling switched off, shows an honest empty state (“no application performance history yet”), not a bug. The per-device drill is also reachable from a dashboard panel on the /device DEX lens (“Application performance over time” — same retained data, no live query, no Execute permission; no upgrade action needed). The per-device drill is behavioural PII — scoped + audited (dex.device.app_perf.view, fail-closed); the fleet and group aggregates suppress any app/version on fewer than 10 devices to a count only (no singling-out). Response/execution reads fail closed on a corrupt RBAC store (#1634, partial). The response readers (query_responses + aggregate_responses MCP tools, GET /api/v1/executions/{id}/visualization, GET /api/responses/{id} / /aggregate / /export) route through a per-agent management-group filter. The only operator-visible change in this release is fail-closed behavior under a corrupt/load-failed rbac.db: these surfaces now return zero rows (the legacy /api/responses/{id}/aggregate returns 503) instead of exposing the whole fleet via the legacy read fallback. Not yet changed: under normal RBAC operation these reads are not management-group-scoped — a holder of global Response:Read still sees all agents’ responses (the filter is inert under the current global gate; the gate change that makes scoping effective is tracked under #1634). RBAC explicitly disabled is unchanged. No operator action required. MCP agentic write surface + A2 discovery + A4 error-shape (R2, #289 / #1794). Five MCP write tools (set_tag/delete_tag/approve_request/reject_request/quarantine_device) ship with a ticket-then-recall approval flow, plus the /api/v1/discover/* discovery family. The approvals store gains additive consumed_at + consumed_by columns (auto-migrated at first startup; no operator action). Breaking wire-shape change: many /api/v1 error bodies that were previously {"error":"<string>"} are now the nested A4 object {"error":{"code","message","correlation_id",…}} — a REST client that read error as a string must migrate to error.code / error.message (see rest-api.md §Error envelope). The MCP write surface is gated behind the existing tier model + a maker-checker approval workflow; audit mcp.<tool> covers every write. NVD CVE-store schema migration (v1→v2) + full CPE version-range matching. The server-side NVD store is reshaped on first startup (flat cve → normalized cve + cve_match); /api/nvd/match now evaluates real CPE version ranges. Operator-visible on upgrade: (a) the migration drops and rebuilds the local CVE mirror — vulnerability-matching coverage is reduced until the next NVD sync completes (rate-limited; up to a few hours without an API key), self-healing and logged with a warning at migration time; (b) GET /api/nvd/status total_cves now counts distinct CVEs (was one row per affected product) so it reads lower after upgrade even once fully synced, and near-zero during the rebuild window — expected, not data loss. Any SIEM/dashboard alerting on the total_cves magnitude should be re-baselined. No config change or action required. NVD CVE sync now actually runs (was dormant). A rate_limit() integer overflow meant the server-side NVD sync slept ~292 years before its first request, so it never populated on any prior deployment (/api/nvd/status total_cves stayed at the built-in seed). It now runs on startup. Operator-visible: (a) the server makes new outbound HTTPS requests to services.nvd.nist.gov — restricted-egress / air-gapped deployments that silently never reached it before may now log connection failures (set --no-nvd-sync to disable, or --nvd-proxy); (b) total_cves grows from the seed as the sync populates. No config change required to benefit. NVD sync now builds the FULL CVE catalog (newest-first), not ~20 keywords. The sync backfills every CVE published within a configurable window — --nvd-backfill-years / YUZU_NVD_BACKFILL_YEARS (default 8 years; 0 = full history) — newest-first and resumable across restarts, then settles into a periodic last-modified freshness re-check. Operator-visible: the server makes sustained HTTPS requests to services.nvd.nist.gov, the local NVD DB grows into the hundreds of MB, and /api/nvd/status total_cves climbs continuously while backfill_complete stays false until the backfill floor is first reached — last_sync_time advances after every successful fetch window and is not a completion signal (use the new backfill_complete + backfill_oldest_published fields, and the yuzu_nvd_total_cves / yuzu_nvd_backfill_complete metrics, for progress). /api/nvd/status enabled semantics corrected: it now reflects whether sync is configured on, so under --no-nvd-sync it reports enabled:false (was true); a monitor keying on enabled to mean “mirror usable” should check total_cves instead. The initial backfill is NVD-rate-limited (hours without an --nvd-api-key, minutes with one) and resumes where it left off if interrupted. Set --no-nvd-sync to disable, --nvd-proxy for restricted egress. Product matching stays name-based (vendor-precise CPE identity pending ADR-0018). Certificate inventory now reads System/SystemRoot keychains and Linux /etc/ssl/certs natively (no more openssl/security CLI shell-out for those stores). The certificates plugin’s list/details/delete actions are unchanged in output shape (same pipe-delimited columns, same field values) — this is a collection-mechanism change only, verified byte-parity against the prior CLI-based output. The macOS login keychain read has ALSO moved off its governed-shell path in this same release (#3406): it now runs as a pre-split argv through the bounded runner, with the console user’s home directory resolved in-process instead of by the shell. Output shape is unchanged. One operator-visible correction: when the console user cannot be determined — most often a slow or unreachable directory service on a domain-joined Mac — certificates.list/details with store=login or all now return an explicit `not_available
0.15.x (next)0.12.00.12.0Fleet visualization three-tier layout + talking sockets + curved tube wires (PR 12). /viz/fleet no longer renders machines on a single flat grid. Cubes now stack into three architectural tiers: frontend on the top Y plane, applications in the middle, databases on the bottom. Classification is heuristic — classifyTier reads listener-port hints (DB/web port sets) and process category, priority db > web > app. Behavioural break for automation consumers: if you scripted SIEM rules or dashboards that filter by “where a cube falls in the canvas”, expect tier reassignments after upgrade. The wire change is additive — schema_minor bumps 3 → 4 with a new optional local_addr field on ListenerSocket carrying the kernel-reported bind address (server-side bounded at 64 bytes per field). Strict-validating consumers pinned to schema_minor == 3 should relax their validator to minimum: 3. Loopback-only listeners (127.x, ::1, [::1], ::ffff:127.x) no longer appear on cube surfaces — they’re not reachable from other instances. New talking-socket primitive: each cube grows a ring of cool-blue dots on its BOTTOM face, one per unique outbound (proto, dst_ip, dst_port); hover surfaces talking: tcp → ip:port. Wire geometry changed: cross-machine connections render as THREE.TubeGeometry along a CubicBezierCurve3 with vertical end-tangents instead of 1px THREE.Line — wires drop straight down out of the source cube floor, run mostly-straight through space, and dock straight up into the destination’s listener sphere. Screen-scrapers that parsed wire colour or geometry need updating. Origin RGB AxesHelper removed from the empty-scene scaffold — the three tier planes replace it as the orientation cue. Default camera reframed to (45, 60, 45) looking at the middle tier (was (35, 30, 35) looking at origin); bookmarked URLs will land on the new framing. Bundle size ~70 → ~84 KB. Known limitation: databases on non-standard ports (Postgres on 5431, etc.) misclassify as app tier unless their process is identified as database by the agent’s process classifier. Rolling-upgrade behaviour: during a staged agent rollout, agents on a build older than the echo.fleet_snapshot action have no topology to push and appear in /viz/fleet as dimmed stale cubes until their agent is upgraded — this is expected, not a regression (previously such agents vanished from the visualization entirely once any agent pushed). Kill-switch change: --viz-disable now also 503s the /viz/fleet and /viz/host/<id> page shells, not just the REST endpoints — an operator who sets the flag no longer sees a half-working page; it also writes a server.viz_disabled audit event at boot. Governance Gate 7 hardening (no operator action required): parser field caps on all agent-controlled strings, an IP-claim reclaim window so a crashed agent no longer strands its IPs forever, CAP-1 eviction keyed on the server clock, per-entry isolation in gateway BatchHeartbeat ingest, and a fix for a registration-replay storm under upstream flapping. Scope-walking YAML fromResultSet: DSL (PR-E). Policies whose spec.scope: used a selector: mapping block previously stored an empty scope (matched all devices — the selector was silently ignored). Existing rows are not migrated, but re-creating or re-importing such a policy after upgrade applies the selector as a real predicate and may narrow targeting — review the intended scope before re-import. Inline flow-mapping scope (scope: {fromResultSet: x}) is now rejected; use the block form. Result-set aliases referenced from fromResultSet: must be drawn from the [A-Za-z0-9_.:*-] charset (no spaces/quotes). Inventory freshness gauge now server-clock-stamped (#1685). yuzu_inventory_stale_agents keys on the server’s receipt time, not the agent-supplied collected_at. A one-time migration (v3) at first 0.15.x startup clamps any inventory_state row whose last_seen/first_seen was stamped from a future-skewed agent clock back down to now. Operator-visible: if any agents had future-skewed clocks, the gauge may show a one-time increase post-upgrade as previously-hidden endpoints re-enter the staleness window with a fresh ~48h grace — genuinely active agents fall back out within two daily sync cycles; this is the intended security correction, not an incident. No operator action required (the YuzuInventoryStaleAgents alert ships disabled). Rollback note: downgrading the server below 0.15.x after v3 has run is data-safe (schema unchanged) but new inventory syncs revert to stamping last_seen from agent time, silently re-opening the clock-skew gap for those rows. DEX application performance over time. New per-app, per-version CPU/working-set trend views (DEX → Performance → “Application performance over time”; REST /api/v1/dex/perf/{apps,app,group} + the list_dex_perf_apps/get_dex_app_perf/get_dex_group_app_perf MCP tools; per-device drill GET /api/v1/dex/devices/{id}/app-perf). No action is required to upgrade. Per-application sampling (procperf_enabled) is now on by default on every OS (see “Behaviour change: every Echo capture source is on by default” at the top of this guide), so the views fill on their own while the daily-sync master switch --inventory-disable is unset; a device where sampling was switched off with procperf_enabled=false stays empty until it is re-enabled via echo.configure. Data appears after the first completed UTC midnight on each sampling device (the agent ships a daily summary, not immediately), and the trends lengthen as days accumulate (fleet ≤180 days, group ≤31 days). A freshly-enrolled device, or one with sampling switched off, shows an honest empty state (“no application performance history yet”), not a bug. The per-device drill is also reachable from a dashboard panel on the /device DEX lens (“Application performance over time” — same retained data, no live query, no Execute permission; no upgrade action needed). The per-device drill is behavioural PII — scoped + audited (dex.device.app_perf.view, fail-closed); the fleet and group aggregates suppress any app/version on fewer than 10 devices to a count only (no singling-out). Response/execution reads fail closed on a corrupt RBAC store (#1634, partial). The response readers (query_responses + aggregate_responses MCP tools, GET /api/v1/executions/{id}/visualization, GET /api/responses/{id} / /aggregate / /export) route through a per-agent management-group filter. The only operator-visible change in this release is fail-closed behavior under a corrupt/load-failed rbac.db: these surfaces now return zero rows (the legacy /api/responses/{id}/aggregate returns 503) instead of exposing the whole fleet via the legacy read fallback. Not yet changed: under normal RBAC operation these reads are not management-group-scoped — a holder of global Response:Read still sees all agents’ responses (the filter is inert under the current global gate; the gate change that makes scoping effective is tracked under #1634). RBAC explicitly disabled is unchanged. No operator action required. MCP agentic write surface + A2 discovery + A4 error-shape (R2, #289 / #1794). Five MCP write tools (set_tag/delete_tag/approve_request/reject_request/quarantine_device) ship with a ticket-then-recall approval flow, plus the /api/v1/discover/* discovery family. The approvals store gains additive consumed_at + consumed_by columns (auto-migrated at first startup; no operator action). Breaking wire-shape change: many /api/v1 error bodies that were previously {"error":"<string>"} are now the nested A4 object {"error":{"code","message","correlation_id",…}} — a REST client that read error as a string must migrate to error.code / error.message (see rest-api.md §Error envelope). The MCP write surface is gated behind the existing tier model + a maker-checker approval workflow; audit mcp.<tool> covers every write. NVD CVE-store schema migration (v1→v2) + full CPE version-range matching. The server-side NVD store is reshaped on first startup (flat cve → normalized cve + cve_match); /api/nvd/match now evaluates real CPE version ranges. Operator-visible on upgrade: (a) the migration drops and rebuilds the local CVE mirror — vulnerability-matching coverage is reduced until the next NVD sync completes (rate-limited; up to a few hours without an API key), self-healing and logged with a warning at migration time; (b) GET /api/nvd/status total_cves now counts distinct CVEs (was one row per affected product) so it reads lower after upgrade even once fully synced, and near-zero during the rebuild window — expected, not data loss. Any SIEM/dashboard alerting on the total_cves magnitude should be re-baselined. No config change or action required. NVD CVE sync now actually runs (was dormant). A rate_limit() integer overflow meant the server-side NVD sync slept ~292 years before its first request, so it never populated on any prior deployment (/api/nvd/status total_cves stayed at the built-in seed). It now runs on startup. Operator-visible: (a) the server makes new outbound HTTPS requests to services.nvd.nist.gov — restricted-egress / air-gapped deployments that silently never reached it before may now log connection failures (set --no-nvd-sync to disable, or --nvd-proxy); (b) total_cves grows from the seed as the sync populates. No config change required to benefit. NVD sync now builds the FULL CVE catalog (newest-first), not ~20 keywords. The sync backfills every CVE published within a configurable window — --nvd-backfill-years / YUZU_NVD_BACKFILL_YEARS (default 8 years; 0 = full history) — newest-first and resumable across restarts, then settles into a periodic last-modified freshness re-check. Operator-visible: the server makes sustained HTTPS requests to services.nvd.nist.gov, the local NVD DB grows into the hundreds of MB, and /api/nvd/status total_cves climbs continuously while backfill_complete stays false until the backfill floor is first reached — last_sync_time advances after every successful fetch window and is not a completion signal (use the new backfill_complete + backfill_oldest_published fields, and the yuzu_nvd_total_cves / yuzu_nvd_backfill_complete metrics, for progress). /api/nvd/status enabled semantics corrected: it now reflects whether sync is configured on, so under --no-nvd-sync it reports enabled:false (was true); a monitor keying on enabled to mean “mirror usable” should check total_cves instead. The initial backfill is NVD-rate-limited (hours without an --nvd-api-key, minutes with one) and resumes where it left off if interrupted. Set --no-nvd-sync to disable, --nvd-proxy for restricted egress. Product matching stays name-based (vendor-precise CPE identity pending ADR-0018). Certificate inventory now reads System/SystemRoot keychains and Linux /etc/ssl/certs natively (no more openssl/security CLI shell-out for those stores). The certificates plugin’s list/details/delete actions are unchanged in output shape (same pipe-delimited columns, same field values) — this is a collection-mechanism change only, verified byte-parity against the prior CLI-based output. The macOS login keychain is unchanged (still reads via the existing governed-shell path, a deliberate exception — see docs/agent-spawn-sink-manifest.md). No operator action required; mixed old/new-agent fleets and rollback are safe (no server-side or schema change).
0.15.x (next)0.12.00.12.0PatchManager now runs on PostgreSQL (ADR-0062). /api/patches/* (patch inventory + deployment tracking) moved off its own patches.db SQLite file onto the shared Postgres substrate (schema patch_manager). No data carries over from a pre-Postgres install (fresh-start-by-default, ADR-0009) — any deployment record that existed before upgrade is gone and must be re-created via POST /api/patches/deploy. Server startup now fails closed if the patch_manager schema can’t be created/opened — a posture upgrade from the SQLite era, where construction was unconditional/best-effort and no caller ever checked whether the store had actually opened; confirm success via /readyz (patch_manager is now reported by both /readyz and /healthz, absent from both before this release). Removed: automatic patch-deployment orchestration. PatchManager::execute_deployment() (the scan → install → verify → reboot workflow) had zero production callers on any released build — nothing ever wired a dispatch/OS-lookup callback to it — and is deleted, not ported; POST /api/patches/deploy still creates a deployment + per-target rows, but nothing in the server drives them through that workflow automatically (see #3669, filed alongside this change, and docs/capability-map.md §8.3/§8.4/§8.6). Patch inventory (GET /api/patches) is separately unwired: record_patches(), the only method that writes it, also has no production caller — this predates the migration and is not something upgrading changes — so GET /api/patches returns empty in every real deployment today; see docs/capability-map.md §8.5/§8.7 and #3676. No operator action required beyond re-creating any in-flight deployment after upgrade.
0.15.x (next)0.12.00.12.0WorkflowEngine now runs on PostgreSQL (ADR-0064). /api/workflows* and /api/workflow-executions/* moved off workflows.db SQLite onto the shared Postgres substrate (schema workflow_engine). No data carries over from a pre-Postgres install (fresh-start-by-default, ADR-0009) — any workflow definition and its execution history that existed before upgrade is gone; re-create workflows via POST /api/workflows (or product-pack re-install). Server startup now fails closed if the workflow_engine schema can’t be created/opened — a posture upgrade from the SQLite era, where construction was unconditional/best-effort and no caller ever checked whether the store had actually opened; confirm success via /readyz (already reported before this release) and /healthz (newly reported — was absent before this release). Delete semantics changed: DELETE /api/workflows/:id now soft-deletes. The response shape is unchanged (`{“deleted”: true
0.15.x (next)0.12.00.12.0Guardian file-hash max_bytes now has a hard ceiling (#2233). An authored file-hash-equals rule’s max_bytes (the hashing-DoS cap) was previously accepted unbounded from the authoring API. It is now clamped to 1 GiB (kMaxFileHashBytes) on the agent, and the server rejects a new/edited rule authoring a value above that ceiling in either JSON wire form (400). Operator-visible only if you have a PRE-EXISTING file-hash-equals rule authored (before this release) with max_bytes above 1 GiB, watching a file at or above that size: after upgrade, that file reports <oversize> instead of being hashed — a compliance-verdict change with no authoring-time signal (the rule already exists, so the new server-side reject cannot retroactively catch it). List your rules via GET /api/v1/guaranteed-state/rules (the route returns every rule; there is no server-side filter), check any file-hash-equals rule for max_bytes over 1073741824, and re-author within the ceiling if the larger cap was intentional. No operator action required otherwise. DEX and management-group reads now fail closed on a degraded read instead of answering a healthy/empty result (#4855, #1762) — see “Behaviour change: DEX device score and management-group member reads now fail closed on a degraded read (#4855, #1762)” below.
0.15.x (next)0.12.00.12.0BatchHeartbeatResponse gains unknown_session_ids and unknown_session_ids_truncated (#1197), and the gateway now acts on them - restart the gateway to deploy (this disconnects every agent it holds, and released v0.13.0 and v0.14.0-rc6 agents stayed wedged after a graceful gateway restart in testing, so upgrade agents to a build with the #5183 fix first or expect to restart the agent service on them; see “Agent dependency” in What happens when the server restarts); no wire change and no new required configuration. The server lists, per BatchHeartbeat, the session ids it does not hold, and the new gateway reads that list and re-registers exactly the sessions it still holds (the registration-replay drip, bounded by a per-session guard and a queue cap). The behavioural change is that, after a server restart, agents behind a new gateway become known to the server again without operator action; before, they stayed unknown (/health agents.online stayed 0 in the observed runs). Any pairing interoperates: an OLD server never sends the fields, so a new gateway sees an empty list and does nothing; an OLD gateway ignores the added response fields (proto3 unknown fields, field numbers 2 and 3) and keeps the old behaviour. Recovery needs the new gateway and a server that sends the fields. Observed on a local rig: the agent was known again about 17 s after the server was healthy, on its first heartbeat after that; after an outage long enough to open the gateway circuit breaker, recovery waits for the breaker’s remaining backoff (58 s observed after a 300 s outage with 1 agent; with 10 and 30 agents the breaker opened in later runs and the breaker’s own replay recovered them, with all agents online 85.2 s and 50.4 s after the server was healthy; the backoff is capped at 300 s, and steps above 160 s were not observed). A second operator-visible change is a new yuzu_server_gateway_route_desync_total{op="batch_heartbeat",outcome="malformed_session_id"} series (pre-seeded at 0, expected to stay 0, no alert): over-length (more than 64 bytes) unknown session ids in a BatchHeartbeat are now counted under that series, per entry, instead of under op="renew_leases", outcome="unknown_session". The gateway adds three counters and two optional application-env tunables, described under What happens when the server restarts, which also states what the reconcile does not promise (fleet completion inside a route lease, dispatch reachability after the session is adopted, several core replicas).
0.15.x (next)0.12.00.12.0Gateway session ids are now resolved against the durable Postgres route directory (HA WS-5 slice 2, ADR-2002 §7f); server restart only, no gateway restart, no flag, no proto field. (This release also carries a separate gateway_route_store v5 migration, a one-time exclusive lock at the first start: see the v5 section below.) Heartbeat, inventory and stream-status RPCs look a presented session_id up in gateway_route_store.agent_routes instead of one replica’s in-memory map. After a server-only restart the server still lists directory-known sessions in unknown_session_ids so the gateway replays ProxyRegister and rebuilds the local registry (drip-paced; ~1200 s for 10k agents at 100 ms), while leases keep renewing; this needs a gateway with the #1197 reconcile (an older gateway gives the pre-reconcile behaviour). Rolling deploy: an old replica reports directory-known sessions as unknown, so drain old replicas first. Run exactly ONE server replica until the safe-to-scale gate lifts (ADR-2002 §7c); this change does not lift it. Rollback is safe (the previous binary boots; see the v5 section’s rollback bullet for what it does on a race). A degraded directory read is fail-soft for heartbeats; a NotifyStreamStatus CONNECTED during it is accepted only for a session this replica holds locally and is otherwise refused (connected_unverified; the agent regains dispatch at its next reconnect). A session id matching more than one live row is refused and pages YuzuGatewaySessionAmbiguous; a ProxyRegister replay presenting an id another agent’s row holds is refused and pages YuzuGatewaySessionHeldByOtherAgent. See gateway.md.
0.15.x (next)0.12.00.12.0Gateway heartbeat admission is now bound to the connection that opened the session (gateway restart required; breaking for L7-fronted gateways). The gateway admits an agent Heartbeat only on the HTTP/2 connection that opened the session’s Subscribe stream; any other heartbeat is answered NOT_FOUND and not forwarded. Deploy with a gateway restart, and check for an HTTP/2-terminating hop in front of :50051 first. A rejected agent recovers by re-registering. That logic exists from v0.13.0, but the released v0.13.0 and v0.14.0-rc6 agents wedge in their reconnect path with default settings (bug #2182, fixed by PR #5183, shipped in 0.15.0) and recover only with --no-auto-update; v0.12.0 never recovers by itself (observed; older versions were not tested; v0.14.1 does not contain the fix and was not tested). Upgrade the agents first, then the gateway, with 0.15.0 or later (the #2182 fix); until the agents are upgraded, restart an agent that stays rejected. Agents that do not connect through the gateway are not affected by this requirement (see Older agents). No wire or server change. See the section “Breaking: gateways fronted by an HTTP/2-terminating proxy or mesh sidecar” below and Heartbeat admission.
0.15.x (next)0.12.00.12.0--gateway-upstream now refuses to start without gateway peer authorization (breaking for own-certificate and plaintext installs). The gateway-upstream service admits only a gateway whose certificate key (SPKI SHA-256) is pinned, which carries serverAuth and is inside its validity window; the server exits at boot, before binding a port, when --gateway-upstream is set and that cannot be arranged. Generated default certificates are pinned automatically; your own certificates need --gateway-peer-pin / --gateway-peer-pin-file first; a plaintext (--no-tls) rig needs --insecure-gateway-peer; the Linux package unit passes --gateway-upstream unconditionally. Agents and gateways are unaffected on the wire. See the section “Breaking: --gateway-upstream now refuses to start without gateway peer authorization” below.
0.15.x (next)0.12.00.12.0The gateway now holds stream-status notifications instead of dropping them (#4632; gateway restart required). Deploy it with a gateway restart, which disconnects every agent the node holds, so read “Upgrade day” and “Agent dependency” in What happens when the server restarts first (released v0.13.0 and v0.14.0-rc6 agents do not recover by themselves after a gateway restart). Read the Gateway upgrade note for #4632 before you restart: a gateway whose configured cluster_id cannot be represented as UTF-8 now refuses to start. There is no hot code load for this change: the upstream client’s code_change is the identity function while its state gained fields, so deploy it by restart, not by a hot upgrade (INFERRED from the code, not run). No wire change and no change to the minimum agent or server version: NotifyStreamStatus is unchanged, so any pairing interoperates. What changes: a CONNECTED or DISCONNECTED notification that arrives while 10 are in flight or the circuit breaker is open is held in a bounded, memory-only store and sent first in first out, where the old gateway dropped it; when the breaker goes half open the new gateway sends one held notification as the probe, where the old one let up to 10 notifications through; and an agent-supplied peer address is sanitised and capped before it is held (a value that is not valid UTF-8 becomes invalid, and one over 255 bytes is cut at a character boundary). New settings notify_pending_max (default derived from the agent listeners: three per connection they admit, the sum of their limits.max_connections, at least 15000, so 15000 on a listener without a ceiling and 180000 with the 60000 ceiling the packaged, production and reference configs set; sys.config.prod and the reference compose config also set 180000 explicitly; valid range 1 to 1000000; a value outside the range or not an integer falls back to the default with a WARN that names the key; see “Upstream client tuning” in gateway.md); size it at least as large as the largest Subscribe burst you expect; and notify_probe_timeout_ms (default 15000, valid range 100 to 300000, not present in any shipped sys.config, same fallback rule), the gateway’s own deadline for the half-open notification probe: a probe worker that has not reported by then is killed and the probe fails like any failed probe. The other behaviour changes of the same restart are listed in the upgrade note below the table. Metrics: new gauges yuzu_gw_upstream_notify_pending and yuzu_gw_upstream_notify_inflight and counter yuzu_gw_upstream_notify_coalesced_total. The reason label of yuzu_gw_upstream_notify_dropped_total is now the closed set at_capacity and unknown: the value circuit_open is retired, so during a rolling upgrade old nodes still export circuit_open and new nodes export at_capacity, and a rule or dashboard that sums the family sees both. A rule or panel of your own that selects reason="circuit_open" stops matching once every node is upgraded: use yuzu_gw_upstream_notify_pending for held notifications and reason="at_capacity" for lost ones. The rollback is the previous gateway release. A rollback is itself a gateway restart: it disconnects every agent the node holds and carries the same agent-reconnect cost as the upgrade (see “Upgrade day” and “Agent dependency” in What happens when the server restarts, and the released-agent caveat above), so weigh it against the problem you are fixing. It also brings back the earlier behaviours: notifications dropped at 10 in flight or with the breaker open, a request that the gateway’s own validation or encoder rejected (or a server status that means the request itself was wrong) counted as an upstream failure that can open the breaker, and no cluster_id boot refusal. The pending store is memory only, so nothing persists and nothing needs to be cleaned up; the notifications held at that moment are lost, as they were dropped before. Runbook rows and the measured evidence: Stream Status Notification and the evidence record.
0.15.x (next)0.12.00.12.0Gateway: a planned stop needs up to 40 s, deleting a pin file no longer withdraws the pin, and the management listener, circuit breaker, connection ceilings and certificate-expiry alerts change (#5695; deployed by the same gateway restart as the listener move above, so read “Upgrade day” and “Agent dependency” in What happens when the server restarts first: agents without the #5183 fix do not recover by themselves after a gateway restart). (1) Stop timeout. On SIGTERM the gateway now ends its agent sessions and sends their DISCONNECTED notices to the server before its processes stop, within shutdown_flush_budget_ms (default 20000 ms, valid 0 to 600000). Every shipped compose file sets stop_grace_period: 40s on the gateway service and deploy/systemd/yuzu-gateway.service sets TimeoutStopSec=40. A supervisor of your own must allow a stop timeout above shutdown_flush_budget_ms, with room for the processes to stop after it (40 s for the default budget): Docker’s default is 10 s (a docker run, or a compose file you maintain without stop_grace_period), a Kubernetes pod’s default terminationGracePeriodSeconds is 30 s. Raise the timeout and the budget together. A stop cut short by SIGKILL sends no notices, and the server keeps those sessions listed as online until the agents register again. See Planned stop. (2) Pin files. A key read from a {cert_file, Path} entry of mgmt_peer_pins now stays pinned when the file later cannot be read or parsed (reported by yuzu_gw_cert_file_readable{role="mgmt_peer_pin"} reading 0 and by YuzuGatewayMgmtPinUnresolved), so deleting, emptying or corrupting the file no longer withdraws the pin: it does not revoke the pinned key, which the gateway keeps admitting. To change or revoke the pinned key, replace the file with the new certificate (picked up within mgmt_pin_refresh_ms, 5 s by default). Removing the entry from mgmt_peer_pins also revokes it, but applies only at the next gateway restart, which disconnects every agent on the node, and agents without the #5183 fix stay wedged after it. The kept key is only ever one that was pinned; a file that has never been readable since the gateway started still pins nothing (fail closed). Withdrawing a certificate without a file is the serial deny-list mgmt_peer_denied (#6016, read at boot, so a change applies at the next gateway restart, which disconnects the node’s agents). See Certificate rotation and expiry and Authorizing the server by its identity. (3) Management listener. A listener that routes the ManagementService defaults to 4096 concurrent HTTP/2 streams (an agent listener keeps 1024) and 100000 received frames per 10 s (Cowboy’s own limit, kept on the agent listener, is 10000), because the server sends a whole command fan-out over one connection; it has no per-connection memory ceiling unless limits.max_connection_memory_bytes sets one. A value in the entry’s http2 or limits map still wins, so a management entry you wrote with a lower max_concurrent_streams keeps it. See gRPC listeners. (4) Connection ceilings and held notifications. The packaged release config (gateway/config/sys.config, which the release packages and the systemd unit run) sets limits.max_connections to 60000 on the agent listener and 256 on the management listener, the values sys.config.prod sets; a listener without the key has no connection ceiling. Change the agent value together with the unit’s LimitNOFILE. notify_pending_max defaults to three per agent-listener connection (at least 15000), so 180000 with that ceiling, and sys.config.prod and the reference compose config set 180000 explicitly. (5) Circuit breaker. The longest wait before the breaker probes the server again, circuit_breaker_max_reset_timeout_ms, defaults to 30000 ms (was 300000), and every shipped config now sets 30000. A config copied from an earlier release sets 300000 explicitly and keeps it: change it, because YuzuGatewayUpstreamCircuitOpened assumes the breaker reopens at least every 4 minutes or so. See The upstream client. (6) Alerts (check your Alertmanager routes and silences). YuzuCertificateExpiringSoon (server, warning) now fires from 30 days before expiry instead of 7; the 7-day band is the new critical YuzuCertificateExpiryImminent, and YuzuCertificateExpiryCritical stays at 24 hours. The gateway gains the same three steps for its own certificate files (YuzuGatewayCertificateExpiringSoon, YuzuGatewayCertificateExpiryImminent, YuzuGatewayCertificateExpiryCritical). YuzuGatewayMgmtAuthRejected no longer counts reason="pin_mismatch": that reason has its own YuzuGatewayMgmtPinMismatch, which stays quiet for 4 minutes after the gateway publishes a rotated pin; the old name keeps its silences and routes for the other reasons. The server’s YuzuGatewayForwardUnauthenticated is unchanged and can still fire for up to an hour after a rotation that had a command in flight (#5988 removes that cause). New gateway rules: YuzuGatewayUpstreamCircuitOpened (the breaker stays open for 3 minutes, or keeps flapping), YuzuGatewayUpstreamSlow, YuzuGatewayCertificateFileUnreadable, YuzuGatewayMemorySheddingActive, YuzuGatewayMemorySheddingTransitionFailing, YuzuGatewayMemorySheddingInactive (shedding enabled but not running), YuzuGatewayListenerNearConnectionCeiling, YuzuGatewayFileDescriptorsNearLimit and YuzuGatewayProcessBacklogged. (7) Memory-shedding marks are checked at boot. With memory_shedding enabled, a low mark not below the high mark, a mark at or above the gateway’s memory limit, or a mark that is not positive now refuses to start (Refusing to start: {yuzu_gw, memory_shedding}: ...); before, such a config booted with shedding silently off. Setting only high_water_bytes (or only low_water_bytes) is now valid: the other mark is derived from it. Check a memory_shedding block you wrote against the gateway’s memory limit before the upgrade. See “Node-wide memory load shedding” under gRPC listeners. Rollback: the previous gateway release with its own config (see the listener move above); a rollback is itself a gateway restart, with the same agent-reconnect cost.
0.15.x (next)0.12.00.12.0registry plugin 2.0.0: typed, verified writes; registry links refused (breaking; agent plugin). Once the 1.1.0 set_value, delete_value and delete_key definitions are in the store (an existing install keeps the 1.0.0 ones until they are re-imported; see the same page’s “Breaking” section), they also reach an agent whose registry plugin is still 1.x, and nothing compares a plugin or agent version at dispatch: such an agent writes an EMPTY REG_SZ for a data_hex-only set_value (with type=REG_DWORD and data_hex only it throws and writes nothing), stores the new value types as REG_SZ text, answers `status
0.15.x (next)0.12.00.12.0The plain-HTTP port answers the four health probes when HTTPS is on (#5906). GET/HEAD on /livez, /readyz, /health and /api/health on the redirect port now return 200 (or 503 while the HTTPS listener is not serving, the server is draining, or Postgres is down) instead of 301, in cleartext; every other request still redirects. Server restart only; see the section below.
0.15.x (next)0.12.00.12.0The built-in server certificate is re-issued once with a server identity URI (#6016). Same root, same key, so existing mgmt_peer_pins stay valid and the gateway needs no action for the re-issue. The gateway in this release can also admit the server by that identity (mgmt_peer_auth_mode, mgmt_peer_identity, mgmt_peer_denied; new reasons on yuzu_gw_mgmt_auth_rejected_total, new counter yuzu_gw_mgmt_auth_admitted_total, new alert YuzuServerIdentityUriAbsent): the identity is an explicit opt-in: the shipped reference configs and an unset mode are all pin, so nothing changes until you set identity_or_pin or identity. The old certificate shows as revoked with reason superseded in the CA inventory and the next CRL (bookkeeping, not a compromise). New audit action server.default_certs_server_identity_upgraded, new gauge yuzu_server_identity_uri_present. See Authorizing the server by its identity and “Behaviour change: the built-in server certificate is re-issued once with a server identity URI”.
0.15.x (next)0.12.00.12.0Breaking, fresh deployment only: stored responses of Forensics-tier plugins are served only to callers holding Forensics:Read (#4990). Applies to every stored-response read path (REST, MCP, dashboard) and to the server-side consumers of those rows (bundle collation, result-set membership, policy checks, the live output stream, execution error text), with the limits listed under “Known limits” in that section; other readers see the rows as absent, with counts and caps computed over what they may read, so an empty result no longer implies no data. Viewer, Operator, PlatformEngineer and ITServiceOwner, and with RBAC off every non-administrator session, no longer see these rows. New built-in ForensicsAnalyst role (read-only, assign it fleet-wide with RBAC on) and new audit action response.forensics.read. A read whose authority cannot be determined is refused with a retryable error rather than a partial result. This release is a hard cutover: it supports initial deployment into a fresh database only and provides no upgrade path from an earlier server’s database (see “Deployment contract” in the section; the agent and gateway columns above are the minimum agent and gateway versions, not a supported upgrade starting point for the database). See “Behaviour change: stored responses of Forensics-tier plugins need Forensics:Read (#4990, fresh deployment only)“.
0.15.x (next)0.12.00.12.0MFA step-up is enforced by one central check on 37 named routes, before the route’s own checks (breaking order of checks; server restart only, no agent or gateway change, no migration). A stale or missing MFA proof is now answered with the 401 step-up envelope before the route’s own 403 (permission, token_session, not_local; a cross-site cookie write is answered 403 by the server-wide same-site check before the step-up check); engine principals receive 403 on every step-up-gated route regardless of grants (audited as mfa.step_up.required, no longer as the per-route handler denial row); a server with no wired authentication store answers 503 on these routes, and an unreadable store answers 503 to local sessions; POST /api/settings/mfa/recovery-codes and POST /api/settings/mfa/disable are newly gated; SAML sessions still cannot satisfy step-up (403, unless the window is disabled with --mfa-step-up-window-secs 0). The server now exits with status 1 at boot when a mutating route has no step-up classification, which can only happen on a fork or custom build (a restart policy turns it into a restart loop; roll back to recover). Other mutating routes and all MCP tools are not MFA-step-up gated in this release. New metric yuzu_auth_mfa_step_up_gate_total and alerts YuzuStepUpGateUnavailable and YuzuStepUpWindowDisabled. See Breaking: MFA step-up is checked once.
0.15.x (next)0.12.00.12.0Seven runtime-config keys are administrator-only on PUT /api/config/{key} (breaking for non-administrator and MCP-tier callers; server restart only, no agent or gateway change, no migration). plugin_signing_required, dex_alert_routing, dex_blast_min_devices, dex_blast_window_seconds, dex_blast_cooldown_seconds, dex_cohort_export_key and audit_retention_days now answer 403 to a supervised-tier MCP token and, with RBAC on, to a non-administrator role that holds Infrastructure:Write; every other config key is unchanged. Server restart only; see the first section on this page.
0.15.x (next)0.12.00.12.0The runtime-configuration read shows a caller only the overrides it could write (breaking for MCP-tier and non-administrator readers; server restart only, no agent or gateway change, no migration). GET /api/config, GET /api/v1/config and the MCP tool get_runtime_config leave out the oidc_*, plugin_signing_required, DEX alert and audit_retention_days overrides unless the caller could write them; an administrator account’s GET /api/config body is unchanged apart from log_level reading warn. heartbeat_timeout is limited to 30 to 86400 seconds on every write route. Mixed-version fleets are not a concern (server-side only).
0.15.x (next)0.12.00.12.0The activity-record feature is renamed Echo (breaking; hard cut, no aliases). The agent plugin is echo (was tar), and the REST routes, MCP tools, metrics, audit actions, instruction ids, heartbeat tags and the result-set matcher kind carry the new name. Upgrade the server before the agents: the server accepts an agent that still reports tar and writes the old name on the wire to it (an upgrade-window shim, also on the gateway directory route), but an upgraded agent talking to a server that has not been upgraded has no fallback and every Echo dispatch to it is refused with plugin_not_found. Deliver the agent by package installer: an OTA update replaces the binary only, and the agent defers adopting tar.db as echo.db until echo.<ext> is installed. A carried-forward server database (not a supported upgrade path under the deployment contract) has its stored references rewritten in eleven stores on the first start. See “Behaviour change: the activity-record feature is renamed Echo (tar.db adopted as echo.db; stored references migrated)“.
0.14.x0.12.00.12.0Fleet visualization intra-cube edges (PR 8). /viz/fleet now draws faint white lines (opacity 0.3) inside each machine cube connecting process dots that are reciprocal ends of a loopback TCP socket (127.0.0.1 / ::1). Two operator-visible changes: (a) wire shape — /api/v1/viz/fleet/topology schema_minor bumps 1 → 2 and a new optional dst_pid field appears on scope: local connection edges. Renderers that ignore unknown keys per the contract see no break; strict-validating consumers pinned to schema_minor == 1 should relax their validator to minimum: 1. (b) dropped unmatched halves — unpaired Local-scope edges (kernel snapshot race during teardown, agent’s 4096-connection cap cutting a partner) are now dropped server-side before serialisation. Integrations counting connections array length per machine as a proxy for active IPC pairs should re-baseline after upgrade; the count trends marginally lower. Lines appear only when the host has active loopback flows (e.g. Prometheus scraping node_exporter, a client talking to local Redis / Postgres); a fresh agent with no inter-process loopback shows process dots but no lines — expected, not a regression. Windows agent (#5196): set update-signing options in the YuzuAgent service’s Environment registry value, not its binary path. Every installer run rewrites the binary path and silently drops them, and uninstalling deletes the Environment value, so a deployment that uninstalls first must set it again (Windows: the service’s Environment value in server-admin.md). Agent installers from 0.14.0-rc1 to rc5 also stop with exit code 7 wherever PowerShell runs in Constrained Language Mode; use this release’s. Before upgrading Windows agents: this installer stops with exit code 7, naming the reason in its /LOG= file, if C:\ProgramData\Yuzu\agent-certs exists but is not secured (for example a folder created or pre-staged by hand, or one a Group Policy adds permissions to), if a file in it is not owned by Administrators or SYSTEM, or if it or anything in it is a junction, symbolic link, hard link or subdirectory (a backup subfolder, say). The agent service is left running when it stops. Provision the bundle after installing, by copying it in as an administrator. The trust-anchor procedure block in server-admin.md stops on the same conditions, naming the reason (and on a healthy or rc1–rc5 endpoint completes, repairing rc files, as long as each file in it is owned by Administrators or SYSTEM; icacls "<file>" /setowner *S-1-5-32-544 /L fixes one you placed yourself): run it on a few endpoints first, and pilot the upgrade before a fleet-wide push. Linux native packages need a recent distribution: the server needs Ubuntu 26.04 or Fedora 42 class, the agent Ubuntu 24.04 class or newer; Ubuntu 22.04, Debian 12 and RHEL/Rocky 9 are not supported by the native packages. Check before upgrading older hosts, or use the container images (Supported Platforms in the user manual README, #5143).
0.13.x0.12.00.12.0Fleet visualization process layer. /viz/fleet now renders interior process dots inside each machine cube, coloured by category (system/browser/database/web/runtime/other) — no operator action required, but operators upgrading from a 0.12.x build will see the dashboard suddenly populated with thousands of small spheres on next page load. Process data was already collected via echo.fleet_snapshot since 0.12.x; PR 7 only renders it. To suppress process visibility for specific agents (privacy-sensitive hosts, regulated workloads), set process_enabled=false on those agents via echo.configure — this also suppresses their dots on the visualization. Hover a dot to see pid/name/user/category; agent-controlled string fields are HTML-escaped and length-clamped before render. Per-cube dot count is soft-capped at 1000 for graceful degradation on heavily-threaded hosts; the cube tooltip still shows the true reported count.
0.12.x0.12.00.12.0Build-time content auto-import. All YAML files in content/definitions/ (217 InstructionDefinitions) and content/packs/ (10 InstructionSets at this version) are now embedded in the server binary and auto-imported on every startup. Existing operator-customised definitions with matching IDs are NEVER overwritten — conflicts are silently skipped. Behaviour change for upgrades: definitions that an operator previously DELETED via the REST API or dashboard will reappear after upgrade because the auto-import treats a missing row as “needs creation”. To permanently suppress a shipped definition, set enabled: false via the dashboard or PATCH /api/v1/definitions/{id} rather than DELETE-ing the row. Each auto-import write emits an audit_events.action="content.bundled_import" row with principal=system so operators can audit which definitions were inserted at boot. Yuzu dark navy palette + Inter webfont (visual change every operator sees) and Apache ECharts chart renderer (replaces bespoke SVG; same payload contract — no operator migration required) ship in the same release.

Rule of thumb: agents and gateway should be the same minor version as the server, or one minor version behind. The server is always upgraded first. Upgrading the server first restarts it while gateways stay connected. A gateway at this version re-registers the sessions the server reports unknown, so a server-only restart recovers without operator action once both sides are upgraded; see What happens when the server restarts. Behind a gateway that does not yet read that report (which includes the server restart that ships this fix, because the server is upgraded first; restart the gateway after upgrading it, and read the next sentence first), agents can read offline after a server restart (observed on one local rig after a SIGKILL restart; graceful upgrade restarts were not tested), and dispatch worked only for the remaining route lease (up to about 90 s). A gateway restart disconnects every agent that gateway holds, and the agent then has to register again by itself: in a graceful gateway restart test, released v0.13.0 and v0.14.0-rc6 agents with default settings stayed wedged and v0.12.0 never re-registered (bug #2182, fixed by #5183, shipped in 0.15.0), while a build with the fix re-registered in 11 to 12 s. Upgrade the agents to 0.15.0 or later (the first release with #5183) before the gateway restart, or expect to restart the agent service on older agents behind it; see “Agent dependency” in What happens when the server restarts.

Behaviour change: new Crypto page, REST routes and MCP tools are reachable with Inventory:Read

What changed. The server adds the Posture > Crypto page (/crypto), two REST routes (GET /api/v1/crypto/libraries and /api/v1/crypto/libraries/devices) and two MCP tools (get_crypto_libraries, list_crypto_library_devices). This release also adds a CycloneDX 1.7 export of the same data (GET /api/v1/crypto/cbom and the MCP tool export_crypto_cbom), also reachable with Inventory:Read alone. All of them are gated by Inventory:Read alone, so any custom role that already holds Inventory:Read now reaches them.

Action. None required. To keep a custom role out of the new surface, remove Inventory:Read from it. Rolling back removes the page, routes and tools; nothing is migrated.

Behaviour change: gateway sessions are unique per agent; a one-time migration (gateway_route_store v5, #5683)

Short answer. A normal upgrade needs no action. The first start takes an exclusive lock on agent_routes (about 2.4 s per million rows) and the health probes are silent meanwhile. If a blocking session holds the lock past 30 s the server exits and a supervisor retries about every 35 s until it is cleared. Details follow.

What changed. In a gateway-fronted deployment the route directory enforces at most one non-NULL route row per session id by a database unique index; the tombstoned-victim window is not closed; run one replica. Migration v5 of the gateway route directory adds a unique index on gateway_route_store.agent_routes(session_id) (tombstoned rows are not indexed) and drops the old non-unique one. Two agents racing to adopt one unheld session id used to both succeed; now the second is refused (FAILED_PRECONDITION, counted as yuzu_server_gateway_route_desync_total{outcome="session_held_by_other_agent"}, alert YuzuGatewaySessionHeldByOtherAgent).

Action. None for a normal upgrade. On a very large directory, schedule the first start for a quiet period and raise any startup probe timeout. To learn WHICH agents share a session id (the migration erases the association), run the duplicate query from step 1 of gateway.md “Recovering a missing unique session index” before the first start and keep the output with the change record.

Behaviour change: new ca_store indexes, built during the upgrade

What changed. The CA store’s migration v5 adds three indexes on ca_issued (one row per certificate Yuzu ever issued). They back the Yuzu-issued certificate counts and the certificate list behind them on the /crypto dashboard (ADR-0070 D13).

Action. On a large CA store, schedule the upgrade for a quiet period, and raise any startup or liveness probe timeout first (see “Health probes do not answer” above). Rolling back leaves the v5 indexes in the database; they are harmless to the older binary.

Behaviour change: new crypto_dashboard_layout_store schema, erased on deprovision

What changed. The server keeps each operator’s /crypto dashboard layout (ADR-0071) in a new Postgres schema, crypto_dashboard_layout_store, keyed by the operator’s username.

Action. Confirm the server’s database role can create schemas (it already must, for every other store). Rolling back leaves the crypto_dashboard_layout_store schema in the database; it is harmless to the older binary.

Behaviour change: /crypto becomes a per-operator widget dashboard with Yuzu-issued certificate counts

What changed. The Crypto page (/crypto) is now a grid of widgets each operator adds, removes and reorders (see The dashboard). The library figures are still there as widgets; new widgets count the certificates Yuzu’s own internal CA issued (agent, server, gateway, HTTPS and code-signing certificates alike), never the certificates on your devices.

Action. Review which custom roles hold Inventory:Read if enrolment counts should not be visible to them. Rolling back removes the widgets and routes. The two sections above cover the ca_store indexes and the layout schema this dashboard relies on.

Behaviour change: the built-in server certificate is re-issued once with a server identity URI (#6016)

On first boot after the upgrade, an install that uses the built-in default certificates has default-server.pem re-issued once, under the same root and the same key, with one added Subject Alternative Name: the server identity yuzu://<ca-fingerprint>/server. The previous certificate then appears revoked with reason superseded in the CA inventory (GET /api/v1/ca/issued, the CA page) and in the next CRL. That is bookkeeping, not a key compromise: the new certificate certifies the same key. Existing mgmt_peer_pins entries stay valid because the key is unchanged, so the gateway needs no action. The one-time event is audited as server.default_certs_server_identity_upgraded and logged as a WARN line. The CRL carries no revocation reason code, so the superseded reason is visible in the CA inventory only; a CRL consumer sees an ordinary revoked serial.

The gateway in this release can admit the server by that identity instead of by its key pin. Nothing changes until you opt in. The shipped reference configs (gateway/config/sys.config.prod, deploy/docker/reference-gateway-sys.config) set {mgmt_peer_auth_mode, pin} explicitly, and an installation with its own sys.config that does not set the key is pin too (an unset mode is always pin, never the identity, so a configuration with no pins and no mode still refuses to boot on an exposed listener). If you set mgmt_peer_identity while the mode is pin, the identity is ignored and the gateway logs a warning saying so. Once you opt in, a server key renewal that keeps the identity needs no change on the gateway when no pin is configured (a pin kept in identity mode must follow the renewal); in identity_or_pin the identity admits the renewed key whatever the pin says, and there is no pin_mismatch window. In identity_or_pin with pins configured, a peer that fails both paths because it has no identity (no_uri_san, identity_mismatch) is reported with the pin’s reason (pin_mismatch, no_pins_resolved); identity_unresolved (the gateway could not resolve the identity) and revoked are always reported as themselves, and a pin that never resolved is reported (yuzu_gw_mgmt_auth_pin_unresolved_total) even while the identity admits the traffic. The gateway’s rejection log line now ends see mgmt_peer_auth_mode, mgmt_peer_pins) (it ended see mgmt_peer_pins), so a SIEM or log rule that matches the old text needs updating. The identity trusts the issuing CA’s issuance policy: do not enable it where the management listener’s cacertfile is a bring-your-own corporate CA that lets a requester choose a URI SAN with serverAuth (stay in pin); an identity over a cacertfile that is not demonstrably exactly one readable certificate (a bundle, or an empty, unreadable or half-written file) refuses to boot unless the mode is identity with a pin (identity_over_ca_bundle). See Authorizing the server by its identity for the modes, how the identity is resolved, the serial deny-list mgmt_peer_denied and the trust trade-off (the identity needs a certificate the install CA signed, not the server’s private key).

To move a gateway to the identity: (1) upgrade the server to this release and confirm yuzu_server_identity_uri_present is 1 on every replica (YuzuServerIdentityUriAbsent warns when it stays 0); (2) set the gateway to identity_or_pin and keep the pin (the reference configs ship pin, so this is a change you make); (3) set identity only when both the gauge reads 1 on every replica and, after real command traffic through the gateway, yuzu_gw_mgmt_auth_admitted_total{method="identity"} moved while {method="pin"} did not; neither is sufficient alone (the gauge is set once at server boot and describes the file default-server.pem, not the certificate the server presents to the gateway, and a replica on the previous binary does not export it; {method="pin"} standing still on an idle gateway proves nothing). After that the pins may be dropped or kept as an extra constraint that must also match. mgmt_peer_auth_mode is read at boot, so a change applies when the gateway next restarts; do that only if its agents can take the disconnect, because agents without the #5183 fix stay wedged after a gateway restart.

Bring-your-own certificates, --no-default-certs and a server given its own --cert/--key carry no identity and stay on pins. yuzu_server_identity_uri_present is 1 once default-server.pem carries the URI; it describes the file, so it is a readiness signal only when the agent and management listeners use the built-in certificates. Operator-supplied certificates, --no-default-certs and a server given its own --cert/--key are not touched by the server’s re-issue, and rolling back the server is safe (an older server build ignores the extra name in its certificate).

Rolling back the gateway is not the same. A gateway binary from before #6016 ignores mgmt_peer_auth_mode, mgmt_peer_identity and mgmt_peer_denied (nothing else reads them). After such a rollback (i) the deny-list is not honoured, so a withdrawn serial whose key is still in a pin is admitted again, with no signal; (ii) if the node has no pins (dropped for identity, or identity_or_pin running on the identity alone), the older binary refuses to boot (no_mgmt_peer_pins) and the command plane for that node is down; and (iii) the rollback is itself a gateway restart, which disconnects the node’s agents: do it only if they can take the disconnect, because agents without the #5183 fix stay wedged after a gateway restart. Before you roll back, restore the pins (and the contents of any pin file), make sure they match the key the server presents now (a literal spki_sha256 pin can be stale after a key renewal made on the identity), check each withdrawn serial against the CA inventory, and re-key the server (or accept the re-admission) for any withdrawn serial whose key is still pinned. See Authorizing the server by its identity, “Rolling back”.

If you had already revoked the built-in server certificate for another reason (for example a suspected key leak) and not replaced it, the upgrade is skipped on purpose, with a warning on every boot and yuzu_server_identity_uri_present at 0: re-certifying a key you withdrew would undo your revocation. Replace the key first by re-keying the set (see --cert-san in Server administration: renaming default-marker.json aside re-mints every default leaf with a new key, which also changes the server key every gateway mgmt_peer_pins entry refers to, so update those pins at the same time). The same warning and 0 appear while the certificate directory is read-only or the certificate authority key is not in it; if that is intentional the warning is harmless, and the identity appears the first boot the directory is writable. The same applies if the revocation lands while the upgrade is running: the new certificate is then revoked too (audit detail replacement_withdrawn=true) and the set must be re-keyed.

Behaviour change: the Security and sign-in page (single sign-on, two-factor, HTTPS and agent mTLS)

Settings gains a Security and sign-in page (/settings/security, administrators only). It hosts the same four panels that /settings already loads, so no route, field name or audit row changes and /settings keeps working. What an operator or script can notice:

Behaviour change: dashboard API-token create and revoke apply MFA step-up

POST /api/settings/api-tokens and DELETE /api/settings/api-tokens/{id} (the Settings page’s API-token Create and Revoke buttons) now apply the same step-up check as the REST token routes.

Behaviour change: web UI wording and layout (Console, Home after sign-in, hidden quarantine controls)

Behaviour change: the Groups and enrolment page; the Settings enrolment panels look and act differently

Behaviour change: the plain-HTTP port answers the four health probes when HTTPS is on (#5906)

What changed. With HTTPS on, GET and HEAD on /livez, /readyz, /health and /api/health on the redirect port (--web-port, 8080 by default) now return the probe’s answer instead of 301: 200, or 503 while the HTTPS listener is not bound or has exited (https_listener_not_serving), the server is draining, or Postgres is down. Every other request on that port still redirects. The bodies are the unauthenticated ones the HTTPS port already served, now also readable in cleartext on this port (version, store names, agents.online and the tls block). --no-https-redirect removes the listener.

What to check.

Behaviour change: new agent, policy and fragment identifiers are restricted; page links encode identifiers

GET /api/me and GET /api/v1/me now report effective_role, elevated, elevated_until (only while elevated) and can_access_settings, and the shell shows the Settings entry only when can_access_settings is true. Users with an active just-in-time elevation now see the link; RBAC PlatformEngineer or Administrator-label holders who are not legacy admins no longer see a link that redirected away. During a rolling upgrade an older replica omits the new keys, so the Settings link is hidden for every user, administrators included, whenever /api/me is answered by an older replica; open /settings directly by URL (it is unaffected) or wait for the upgrade to finish. Tabs opened before the upgrade keep the old shell until reloaded. No configuration change is needed.

Behaviour change: GET /api/v1/me and GET /api/me no longer invent a role (#2146)

This is listed as a behaviour change rather than a break because the value that goes away was fabricated, was never documented for an engine principal, and rbac_role is now documented as an optional key.

⚠️ Breaking: the audit log is admin-only while RBAC is off, and audit reads are bounded (#5771)

Audit trail access. With role-based access control switched off (the shipped default), only administrators can now read the audit log. A non-admin signed-in user or API token gets 403 on GET /api/audit, GET /api/v1/audit, GET /api/v1/audit/auth-sample, GET /api/v1/engine-principals/audit/no-admin, the MCP tools query_audit_log and audit_engine_no_admin, and the yuzu://audit/recent resource. Previously any signed-in user could read it. To give a non-admin the trail, enable RBAC and assign a role that holds AuditLog:Read. Admins and RBAC-enabled AuditLog:Read holders are unchanged.

Find what broke. Denials are audited and counted. As an admin, run GET /api/v1/audit?action=auth.permission_required&since=<upgrade epoch>&limit=1000 and read principal on the rows whose detail begins topology floor: non-admin role denied AuditLog:Read. Alert on sum(yuzu_auth_topology_floor_denied_total{permission="AuditLog:Read"}) > 0. A poller that treats 403 as “no new events” stops collecting evidence silently, so make evidence collectors fail loudly on any non-200.

Two ways to restore access. (1) Enable RBAC and assign a role holding AuditLog:Read; read Enabling RBAC first, because a user with no management-group role then sees no agents. (2) Run the collector on an admin-owned token; while RBAC is off that token carries full admin authority, so prefer (1) for evidence collectors.

Rolling upgrade. An older replica still serves the trail to non-admins until every replica is upgraded.

Bounded audit reads. GET /api/audit now clamps limit to 1000 and returns 400 for an offset above 10000, matching GET /api/v1/audit and the MCP tool. Callers that paged deeper or asked for more than 1000 rows should narrow the window with since/until. MCP query_audit_log now rejects a wrong-typed or negative filter with an invalid-params error (-32602) instead of silently ignoring it. query_audit_log string filters are now limited to 1024 characters and a longer value is -32602. GET /api/audit and GET /api/v1/audit return 400 for a NUL or invalid UTF-8 in a string filter (a NUL used to truncate the filter and widen the read), and GET /api/audit also returns 400 for a negative since/until or an inverted window. query_audit_log now returns has_more beside entries in its structured result; the text channel is unchanged. During a rolling upgrade an older replica omits has_more, so a client that validates structuredContent against the new output schema can fail against it until every replica is upgraded. Clients that sent such filters should shorten or drop them.

Role reads fail closed. GET /api/v1/rbac/roles and GET /api/v1/rbac/roles/{role}/permissions now answer 503 when the permission store is degraded (previously an empty 200) and 404 for an unknown role (previously 200 {"data":[]}).

Behaviour change: POST /api/v1/hardware/{id}/sync refuses cross-site cookie writes

What changed. A call to POST /api/v1/hardware/{id}/sync (and the new fleet form POST /api/v1/hardware/sync) authorised by a session cookie must carry an Origin or Referer header naming this site (or an entry from --csrf-trusted-origin), otherwise it answers 403 cross-site request refused and is audited as csrf.denied. API, MCP and engine token callers are not affected.

What to check. A script that signs in with a cookie and calls the per-device sync route without Origin or Referer now gets 403: use an API token, or send the header. The Hardware and Software pages already send it. Behind a reverse proxy that rewrites Host, declare the public address with --csrf-trusted-origin.

What changed.

What to check. A script that signs in with a cookie and calls these routes without Origin or Referer, or posts the role body as a form, now gets 403 or 415: use an API token, send the header, or post JSON. A SIEM or alert rule that counts csrf.denied rows will now see the three user-management routes as targets, and a rule on user.create, user.delete or user.role_change rows will now also see attempted rows.

Behaviour change: MCP revoke_upload_grant is audit-first and refuses during an audit-store outage (#6169)

What changed. The MCP tool revoke_upload_grant now writes an upload_grant.revoke audit row with result attempted before it revokes anything, then a success or failure row, the same rows its REST twin DELETE /api/v1/upload-grants/{grant_id} writes. If the attempted row cannot be stored, the tool answers a retryable -32603 error carrying retry_after_ms and the grant is not revoked. It stays redeemable until it expires at its TTL. The REST route already refused the same way, with 503, so no transport can revoke a grant while the audit store is down. The success result no longer includes audit_persisted. mint_upload_grant is unchanged.

What to check. An MCP client that treated revoke_upload_grant as always succeeding while the grant store is up must handle the retryable error: retry after retry_after_ms, and on the supervised tier request a fresh approval, because the refused call consumed its ticket. A client that read audit_persisted from this tool must stop. A SIEM or alert rule that counts one upload_grant.revoke row per revoke now sees two (attempted, then the outcome), so match on result=success. An attempted row with no outcome row for the same target_id means the outcome is unknown: check the grant’s state with list_upload_grants.

What changed.

What to check. A script that signs in with a cookie and calls this route without Origin or Referer now gets 403: use an API token, or send the header. A SIEM or alert rule that counts one config.update row per change now sees two; match on result=success. Behind a reverse proxy that rewrites Host, declare the public address with --csrf-trusted-origin or the General page’s Save is refused. A stored value below 30 cannot be cleared by saving 30 on the General page, because that page already shows 30 as the current value: save another value first, or PUT it with a token. The Developer page’s fragments are now administrators only (a non-admin holding Infrastructure:Read gets 403 from them), and the permission grid no longer writes a denial row on each load. Whatever you set, keep heartbeat_timeout well above the heartbeat interval your agents run with: 30 leaves no margin for a late heartbeat.

⚠️ Breaking: OIDC refuses RSA signing keys under 2048 bits or with a degenerate exponent (#1901, #6058)

OIDC sign-in no longer accepts an RSA signing key whose modulus is under 2048 bits, or whose public exponent is below 3, even, or wider than 32 bits (65537 is the near-universal choice). Such a key is skipped when the IdP’s key set is parsed, with the warning OidcProvider: failed to parse JWK kid=<kid> (RSA modulus under 2048 bits) (or (RSA exponent is 1), (RSA exponent is even), (RSA exponent over 32 bits)), and its siblings are kept.

Also in this release (a fix, not a break): a token with no kid member is now checked against each usable signing key of the IdP in turn, at most the first 16 in the order published, instead of only the first. An IdP that omits kid while it publishes several signing keys, as it rotates, no longer has sign-ins refused intermittently.

Behaviour change: OIDC sign-in now requires the issuer’s discovery document (#5692)

What changed. The server no longer derives the OIDC authorization and token endpoints from the issuer URL, and the JWKS location (jwks_uri) is no longer left empty. All three are read from {issuer}/.well-known/openid-configuration the first time a sign-in needs them. Before this change an IdP whose endpoints were not the derived ones (Keycloak, for one) answered the sign-in redirect with an HTTP 404 (the browser was redirected to an authorization URL the IdP does not serve), and every ID token failed signature verification because no JWKS was ever fetched. Both failed closed; both now work. Neither server start nor a Settings save contacts the IdP, so a bad issuer is not noticed until the first sign-in or until someone clicks “Test Connection” (a connectivity check; it is not a guarantee of what a sign-in will hit, see Test Connection).

No working sign-in can regress, but a previously green Test Connection can now be red. In v0.13.0 and later, before this change, the server did not read jwks_uri from the issuer’s discovery document, so ID token signatures could not be verified and sign-in failed closed. There is therefore no working sign-in to break in those releases. “Test Connection” is stricter now: it can turn red for an issuer that differs from the document’s issuer by even a trailing slash, a document with no jwks_uri, an endpoint on a different scheme than the issuer, a JWKS that cannot be fetched or holds no usable key, or an address the egress policy refuses (see below). A green “Test Connection” before this change did not prove that sign-in worked. Rollback is a binary downgrade with no schema change, and it returns to the previous OIDC behavior, which cannot complete a sign-in: it does not restore a working one.

Redirect URI. The earlier manual said the Redirect URI was auto-computed when left blank; outside the startup-flag default (http(s)://localhost:<port>/auth/callback) that was never true, and a provider without one never fetches discovery. Behavior change: a Settings save (POST /api/settings/oidc) with a blank Redirect URI is now refused with 400, a red message naming the field and an error toast, where before it was saved and GET /auth/oidc/start then answered 500. Enter the callback URL registered at the IdP.

Review a forgotten OIDC configuration before upgrading. An OIDC configuration left in place but unused (in v0.13.0 and later, before this change it could not complete a sign-in, so nobody noticed) becomes a live sign-in on upgrade. If --oidc-admin-group is also set, members of that IdP group can now sign in as administrators. Check the flags and the group, or remove the flags.

Breaking: an IdP on a private network now needs --oidc-allow-private-networks. The server now refuses to make an OIDC request (discovery, the JWKS, Test Connection, the token POST) to a loopback, RFC 1918 (10/8, 172.16/12, 192.168/16), carrier-grade NAT (100.64/10) or IPv6 unique-local (fc00::/7) address unless it is started with --oidc-allow-private-networks (env YUZU_OIDC_ALLOW_PRIVATE_NETWORKS). This applies to the issuer host and equally to the hosts in the discovery document’s jwks_uri and token_endpoint, which are checked when they are used. A self-hosted Keycloak or ADFS on an internal address, and a lab IdP on localhost, will fail with request failed: address not permitted by the OIDC egress policy (and Test Connection will show the same text) until you set the flag. Link-local addresses (including the cloud metadata address 169.254.169.254), the listed cloud metadata addresses (100.100.100.200, 168.63.129.16, fd00:ec2::254), multicast and unspecified addresses are NEVER allowed, with or without the flag; Oracle Cloud’s 192.0.0.192 is the exception, because it lies in the 192.0.0.0/24 special-purpose block that the flag admits. The flag is a server start-up option only; it cannot be set from the dashboard or the REST API. Windows caveat: the token request is checked before WinHTTP resolves the name again, so a DNS change in between is not covered there, and a name whose answer mixes permitted and refused addresses is refused outright (WinHTTP cannot be pinned to the permitted one). Details: authentication.

Breaking: plain http:// to a public address is now refused, with no switch. An OIDC request over http:// (the issuer, or a discovery-document endpoint) whose target address is public is not made; the error is request failed: plain http is not permitted to a public address (token endpoint request failed: ... for the token POST), and Test Connection shows the same text. https:// to a public address is unchanged. http:// to a loopback, private, carrier-grade NAT or unique-local address still works, but only with --oidc-allow-private-networks, and it stays exposed to anyone on that network path: the signing keys, the token response and the client secret travel unencrypted there. If an IdP on a public address is configured with an http:// issuer, change it to its https:// URL.

What to do.

Behaviour change: sign-in moves to the landing page, /login redirects

What changed. /home is now the sign-in page (and the landing page for a signed-in user). A page requested without a session is redirected to /home?next=<screen key> instead of /login, and GET /login answers 302 to /home, forwarding only a valid screen key and a known SSO error code. POST /login, /login/mfa, /login/mfa/enroll, the OIDC and SAML callbacks and the session cookie are unchanged. Signing in no longer leaves the user on the Dashboard: password and single-sign-on sign-ins both end on /home, which offers an “Open Dashboard” button. A request sent by htmx without a session now answers 401 with an HX-Redirect: /home header rather than a redirect, so a panel never swaps the sign-in page into itself.

What to check. Any monitor, load-balancer health check or script that expects GET /login to answer 200 must probe /livez, /readyz, /health or /home instead (/home answers 200 without a session). Scripts that sign in with POST /login need no change. Bookmarks to /login keep working through the redirect.

Behaviour change: kill-switch audit rows carry the operator’s reason

The detail of the plugin_config.kill_switch.set audit rows (both the attempted row and the success or failure row, over REST and MCP) now carries , reason="<text>" when the caller gave a reason. A reason over 512 bytes is refused before anything is written; if one ever reached the audit writer it would be cut at a character boundary and , reason_truncated=true would follow the closing quote. Before, detail was exactly enabled=true|false, with , protective_exempt=true appended for a protective pair. A SIEM rule that matches the whole string (detail == "enabled=false") stops matching a call that carries a reason: match enabled=true|false as a prefix instead. The reason is free text entered by an operator (at most 512 bytes before escaping; \ and " are backslash-escaped), so do not parse it as structured data. A call without a reason is unchanged. See Audit Log.

Behaviour change: the plugin-config list is authorized by one list-read gate

GET /api/v1/plugin-config, the MCP list_plugin_config tool and the Plugins and updates panel now decide from the one ADR-0017 list-read gate instead of a permission check stacked on a list check. What you may notice:

Behaviour change: Guardian heartbeat reconcile is admission-controlled and fails closed on a degraded tag or property store (#5513)

What changed. The heartbeat reconcile is the re-push the server sends to an agent that reports a policy generation behind the server’s (see Guaranteed State). Three things differ from the previous release:

New flags (all per replica). --guardian-reconcile-budget-ms-per-s (default 500), --guardian-reconcile-burst-ms (default 1000) and --guardian-reconcile-max-per-s (default 1000), each with a YUZU_GUARDIAN_RECONCILE_* environment variable; see Server Administration. The defaults are provisional starting values, not derived from fleet measurement. No flag needs setting to upgrade.

Who notices. In steady state (no policy change), only agents that are behind are re-pushed, so shed stays near zero. Admission matters after a policy bump or a fleet-wide restart, when many agents are behind at once. At an estimated 5 ms per admitted push (not measured end to end), one replica at the defaults admits about 100 pushes per second, so a fleet of 3,000 agents converges in about 30 seconds and a fleet of 30,000 in about 5 minutes; use the yuzu_server_guardian_reconcile_duration_seconds histogram to replace the estimate with your own figure.

What you will see.

Rolling upgrades with more than one server. A replica still on the previous release keeps the old behaviour (no budget, and a narrowed push on a degraded tag or property read) until it is upgraded. The REST, MCP and dashboard policy-bump fan-out now also aborts on a degraded tag or property read (#5610): REST answers 503, MCP returns an internal error with a retry hint, and the dashboard answers 200 with a warning toast. A replica still on the previous release narrows the push instead.

No config or data migration is required. The agent binary does change (the lower-generation refusal above); agents that have not been upgraded keep the previous behaviour.

Behaviour change: an over-long facet value is left out of the response’s facets (#5623)

What changed. A facet value over 2048 bytes (measured in bytes on the value as stored) is now excluded from a response’s facets, never truncated. The response row, its output and every other facet of that response still persist. Before, one such value made the whole facet insert fail and the response lost ALL of its facets. The limit exists because the value is part of two database indexes, which refuse an entry over about 2.7 KB. A new counter, yuzu_server_response_facets_dropped_total{reason} (value_too_long, cap, batch_failed), counts the distinct (column, value) facets that did not persist; it is published only after the response commits, is pre-seeded at 0 at boot, and a failed commit publishes nothing. No alert rule ships or is owed: it is a data-completeness signal, and a per-response marker and alert are tracked in #5630. See metrics.md.

Stored values. Older servers indexed values that fit the database index limit: about 2.6 KB for an incompressible value (less the instruction id length), and far more for a value that compresses well (measured on PostgreSQL 18.4: 99,999 bytes of repeated text inserted, 499,998 were rejected). So values from 2049 bytes up to that limit were indexed before and are not now. Rows already stored by an older server stay readable until retention removes them. Facet rows are removed with their response by the existing reaper (--response-retention-days, default 90); nothing new is added.

Effect on the dashboard. For newly ingested responses a value over 2048 bytes is no longer indexed, including values between 2049 bytes and about 2.6 KB that older servers could index, and compressible values far larger than that. The facet filter, the dropdowns and the group-from-results and create-group counts therefore show nothing or “0 agents” for such a value, and a direct group-from-results submit answers 422 “No agents match” and creates no group when no agent matches. Every one of these surfaces now says so when an active exact-value filter (typed, or applied by a saved View’s default) is longer than 2048 bytes:

For a value in the 2049 bytes to about 2.6 KB band the answer 0 can be wrong: agents whose newly ingested output contains the value are not counted, because the value is not indexed. Rows ingested before the upgrade, when older servers indexed it, are still counted, so the count from older rows can still match.

5,000-facet cap. Above 5,000 distinct facets in one response, the facets kept are now chosen in a fixed order: most lines first, then column index ascending, then value ascending. Before, they were whichever the server happened to hold first, so a rare value could be cut and the choice was not repeatable. Different rare values may therefore survive or be lost compared with before.

Lost facets are not recoverable. A response whose facets were lost (batch_failed, or cut by cap) has no facets re-derived later, and nothing marks the response as affected (#5630). What to do depends on the reason. batch_failed (the facet insert was refused, for example by an unusually long echoed command id): run the command again; the new run gets a fresh response, and RECREATE any static group built from the affected response from the NEW run, because running the command again does not repair an existing group. cap: the same output keeps the same 5,000 facets, so a re-run cuts the same ones; use a command or filter that returns fewer distinct values, or accept the short result (#5630 tracks a marker). value_too_long: there is nothing to re-run, the value is never indexed. To find a response that lost facets, search the server log (a cap or batch_failed response logs a warning with its response_id); a value_too_long exclusion logs only at debug level, so at the default level only the counter shows it.

Rolling upgrade. The change is server-only: there is no schema, configuration, agent or gateway step, the order in which replicas are upgraded does not matter, and rolling back is starting the previous server binary. What differs during a rollout:

See response-store.md (Facets are bounded).

Behaviour change: the last administrator can no longer be deleted or demoted (#4966)

What changed. Deleting (deactivating) an account and demoting a local admin account now go through one guarded write that refuses to remove the last member of a protected administrator population: the active local role='admin' accounts (with RBAC enforcement on or off) and, with enforcement on, the accounts that hold the Administrator grant. This covers the dashboard user delete and role change (409) and every SCIM deactivation, delete, active:false and group-derived demotion (409, scimType mutability). It is a recovery rule, not proof that anyone can sign in. The audit results and the new counters are listed in the changelog entry and in docs/user-manual/rbac.md, “Last-Administrator guard for account deactivation and demotion”.

What to do.

Behaviour change: POST /api/instructions/{id}/execute now validates params against the stored schema

What changed. The route checks the request’s params against the definition’s stored parameter_schema and answers 400 (invalid params: <path>: <reason>) when they do not conform. The check runs after the permission checks and before the approval gate, so a refused call creates no approval ticket, execution row or dispatch. It is strict: a name the schema does not declare, a value of the wrong type or outside enum, pattern, minimum/maximum or length bounds, a missing required parameter, a string containing a NUL, and a non-object params are all refused. A definition with no stored schema (empty, whitespace or {}) is not validated. A stored schema that cannot be prepared makes the route answer 500 instead of dispatching. A schema is also checked when it is written: POST /api/instructions/import refuses an over-limit or invalid parameter_schema with a 400, and a bundled definition that fails the same check makes the server refuse to boot (the golden test guards the shipped content). Updating a definition (PUT, the YAML editor, response templates) keeps its stored schema. Details: REST API, Instructions.

Who is affected. Callers that send something the stored schema does not accept:

Not affected. The 7 shipped definitions that have no parameters: block and so store no schema (windows.app_control.wdac_policy, windows.app_control.applocker_policy, crossplatform.local_security_policy.password_policy, ...lockout_policy, ...audit_policy, ...sudoers and windows.rdp.status), definitions saved from YAML or created in the dashboard, and every other dispatch surface (workflow steps, schedules, policy remediation, result-set producers, MCP execute_instruction, POST /api/command), which do not validate yet.

A stored schema is not refreshed by an upgrade. The bundled reseed inserts a definition only when its id is absent (ON CONFLICT (id) DO NOTHING), so the six corrections above reach fresh installs only; an existing install keeps the old stored schema and keeps refusing the same calls until it is replaced (export the definition, delete it and import it again, signed unless the server allows unsigned definitions; see Replacing a stored parameter schema). Comparing the parameter declarations at v0.14.0 with this release, exactly 6 definitions differ in what the server enforces (the six above), and 13 more differ only in description text and enforce what they did (agent.content_dist.upload_file, device.agent_logging.get_log, device.event_logs.errors, device.event_logs.query, device.filesystem.find_by_hash, device.filesystem.read, device.filesystem.search, device.filesystem.search_dir, device.script_exec.bash, device.script_exec.exec, device.script_exec.powershell, device.windows_updates.patch_connectivity, workflow.patch_connectivity_audit).

Detecting it. Each refusal writes an instruction.execute audit row: result=denied with detail reason=param_schema path=<path> for a 400, or result=failure with reason=param_schema_invalid for the 500. Each also increments yuzu_server_instruction_param_rejected_total{route="instruction_execute",reason}. See Audit log and Metrics.

Rollback. No migration and no schema change. Rolling the server back removes the check, and while replicas run different releases the same call can be refused by one and accepted by another. A release without the check still rewrites a stored schema in one case: a YAML-editor save on the older release writes {} over it, because that route builds the definition from the YAML alone. The other update routes of the older release (PUT, response templates) write back the schema they loaded. A definition that was hit this way is not validated, on this release or any other, until its schema is replaced (see Replacing a stored parameter schema); rolling forward does not restore it.

Behaviour change: response routes parse numeric parameters strictly and exports are bounded (#4644, #4703)

Breaking for scripts that relied on a malformed value being silently misread. On the legacy /api/responses/{id}[/aggregate|/export] routes, the /api/v1/responses/{id}[/aggregate|/export] routes, GET /api/v1/executions/{id}/responses and MCP query_responses, status, since, until and limit (and the legacy catch-all route’s offset) must now be one whole base-10 integer. A value that used to be read as a different one is now 400 (-32602 on MCP): since=1e9 (was epoch second 1), status=0x1 (was 0), limit=100abc (was 100), since=1.5 (was 1), a leading +, whitespace, and a status below -1 (was “no filter”). An empty value and a value too large for its type were already 400.

Who is affected, and what to do.

Exports are bounded. Both export routes stop at 50 MiB of row payload (output plus error_detail) in addition to the 10,000-row limit, and the cut is made inside the database query, so a large export no longer loads every row into the server first. The cap is not operator-tunable. A cut export (row limit with more rows left, or the byte cap) is marked with a top-level result_truncated_by_cap: true (legacy JSON) or pagination.result_truncated_by_cap (v1 JSON), and a cut CSV ends with one extra trailer record, # result_truncated_by_cap cause=row_cap (or byte_cap), padded to the header’s width (7 fields legacy, 10 on v1). A CSV parser that expects a number in the id column will fail on that record, and only on a cut file; that is deliberate, so a cut file is never read as complete. An uncut CSV is byte-identical to before. A cut export also carries an X-Result-Truncated-By-Cap: true header (CSV) and a download name responses-<id>-truncated.json or .csv; a plain curl -o out.csv ... keeps neither, `curl -sS -D

Rolling upgrade. Replicas on the old and the new build answer a malformed value differently (200 versus 400), so a client can see both during the rollout. No configuration or data migration is needed. Two new counters report the effect: yuzu_server_response_param_rejected_total{surface} and yuzu_server_response_export_truncated_total{surface,cause} (Metrics).

Behaviour change: numeric query parameters on other routes are parsed strictly too (#4644)

Breaking for scripts that relied on a malformed value being silently misread. The strict parsing the response routes got (above) now also covers the routes listed in the REST API reference under Numeric Query Parameters: the execution-statistics routes, the executions lists, the audit routes, the instruction, product-pack, workflow and compliance lists, the patch lists, the software-inventory list, the fleet-topology machines_max and the app-performance compare window. A value that is not one whole base-10 integer now returns 400 where it used to be read as something else: limit=100abc (was 100), since=junk on the statistics routes (was 0, “from the beginning”), ?limit=abc on /api/analytics/recent and the patch lists (was ignored), window=7days (was 7).

On MCP, query_audit_log answers -32602 for a since, until or limit that is not a JSON integer or does not fit a signed 64-bit value (a string used to mean 0, unbounded, and 18446744073709551615 wrapped to -1), and the app-performance compare tool does the same for window. Other integer arguments that do not fit saturate instead of wrapping, except three tools where a saturated value would be harmful and the argument is strict and bounded instead (-32602 otherwise): create_offload_target batch_size must be 1 to 10000 (the declared schema range, now enforced), mint_upload_grant declared_max_size must be a JSON integer that fits a signed 64-bit value, and create_guardian_rule version must be 1 to 2147483647. mint_engine_credential ttl_days of the wrong JSON type is -32602 too (nothing is minted; it used to fall back to the 90-day maximum). Security-gating MCP arguments are strict too: issue_code_signing_cert validity_days must be a JSON integer from 1 to 730 (4294967297 wrapped to a 1-day certificate; a string, float or null took the 365-day default), mint_upload_grant ttl_secs must be a JSON integer of at least 1 (a wrong type, 0 or a negative value minted the 900 s maximum; larger values still clamp), create_api_token expires_at must be 0 or a future epoch (a negative value minted a token that never expires), and activate_platform_license seat_count and expires_at must be integers of at least 0 (a negative expires_at was perpetual); each is -32602 and creates nothing. A wrong-typed boolean on create_guardian_rule/update_guardian_rule enabled, push_guardian_rules full_sync and the engine-principal include_revoked is -32602 instead of a bare HTTP 500, and create_offload_target now neutralises the name it writes to audit rows. The two bounds now hold at the REST twins and in the stores, so no writer bypasses them: POST /api/v1/offload-targets reads batch_size as a 64-bit value and answers 400 outside 1 to 10000 (an oversized number is rejected, not wrapped) and for a float, string, boolean or null (a float such as 5.5 used to be truncated), POST /api/v1/guaranteed-state/rules answers 400 for a version outside 1 to 2147483647, and PUT /api/v1/guaranteed-state/rules/{id} and MCP update_guardian_rule refuse a rule whose stored version is already at 2147483647 or above (409 on REST, -32602 on MCP) instead of overflowing the version bump. The store enforces the range only when a write changes the version, so a rule that already holds an out-of-range stored version can still be enabled or disabled from the dashboard. The MCP tools get_execution_statistics_by_agent and get_execution_statistics_by_definition answer -32602 for a limit below 1 (the schema minimum; it used to be replaced by the default 50), and so do their REST twins, GET /api/v1/execution-statistics/agents and /definitions (400 for a limit below 1 or malformed). Two more REST inputs are new 400s: the limit in the body of POST /api/v1/inventory/query and POST /api/inventory/query (below 1 or not an integer; above 1000 still clamps to 1000, and 4294967297 no longer wraps to 1), and the ttl_days of POST /api/v1/engine-principals/{id}/credentials (present but not a JSON integer from 1 to 90; absent still means 90). That mint route’s body is optional only when it is empty or whitespace-only: a truncated document or a body that is not a JSON object ([], 5, "x", null) is now 400 and mints nothing, where it used to mint the default 90-day credential. {} and an object without ttl_days still mint 90 days.

A negative since is refused (400, -32602 on MCP) on the legacy GET /api/audit and MCP query_audit_log, and so is an until of 0 or below (a negative since or until, and an until of 0, each used to mean no bound and are new 400s here). The same holds for a negative since on GET /api/v1/execution-statistics/definitions/agents and their MCP twins, for to=0 on GET /api/v1/audit/auth-sample, and for an until of 0 or below (or a non-integer bound) in the body of POST /api/v1/inventory/query and POST /api/inventory/query; a non-integer since or until in an inventory query body made the body read throw and is a clean 400 now. The bounds are presence-tracked now: -3600 and until=0 used to mean no bound and returned the whole log, statistics set or inventory. Omit the parameter for no bound; since=0 (from=0 on the auth sample) is accepted and matches the same rows as omitting it. On the auth-sample export a present but empty from or to is accepted as unbounded on that side. GET /api/v1/audit follows the same rule: an earlier build of this release line read 0 as unbounded there; an until of 0 or below (or a negative since) is now 400.

The same strictness now covers the dashboard fragments’ page/per_page, the software inventory (the three software fragments answer 200 with an inv-caveat note for a malformed limit and ignore a well-formed one, which uses a fixed page size), DEX perf, network quality, hardware and CA-issued lists, the notification, webhook and offload delivery lists, the DEX, network and result-set limit and the Guaranteed State events limit/offset (trailing characters are 400; an integer outside the result-set page range still uses the default page size), the pre-flight run list limit, the pre-flight run form’s min_gib and window (a malformed value or a negative min_gib is 400; an empty field keeps its default; a window of zero or below uses 30 minutes and a larger value is clamped to 1 to 2880 as a 64-bit value, so one beyond a signed 64-bit integer is 400; see Pre-flight), the hardware sync poll’s await_since (a malformed freshness bound used to read as “no bound”), the Echo process-tree node and the GET /api/v1/events ?since cursor (the workflow stream has the header only). A malformed Last-Event-ID header is served live only (nothing replayed, never the whole buffer) rather than a 400, and -1 and 5abc are never accepted as cursors (5abc used to replay from 5). On the query-parameter routes, a parameter that is present but empty (?limit=) is malformed, not absent, GET /api/v1/audit limit included, except limit on the software list and catalogue routes and fragments, where an empty limit= means the default (see the software installs read). The verify fragment’s window is refused with its usual 200 note instead of a 4xx (the dashboard drops 4xx bodies).

Four admin forms stop silently saving a different value than the one typed. A malformed min_devices, window_seconds or cooldown_seconds on the DEX blast-radius thresholds is a 400 and saves nothing (it used to persist the clamped minimum). On the enrollment-token forms and the API-token ttl_hours, 0x5 used to read as 0; max_uses 0 means unlimited and a TTL of 0 means no expiry, so a mistyped limit minted an unlimited or never-expiring credential. Only an empty field, or a value that parses to a literal 0 (0, -0, 00), means unlimited or no expiry now, and a negative ttl_hours is refused (it would have computed 0). The API-token form refuses with its usual 200 feedback span and mints nothing; the other forms answer 400 plus an error toast. On the batch route an empty max_uses defaults to 1. A malformed rollout_pct on an OTA package upload or rollout change is a 400 instead of silently becoming 100. Behaviour change on the rollout change route only: an empty rollout_pct and a value above 100 are now 400 as well (an empty value used to mean 100 and a larger one was clamped to 100); a negative value still clamps to 0; the upload route keeps an empty value as 100, clamps a larger one to 100 and a negative one to 0, and refuses only a malformed value. A negative limit or offset on GET /api/notifications, and a negative limit on the patch lists (GET /api/patches, GET /api/patches/deployments), is 400 too (the store answered an empty 200), and so is a negative limit on the legacy GET /api/policy-fragments and GET /api/policies (it reached Postgres as LIMIT -1 and answered 503; the /api/v1/ policy routes clamp it as before). A malformed or out-of-range enrollment-token number now also writes an enrollment.token_create or enrollment.bulk_token_create row with result=denied and detail=invalid_numeric_parameter (see Audit Log).

Security-gating numbers in JSON request bodies are strict too. Before / after: POST /api/v1/tokens expires_at of -1, or a value above 9223372036854775807 that wrapped to -1, minted a token that never expires, 5.5 or 1e3 minted an already-expired one, and a string, boolean or null (or a non-object body) was a bare 500; now each is 400 and mints nothing (0 or omitted still means never expires, and a past epoch is refused). overlap_secs on POST /api/v1/tokens/{id}/rotate and POST /api/v1/engine-principals/{id}/credentials/rotate must be a JSON integer, and a non-blank body that is truncated or not an object is 400 instead of rotating with the 7-day default (an empty body still takes the default). version on POST /api/v1/guaranteed-state/rules must be a JSON integer in range (5.5 was stored as 5). ttl_secs on POST /api/v1/upload-grants must be a JSON integer of at least 1 when present (a wrong type, 0 or a negative value used to mint the 900 s maximum). On POST /api/settings/enrollment-tokens/batch a JSON number for count, max_uses or ttl_hours used to be silently ignored, so the documented curl example minted single-use tokens that never expired; numbers and numeric strings are now both honoured, a float, boolean, null, overflow or malformed body is 400, and an empty string still means the default. An ip_subnet auto-approve rule with a malformed prefix is refused when added (400); a malformed one already stored used to throw out of agent registration and now never matches, so it never auto-approves. POST /api/schedules interval_minutes (1 to 35791394), day_of_week (0 to 6) and day_of_month (1 to 28) must be JSON integers in range (4294967297 minutes became an every-minute schedule); POST /api/schedules/{id}/enable enabled must be a JSON boolean, the integer 0 or 1, or the exact string "true" or "false" (an omitted key or a blank, empty or whitespace-only, body still enables), and any other string, another number, null or a malformed body is 400 and changes nothing ({"enabled":0} used to arm the schedule). POST /api/command stagger and delay must be integers from 0 to 300 (a float, string or null read as 0, so a delayed command ran immediately; a value above 300 was silently capped by the agent at 300 seconds and is now a 400). A result-set matcher {"kind":"echo_rows_ge","n":N} needs an integer N from 1 to 2147483647 on POST /api/v1/result-sets/from-instruction-result and the MCP create_result_set_from_instruction_result (400 / -32602), and a matcher that is present but not a JSON object, or a kind that is not a string, is refused too (it used to be dropped, so every responder qualified; omit matcher for that default); a malformed n or kind already stored, or an n below 1, now matches no device instead of throwing in the maintenance tick or admitting everyone. The pre-flight form’s app_min, app_max and os_min must be empty or a dotted-decimal version with an optional -beta / +build suffix; abc, v4.2.0 or 4.2.x is 400 and starts no run (it read as version 0 and passed every device). POST /api/v1/license seat_count and expires_at follow the MCP twin: JSON integers of at least 0, else 400 (the licence surface is dormant until the LicenseStore is wired, ADR-0048). Rolling upgrade: replicas on the old and new build answer these inputs differently.

Three more inputs that used to be read as something else are refused (400 on REST, -32602 on MCP), and one stored value is repaired: a conditions[].value on the inventory evaluate and result-set routes that is a JSON number is accepted with ==, !=, >, >=, <, <= (and a boolean with == or !=); all six compare it as a number against the raw stored value (exact for integers and for integer against a decimal, a stored numeric string counting as that number; JSON integers are exact and every JSON float (a number written with a decimal point or exponent, such as 5.0, 1e3 or 5.5, or an integer outside -9223372036854775808 to 18446744073709551615, which arrives as a float) is refused because its original text is lost once parsed: send the decimal as a JSON string under >, >=, < or <=, which is accepted as a whole number within int64/uint64 (digits or an integer-valued exponent form such as "1e16", read exactly) or with at most 15 significant digits, a magnitude below 2^53, a value that is zero or at least the smallest normal double, and a value within the range of a double, and a plain-decimal string such as "1e400" is refused instead of ordering lexically; == and != with a string stay a text compare, so numeric equality covers JSON integers only; write a whole number as an integer up to 18446744073709551615 or as digits, and a string such as "9007199254740993.0" is read as that exact integer; a non-integer at or above 2^53 such as "9007199254740992.5" is refused), so behaviour change: a stored value that is not a number, such as "5 GB" or "N/A", no longer satisfies >, >=, < or <= against a numeric operand (it was read as 5 or compared as text), and any other operand type is refused (a number used to become the empty string, so {"op":">","value":5} selected every device that had the field); a string operand that is a plain decimal number ("4", "2e-7") is ordered numerically under >, >=, <, <=: a stored number or fully numeric string compares exactly (a stored integer within int64/uint64, including an integer-valued exponent spelling that fits (such as "564141138995e7"), and a stored decimal with a fraction of at most 15 significant digits and a magnitude below 2^53 compare exactly; any other stored decimal, one of more digits or a non-integer at or above 2^53, or a stored negative integer below -9223372036854775808, is compared as the nearest double), a stored string that merely starts with a number (a dotted version "1.2.3", "5 GB") has its leading number read and compared the same way (use version_gte / version_lte for dotted versions), and any other stored value ("N/A", "", a boolean) never matches, where a JSON number operand never matches a stored string that is not a number in full; while == / != and a non-numeric string operand such as "v2" are unchanged; a result-set matcher whose column, op or value is present with the wrong type no longer qualifies every responder; the DEX window on GET /api/v1/dex/signals, /dex/scope, /dex/signals/{obs_type} and their MCP tools (and get_dex_device_score) must be 24h, 7d, 30d or all (90d, a non-string such as 30 or null, or an empty ?window= was answered with 7 days; only an omitted window keeps 7d). Separately, a stored schedule interval_minutes above 35791394 now advances instead of failing in Postgres on every tick (new schedules are capped at that value).

Still lenient, and not an exhaustive list: the n retry-attempt counter on the live-snapshot poll fragments (the deployment, device, hardware, DEX, pre-flight and Echo process-tree polls), which is clamped to a fixed range per poll (1 to 100000 on the deployment and pre-flight polls, at most 50 on the others), and values parsed from stored rows, agent output or server configuration rather than from a request. Numeric path segments matched by a (\d+) pattern (notification read/dismiss and the webhook routes) now answer 404 for an id that does not fit a signed 64-bit integer; they used to fail with 500. The auto-approve rule index answers 200 with an audited denied row (rule_not_found) and changes nothing.

What to do. Send whole integers, and omit a parameter to get its default. No configuration or data migration is needed, but review Settings > Auto-Approve for ip_subnet rules with a sign or an odd prefix (they match nothing, so those agents go to the pending queue), and optionally list API tokens with expires_at=0 and enrollment tokens minted before the upgrade. During a rolling upgrade, replicas on the old and new build answer a malformed value differently (200 versus 400).

Changed: a degraded execution-statistics read is a 503, not an empty 200 (#4644)

GET /api/v1/execution-statistics, /agents, /definitions, GET /api/v1/statistics and the four MCP statistics tools used to answer a store failure (execution store not open, pool timeout, failed query) with 200 and an empty list or an all-zero summary, which reads as “no executions”. They now answer 503 with retry_after_ms: 2000 on REST and a retryable error with retry_after_ms on MCP. A filter or confined scope that genuinely matches nothing is still 200 with an empty data array. Each degraded read also logs a warning on the server (ExecutionTracker::try_get_<reader> degraded: tracker not open, pool exhausted or query failed).

What to do. Treat a 503 from these routes as retryable. A dashboard or script that read data: [] as “nothing to report” during an outage will now see the error instead.

Changed: a saturated pre-flight response window is inconclusive (#4644)

Each pre-flight check reads its newest 50,000 responses for a run. On a very large cohort a device that had answered could be pushed out of that window and then read as “no response” while the runner kept re-dispatching. A device absent from a full window now shows “inconclusive: response cap reached”, stays Incomplete (never in Deploy’s go cohort) and the runner stops re-dispatching that run; the run still completes at its deadline. Re-run against a smaller group to resolve those devices.

Behaviour change: per-agent execution statistics are confined to the caller’s scope (#3526)

GET /api/v1/execution-statistics/agents and the MCP tool get_execution_statistics_by_agent used to gate on a bare Execution:Read check. That check admits only a holder of a global grant, and it returned one row per agent across every execution of every operator with no per-agent filter. A management-group-scoped-only operator (no global grant) was refused with 403. Both now go through the same admit-then-filter fleet-read gate as GET /api/v1/executions (ADR-0017): the visible-agent set is applied in the query before the grouping and the row limit, and an empty visible set returns no rows, never the fleet. This widens who is admitted and narrows what a confined caller sees: nothing a caller could read before is now refused, with one exception: the gate fails closed, so when the RBAC or management-group store is not open the route answers a retryable 503 (retry_after_ms 5000), even to a global admin, where the bare check answered 200.

During a rolling upgrade, replicas on the old and the new build admit and return different row sets to a confined caller.

Behaviour change: the software installs read validates its inputs and audits an out-of-scope target

Breaking for scripts that relied on a malformed value being tolerated. GET /api/v1/inventory/software and MCP query_installed_software now share one parser with the catalogue routes:

Audit trail addition. A denied audit row (inventory.software.query, or inventory.software.export for an export; mcp.query_installed_software on MCP) is now also written when a request names an agent_id outside the caller’s scope. The response is unchanged (under push-down: an empty page, devices_omitted 0). A SIEM rule that counted denied rows from these routes will see them for this case too.

Who is affected, and what to do. Scripts that sent a malformed or over-long value, or a non-string MCP argument: send well-formed values. Scripts that retried a 503 on a NUL byte: it is a 400 now and will not succeed on retry.

Behaviour change: a cut visualization, results fragment or Echo scan is reported (#4644)

The execution visualization route (GET /api/v1/executions/{id}/visualization), the dashboard results fragment’s unfiltered read and the Echo retention-paused scan page now read through the same SQL-side 50 MiB payload cap as the response exports, and say so when it cuts.

Gateway upgrade note: stream-status notifications and startup checks (#4632)

Read this before restarting a gateway that includes #4632 (the version table above has the headline). A gateway restart is required and disconnects every agent the node holds; there is no hot code load. What changes for an operator:

Operator note: the software-inventory store migration (v7) is a hard cutover (#5172)

Schema v7 of the software-inventory store adds package_id and source columns and a row id to installed_software. The row id is added with a table rewrite under an exclusive lock, so the first start after upgrade takes time that grows with the table (about 10-20 s at 4M rows). A rolling upgrade is not supported for this migration at this stage: above about 2 million rows stop every server replica, then start one and let it finish; below that, start one replica first. The procedure, the row-count query and the free-space guidance are in Installed-Software Inventory (Upgrading).

Behaviour change: yuzu-server.cfg no longer overrides a stored password (#5274); passwords are now changed in the product (#5342)

What changed. Login now reads a local account’s password from the PostgreSQL auth store (auth.users) only. Before this release the server loaded the password hashes in yuzu-server.cfg into memory at boot and checked them first, so a hash in the config file silently took precedence over the stored one on the server that loaded it. That is gone: the config file is now a seed — it provisions the first administrator into an empty auth schema, and is never consulted for an account the auth store already holds.

Who is affected — the password reverts. Anyone who reset a password by editing (or regenerating) yuzu-server.cfg and restarting — including re-running the Windows server installer with a new /ADMIN_PASS on an existing install. That no longer changes the password, and a password rotated that way reverts on upgrade: the password that works is the one in the auth store, which for most installs is the one the account was first created with, and the rotated one stops working. If you had “reset” the admin password through the config file, sign in with the original password (or recover as below). If you rotated it because the old one was exposed, reset it in the product immediately after upgrading — until you do, the exposed password is the live one. At boot the server logs a warning naming each config-file account whose hash differs from the stored one (“#5274 cfg is seed-only; stored credential wins”) and sets the gauge yuzu_auth_cfg_credentials_stale to their count, so you can see which accounts this affects.

The config file is still a bootstrap credential. It seeds the first administrator whenever the auth schema is empty, so rebuilding onto an empty database (a lost or recreated Postgres, a restore onto a fresh database) brings back whatever password the file holds. Keep it protected, and change the administrator password in the product after any such rebuild. The reverse also holds: a fresh install pointed at an already-populated database (for example a reinstall, or a new server joining an existing cluster) seeds nothing — the installer’s /ADMIN_PASS (or the first-run prompt) is written to the new config file but ignored by the server, and the accounts already in the database keep their passwords.

Windows server installer. On an upgrade (a yuzu-server.cfg already exists in %ProgramData%\Yuzu Server), the installer now refuses a non-empty /ADMIN_PASS= or /OPERATOR_PASS= with exit code 11 before it stops the service or changes anything, instead of writing a config file the server would ignore. The interactive wizard skips its account pages on an upgrade. Remove those parameters from your upgrade command lines (SCCM/Intune packages included); see the Windows server installer section of Server Administration (#5196 note).

Breaking — the second first-run account and the Windows installer’s /OPERATOR_USER//OPERATOR_PASS are removed (#5343). They were written to yuzu-server.cfg only and were never provisioned into the PostgreSQL auth store, so the account they created could never sign in. First-run setup now prompts for the administrator only, and a fresh Windows install that passes a non-empty /OPERATOR_USER= or /OPERATOR_PASS= is refused with exit code 7 (nothing changed; an upgrade given /OPERATOR_PASS= still gets exit code 11, and an upgrade given /OPERATOR_USER= alone gets exit code 7). Remove both parameters from fresh-install command lines and create further accounts in Settings → User Management (POST /api/settings/users). Upgrades are unaffected: an existing installation keeps its accounts, and a leftover yuzu-server.cfg entry that has no stored account is reported by name in a warning at boot — delete that line.

Rolling upgrades with more than one server. A server still on the previous release keeps checking yuzu-server.cfg first and caching hashes, so it can keep accepting an old password after a change or reset made on an upgraded server, and it has no password routes (404). Upgrade every server before using Change password / Reset password, or before relying on a password change to lock someone out.

Passwords over 1024 bytes stop working. The new maximum applies at sign-in too: /login answers a password longer than 1024 bytes as a wrong password, without hashing it, and the attempt counts toward account lockout. An account whose existing password is longer than that (only possible if it was set outside the product, for example a hand-generated config entry) can no longer sign in. An administrator resets it to a shorter one (Settings → User Management → Reset password, then Unlock if it locked); if it is the only administrator, use the direct-SQL fallback in the auth-db-recovery runbook.

What to use instead.

Also in this release. Passwords must be 12–1024 bytes (UTF-8 bytes, not characters). A change or reset is one database transaction: the new password, the sign-out of every session of the account on every server, the discarding of an unfinished MFA enrolment, the target’s lockout (reset only) and the audit row are saved together or not at all — a refusal or failure (503 audit_unavailable / store_unavailable) changes nothing. After changing your own password you are signed out and sign in again — no replacement session is issued. A reset leaves the target’s API tokens and any enrolled second factor in place (the response reports the token count). The configured break-glass account cannot be reset over the API. Neither action has an MCP tool, by design (#5357 tracks a temporary-password tool). /login now answers 503 with Retry-After — not 401, and with no lockout strike or auth.login_failed row — when a password cannot be verified because the auth store could not be read or the password changed while it was being checked; clients that treat every non-401 as fatal should retry a 503. New metrics: yuzu_auth_password_changes_total{kind,result}, yuzu_auth_cfg_credentials_stale and yuzu_auth_credential_changed_during_verify_total. New audit actions: user.password_change, user.password_reset.

No config or data migration is required.

Behaviour change: per-OS plugin kill switch rows (#5294)

The plugin kill switch can now be narrowed per agent OS (windows, linux, darwin). The plugin_config_store schema moves to v2 (ALTER TABLE kill_switches ADD COLUMN os). Existing plugin and action rows read back unchanged, so nothing flips on upgrade.

Behaviour change: protective actions exempt from plugin enablement

What changed. Five plugin.action pairs are now protective: the quarantine plugin’s quarantine.quarantine, quarantine.unquarantine, quarantine.status and quarantine.whitelist, and the Guardian rule push __guard__.push_rules. No plugin-enablement gate refuses them any more: not the per-plugin/action kill switch, not the per-OS kill switch (or its agent-presence read), not the plugin-presence filter, and not a degraded plugin-config store. The set is closed. The four quarantine actions are newly exempt; before this change only the containment gate exempted them. Decision record: the 2026-10-12 update in ADR-3005.

A kill-switch row for one of these pairs is still accepted and displayed, but it is kept and inert: it no longer stops dispatch. A plugin-level quarantine / __guard__ row never stops the plugin’s protective actions either, and still gates any non-protective action the plugin declares (none today: quarantine declares only the four, and __guard__ dispatches only push_rules). A PUT on one is audited with protective_exempt=true in the plugin_config.kill_switch.set detail and logs a warning. If you stopped Guardian rule delivery by switching __guard__.push_rules off, that stop ends when you upgrade: rules deploy on the next Baseline deploy or heartbeat reconcile.

Before upgrading.

  1. Find held rows. There is no list route. Search the audit log for plugin_config.kill_switch.set entries whose target_id is __guard__.push_rules, quarantine.<action> or the plugin-level __guard__ / quarantine, each with an optional @<os> suffix. Each write leaves two rows (attempted, then success or failure) and a later enabled=true supersedes an earlier OFF, so take the latest result=success row per target_id. Confirm each with GET /api/v1/plugin-config/<plugin>/kill-switch at every layer: the plugin row, the action row (?action=<action>) and each ?os=windows|linux|darwin variant, reading source in the response: it names the row that wins.
  2. Re-establish the stop through the rule lever first, because the hold ends on upgrade. Disable the Guards (PUT /api/v1/guaranteed-state/rules/<id> with {"enabled": false}) or remove them. This is state-only: it disarms each Guard on the agent’s next Baseline deploy or heartbeat reconcile, so the latency is the reconcile cadence. It disarms the rule; it does not freeze a push already in flight. No lever stops delivery while leaving armed Guards armed: the old switch did that, but disabling or removing a rule disarms it on the next reconcile. It needs GuaranteedState:Write (disable) or GuaranteedState:Delete (remove). PluginConfig:Write alone no longer stops anything, and a service-scoped token is denied on rule writes. A quarantine.* hold has no replacement switch: those actions are governed by their permission (Security:Execute on the catalogue rows) and any approval policy, not by enablement.

Also be aware of:

Behaviour change: Guardian full_sync agent log lines and in-place rule: rows (#4415)

Three changes are live on every agent on upgrade, on both detection paths. The skip of healthy, unchanged rules that motivated them runs only on the Spark path, which production does not enable (prefer_spark is off); see the Spark flip gate AC-18.

⚠️ Docker Compose: copy /etc/yuzu/certs out of the server container before you recreate it (0.14.0 → 0.14.1, #5370)

Who this affects. Affected: a 0.14.0 Docker Compose stack whose server has no named volume on /etc/yuzu/certs while Postgres has one — deploy/docker/docker-compose.yml (the 0.14.0 README quickstart), docker-compose.uat.yml, docker-compose.full-uat.yml, docker-compose.viz-uat.yml, docker-compose.demo.yml used with --keep, any compose copied from them, and Compose Wizard output from before 0.14.1 in Plaintext mode, with ‘Persist generated certs’ unticked, or with named volumes off and an external Postgres. Not affected: docker-compose.reference.yml (which the README quickstart saves as docker-compose.yml since #5419) and docker-compose.reference-gateway.yml (their certs volume).

Why. The server keeps its internal CA, its default certificates and the secrets key-encryption key (KEK, secrets-kek-v1.key) in /etc/yuzu/certs. Since 0.14.0 the KEK is also registered in Postgres, whose data is on a volume. In the affected composes the key files sit in the container’s own writable layer instead, so anything that replaces the container deletes them: an image upgrade, docker compose down then up, up --force-recreate, or an up after a change to the server’s settings. Bumping only the image tag (or YUZU_VERSION) under the old compose file also triggers it. The next start then fails, on every restart, with:

SecretCodec::init() failed for the auth store bootstrap — kek_unresolvable: registered KEK 'secrets-kek-v1' does not resolve through the KekProvider

The 0.14.1 composes add a server-certs volume on /etc/yuzu/certs. Switching to them still recreates the container, so copy the files out first. Do this before anything else touches the stack.

How to run the blocks below:

Block 1 — steps 0 and 1. Step 0 checks that there is exactly one server container and that the keys are in it: the listing must show secrets-kek-v1.key (mode -rw-------) and default-ca.key. If it does not, do not go on; go to “If the server is failing with kek_unresolvable” below. Step 1 stops the server without removing it, because its container still holds the keys.

docker compose ps -a server
docker compose exec -T server ls -ln /etc/yuzu/certs </dev/null
docker compose stop server

Block 2 — step 2: copy the directory out of the stopped container. The copy goes to $HOME/yuzu-keys-backup, outside any checkout, owner-only (0700 directory, 0600 file), and never over an existing file. docker cp ... - writes a tar stream, which keeps each file’s owner and mode. The function keeps the archive only if docker cp succeeds, the whole archive reads back, it holds the CA certificate and key, and every KEK file in it is exactly 32 bytes (the size of a KEK). Otherwise it prints STOP and removes the partial file. It returns instead of exiting, so a failure leaves your shell open. It prints the CA fingerprint; note it. The archive holds the CA private key and the KEK: keep it as safe as the server itself.

BK="$HOME/yuzu-keys-backup"
yuzu_copy_out() {
  local cid out k n
  mkdir -p "$BK" && chmod 700 "$BK" || { echo "STOP: cannot create $BK"; return 1; }
  cid=$(docker compose ps -aq server)
  [ -n "$cid" ] && [ "$(printf '%s\n' "$cid" | wc -l | tr -d ' ')" = 1 ] \
    || { echo "STOP: expected exactly one server container, found: ${cid:-none}"; return 1; }
  out="$BK/yuzu-certs-$(date -u +%Y%m%dT%H%M%SZ).tar"
  ( umask 077; set -o noclobber; docker cp "$cid":/etc/yuzu/certs - > "$out.partial" ) \
    || { echo "STOP: copy failed; do not continue"; rm -f "$out.partial"; return 1; }
  tar -tf "$out.partial" > /dev/null 2>&1 \
    && tar -tf "$out.partial" | grep -qx 'certs/secrets-kek-v1.key' \
    && tar -tf "$out.partial" | grep -qx 'certs/default-ca.key' \
    && tar -tf "$out.partial" | grep -qx 'certs/default-ca.pem' \
    || { echo "STOP: the archive is incomplete; do not continue"; rm -f "$out.partial"; return 1; }
  for k in $(tar -tf "$out.partial" | grep -x 'certs/secrets-kek-v[0-9]*\.key'); do
    n=$(tar -xOf "$out.partial" "$k" | wc -c | tr -d ' ')
    [ "$n" = 32 ] || { echo "STOP: $k is $n bytes, not 32; do not continue"; rm -f "$out.partial"; return 1; }
  done
  mv -n "$out.partial" "$out" && [ ! -e "$out.partial" ] \
    || { echo "STOP: could not move the archive into place"; return 1; }
  tar -xOf "$out" certs/default-ca.pem | openssl x509 -noout -fingerprint -sha256
  OUT="$out"; echo "OK: $OUT"
}
yuzu_copy_out

If you have ever rotated the KEK, check that every version the database has registered has a file in the archive. Each number that docker compose exec -T postgres psql -U yuzu -d yuzu -tAc 'select kek_version from secrets.kek_meta' </dev/null prints needs a certs/secrets-kek-v<N>.key line in tar -tf "$OUT".

Step 3 — switch to the 0.14.1 compose file. In a checkout, git pull. Otherwise download the file again into the SAME directory: a different directory is a different compose project, with its own empty volumes. For a compose of your own, add the volume yourself (see below). Regenerating a Compose Wizard compose is the same switch.

Block 3 — steps 4 to 6: fill the new volume and recreate the server. The function first checks the archive again, and refuses to touch anything unless $OUT names a non-empty archive that reads back in full and holds a 32-byte secrets-kek-v1.key. Step 4 then copies the archive into the server-certs volume. docker compose run creates the volume and mounts it in a one-off container, so the stopped server container is left alone. Step 5 checks that the server user can read a 32-byte KEK in the volume: the files must be owned by the uid the server runs as, 999 for yuzu-server and 1000 for yuzu-server-chisel. Only then does step 6 recreate the server. Each step that fails prints STOP and nothing after it runs. If you opened a new shell after block 2, replace "$OUT" with the path block 2 printed. For the build-from-source stack, add --build after "$OUT"; it is passed on to docker compose up -d.

yuzu_fill_and_recreate() {
  local p="$1" n
  shift
  [ -n "$p" ] && [ -s "$p" ] && tar -tf "$p" > /dev/null 2>&1 \
    || { echo "STOP: no complete archive at ${p:-(unset)}; nothing was changed"; return 1; }
  n=$(tar -xOf "$p" certs/secrets-kek-v1.key 2> /dev/null | wc -c | tr -d ' ')
  [ "$n" = 32 ] || { echo "STOP: the KEK in $p is ${n:-0} bytes, not 32; nothing was changed"; return 1; }
  docker compose run --rm --no-deps -T --user 0 --entrypoint tar server \
      -x -p --same-owner --numeric-owner -C /etc/yuzu/certs --strip-components=1 -f - < "$p" \
    || { echo "STOP: step 4 could not fill the volume; the server was not recreated"; return 1; }
  n=$(docker compose run --rm --no-deps -T --entrypoint sh server -c \
      'test -r /etc/yuzu/certs/secrets-kek-v1.key && test -w /etc/yuzu/certs && wc -c < /etc/yuzu/certs/secrets-kek-v1.key' \
      < /dev/null | tr -d ' \r')
  [ "$n" = 32 ] || { echo "STOP: step 5 found no 32-byte KEK the server user can read; the server was not recreated"; return 1; }
  docker compose up -d "$@" || { echo "STOP: step 6 failed"; return 1; }
  echo "OK: the server was recreated on the copied keys"
}
yuzu_fill_and_recreate "$OUT"

The chiselled yuzu-server-chisel image has no echo, so on it the function stops at step 4 and recreates nothing (the docker compose run there has already created the volume). Do steps 4 and 5 in a throwaway container instead, then step 6 by hand. <project>_server-certs is the volume’s full name, which docker volume ls prints. The listing must show the files owned by uid 1000, with secrets-kek-v1.key 32 bytes:

docker run --rm -i --network none -v <project>_server-certs:/x/certs alpine tar -x -C /x -f - < "${OUT:?run block 2 first}"
docker run --rm --network none -v <project>_server-certs:/c:ro alpine ls -ln /c

Only if that listing shows secrets-kek-v1.key at 32 bytes, owned by uid 1000, recreate the server:

docker compose up -d

Run step 2 once. After step 6 the server container is a new one. If you re-run block 2 then, against a server that came back on the copied keys it only writes a second archive of the same keys; against one that came back without them it prints STOP. Either way the first archive is untouched.

Then confirm the server came back on the same keys. docker compose logs server | grep secret_codec shows verified 1 registered KEK version(s) (the count is of KEK versions, and the line repeats once per store that opens the codec), and this fingerprint matches the one block 2 printed:

docker compose exec -T server cat /etc/yuzu/certs/default-ca.pem </dev/null | openssl x509 -noout -fingerprint -sha256

Keep the archive as the keys half of a paired backup (the KEK pairing rule) or shred -u it; never commit it.

For a compose of your own, the change is one volume on the server service:

services:
  server:
    volumes:
      - server-certs:/etc/yuzu/certs
volumes:
  server-certs:

A new named volume takes its owner from the image’s /etc/yuzu/certs, which the yuzu-server image creates owned by yuzu, so a fresh install needs no chown. The chiselled yuzu-server-chisel image creates that directory only from 0.14.1; on 0.14.0 and earlier chiselled images a new volume is owned by root and the server cannot write to it. The demo stack (docker-compose.demo.yml) runs a one-shot server-certs-init service that gives the volume to the server user, so it also works with older chiselled images.

Rolling back. To return to 0.14.0, change only the image tag/YUZU_VERSION; keep the 0.14.1 compose with its server-certs line. Reverting the compose file strands the keys on the undeclared volume, and down -v under the old file deletes Postgres while leaving the keys.

If the server is failing with kek_unresolvable. Work out which failure it is before you touch anything; several of them leave the keys intact, and only the last one means they are gone. These checks change nothing:

docker compose ps server
docker compose logs server | grep -m1 -E 'kek_unresolvable|provider_failure|kek_orphaned'
docker compose config | grep /etc/yuzu/certs
docker volume ls --filter name=server-certs
docker compose run --rm --no-deps -T --user 0 --entrypoint ls server -ln /etc/yuzu/certs </dev/null

A STATUS of Restarting means the server refuses to start. Then match the output against these cases, in order. They run the listing as root (--user 0), so a wrong owner shows as such instead of Permission denied.

The chiselled image has no ls, chown, cp or echo. For it, run the same commands in a throwaway container on the volume, for example docker run --rm --network none -v <project>_server-certs:/c:ro alpine ls -ln /c (docker volume ls prints the full name).

Only case (e) means the key files are gone. Nothing inside the server can rebuild them.

Behaviour change: the executions list fragment and MCP summarize_working_set (execution kind) use the fleet-read gate (#3526, #4753)

GET /fragments/executions (the dashboard executions list) and the execution kind of the MCP tool summarize_working_set moved from a plain Execution:Read permission check onto the management-group-aware fleet-read gate that GET /api/v1/executions already used. This is hardening plus an admission change, not a fix for a leak: the old gate only admitted callers holding a global grant, and those callers were never filtered. An author-run reachability probe (not independently adjudicated; the JIT-elevated case was checked by code reading only) found nothing served through the old gate that the unconfined GET /api/v1/executions would not serve the same principal, with RBAC on or off (with RBAC off every non-service, non-engine caller whose tier allows it is unconfined by design; an engine principal is refused with 403) (docs/security-reviews/3526-confinement-reachability-probe-2026-10-05.md). There is no admission change for a caller with a global Execution:Read grant, and nothing changes with RBAC off for such a caller. What changes for every caller is the degrade and audit behaviour below, plus one new prerequisite.

Rollback. Redeploy the previous binary. The plain Execution:Read gate returns, so group-scoped-only operators go back to 403 on the fragment. The visible_agents filter is a read-time parameter with no stored state, so there is nothing to migrate or clean up. Audit rows already written with result=denied for mcp.summarize_working_set remain in the audit store.

Behaviour change: service-scoped tokens, the ITServiceOwner ceiling on the fleet-read gate, and GET /api/v1/upload-grants (#3526)

Two chokepoints let a service-scoped API token reach more than the ITServiceOwner role allows (found in a review of the fleet-read gate, #3526). Both now refuse it. This is a tightening for service-scoped tokens only; non-service callers, elevated sessions and engine principals are unaffected.

Breaking for service-scoped tokens, 1: GET /api/v1/enrollment/pending-agents now answers 403. The management-group-aware fleet-read gate (require_fleet_read, ADR-0017) did not apply the ITServiceOwner authority ceiling that require_permission applies to a service-scoped token. It now does, through one shared helper (authz::service_ceiling_check): the token is refused (403, “service-scoped token does not grant : (the ITServiceOwner role does not hold it)”, no permission field, audit auth.fleet_read_required / denied) unless that role itself holds the pair, whatever its minter holds. ITServiceOwner does not hold Enrollment:Read, so a service-scoped token that used to receive a pending-agent view narrowed to its tagged agents now gets the 403. Remediation: Enrollment:Read is intentionally NOT granted to ITServiceOwner; use an Administrator-minted non-service token for that route. The cost is that this replaces a confined credential with an Administrator-grade one, and there is no narrower option today: Enrollment:Read is Administrator-only by default and custom-role authoring is not reachable through REST, MCP or the dashboard yet.

Breaking for service-scoped tokens, 2: GET /api/v1/upload-grants now answers 403. The route’s only gate evaluated the minter’s username, so a service-scoped token inherited its minter’s UploadGrant:Read view (every grant, for a minter holding a global grant). It is now refused before any grant is read (403, “service-scoped tokens may not list upload grants”, no permission field, audit upload_grant.list.access_denied). The MCP tool list_upload_grants already refused a service-scoped token. Remediation: list grants with a non-service token. Non-service sessions are unchanged.

The ceiling applies wherever a seeded ITServiceOwner permission is absent. With the seeded defaults ITServiceOwner holds every other pair the fleet-read routes pass, so nothing else changes for Execution, Response, Inventory, Infrastructure, Policy, GuaranteedState and Workflow:Read. No REST, MCP or CLI surface removes a seeded ITServiceOwner permission today, and a direct database DELETE of the row is re-seeded at the next boot unless the pair is recorded in revoked_seed_defaults, so a refusal on one of those pairs does not arise in a default deployment; the rule is the safeguard for a pair removed through RbacStore::remove_permission (which records it) and for any future surface that narrows the role.

Failure behaviour. A FAILED read of the ITServiceOwner role’s permissions is a retryable 503 (retry_after_ms 5000, an audit row with detail “RBAC read degraded resolving the ITServiceOwner ceiling”, and yuzu_server_rbac_read_degrade_total increments) rather than 403, because an outage is not a missing grant; it still fails closed (the dashboard executions fragment shows it as the HTTP 200 data-degraded="gate" note). This holds on the fleet-read gate and equally on require_scoped_permission, which answered 403 for this failure before this change; only a definitive deny is 403. require_permission is different: a service-scoped token is refused there by the default-deny allow-list whatever the ceiling read returns, so a retry could not succeed, and a failed read answers the same default-deny 403 as a healthy one (no retry_after_ms, no yuzu_server_rbac_read_degrade_total increment); only its audit row differs, ending with ”; ceiling read degraded”. The ceiling read goes through the RBAC authz circuit breaker with a 250 ms acquire budget, so a degraded store fails quickly and, once the breaker is open, requests are answered without touching the pool; an open breaker is counted under the pool_acquire_timeout reason. The breaker bounds how many requests wait, not how long an already admitted read holds its connection: such a read can still wait up to the pool’s lock_timeout (10 s default) or statement_timeout (30 s default). On a dark network path (no reply at all) the wait is bounded instead by the pool’s tcp_user_timeout (10 s), which is confirmed on Linux, unconfirmed on Windows and a no-op on macOS, and until two failures have returned, up to the pool size (16 by default, --postgres-pool-size) of these reads can each hold a connection for that long. The breaker is the one operator permission checks use, so ceiling-read failures can open it and an open breaker denies operators’ cache-miss checks too (fail closed). The breaker counts consecutive failures and any successful authz read, a ceiling read included, resets the count, so a partial fault that lets the role_permissions read succeed while other authz reads fail can delay the breaker opening for operators’ cache-miss checks.

New alert: YuzuRbacBreakerOpen (critical). docs/prometheus/yuzu-alerts.yml now ships a per-replica alert that fires when yuzu_server_rbac_breaker_open has stayed at 1 on a replica for 5 minutes. It means authorization reads on that replica keep failing. While the breaker is open the replica refuses (fail closed) the operator permission checks that miss its permission cache and sheds service-token ceiling reads; cached decisions keep being served for at most about 5 seconds. A breaker that closes again within 5 minutes does not page. Load the updated rule file if you maintain your own copy of the shipped alerts.

List-read A4 body. The A4 body of a service-scoped token’s 403 from the list-read gate no longer carries a permission field; clients should not read it.

The sibling gates changed their budget. require_permission and require_scoped_permission now read the ceiling with the 250 ms authz acquire budget behind that shared breaker. Before, they acquired with the 2000 ms kReadTimeout and were not breaker-gated, but the pool clamps a bounded acquire to 500 ms when it is already saturated at entry (PgPool saturated_fast_fail), so on a saturated pool the wait was up to about 500 ms, not 2 s (a pool that saturated only after the call entered it was bounded by the full 2 s). The change lowers that bound to 250 ms and, once the breaker is open, answers the request without a pool touch.

Rollback. Downgrading the binary restores the previous behaviour for all of the above. The change adds no schema and persists no state.

To find affected callers, search the audit log for action=auth.fleet_read_required with result=denied and a detail containing “ITServiceOwner permission” (definitive deny, 403) versus “RBAC read degraded” (store fault, 503), for the sibling gate action=auth.scoped_permission_required with the same two details (“lacks ITServiceOwner permission” for the 403, “RBAC read degraded” for the 503), and for action=upload_grant.list.access_denied. action=auth.permission_required never carries the “RBAC read degraded” detail: a service token on a route behind the plain permission gate is refused by the default-deny allow-list (“default-deny”, 403) whatever the ceiling read returns, and during an outage of the ITServiceOwner read that row ends with “ceiling read degraded” (a healthy default-deny row does not), so rows from require_permission routes with that ending mark the outage. For Execution:Read the affected routes are the executions drawer’s /fragments/executions/{id}/detail fragment, the legacy GET /api/executions* routes, GET /api/v1/executions with its /{id}, /children and /api/v1/events twins, and the MCP tools on that gate that service tokens can reach at all. POST /api/executions/{id}/rerun and /cancel are not affected: a service-scoped token is already refused at their first gate, Execution:Execute. kServiceScopeGlobalSafe is not widened.

Behaviour change: vuln_scan reports an unreadable config check as UNREADABLE and adds a summary row (#4961)

On Linux, vuln_scan (scan, config_scan) now reports a config file it could not read (/proc/sys/kernel/randomize_va_space, /proc/sys/fs/suid_dumpable, /etc/ssh/sshd_config, /proc/mounts) as UNREADABLE|config|<title>|<path>: <cause> instead of a HIGH/MEDIUM finding. summary always emits a seventh row, summary|UNREADABLE|<n>, and an absent sshd_config reads INFO “not applicable” with the SSH password row now emitted.

Behaviour change: OTA refusal reason for an unreadable agent trust bundle (0.14.0 → 0.14.1, #5249)

Behaviour change: DEX device score and management-group member reads now fail closed on a degraded read (#4855, #1762)

Not a Breaking lead for the DEX routes and the management-group MEMBER-read path — those already documented a 503 response before this release; what changes for them is when it fires, not the documented contract. This does NOT hold for GET /api/v1/management-groups/{id}’s own GROUP-ROW read or its MCP twin get_management_group (governance round-2, #1762): before this release, a degraded group-row read answered the SAME flat, undocumented 404 "group not found" a genuinely nonexistent group id gets — there was no 503 contract for that case at all. This release adds a NEWLY DOCUMENTED, additive 503/retryable error path for a degraded group-row read specifically; the 404 contract for a genuine not-found is unchanged. A client that already treats any 503 from these routes as retryable per the A4 contract needs no code change; a client that inferred “management group not found” purely from a 404 status code should note that 404 now unambiguously means “no such group” (never “could not tell”) — narrower, not wider, than before.

Operator-visible behaviour change (fail-closed reads). Two previously fail-soft read paths now surface a degraded Postgres read as a retryable error instead of a healthy-looking or empty result:

Counters/alerts to watch during and after upgrade: yuzu_server_guardian_read_degrade_total (alert YuzuGuardianReadDegraded; this release adds a new source="guardian_rules" value alongside the pre-existing source="guardian_state", both now pre-seeded to 0 at boot across all three reason values) and yuzu_server_mgmt_group_read_degrade_total (alert YuzuMgmtGroupReadDegraded, now also counting the get_members_checked member-read degrade). A sustained non-zero rate on either during the upgrade window means the affected reads are answering retryable errors, not that devices or groups actually shrank.

No config or data migration is required.

Behaviour change: devices that disconnect cleanly stay on /hardware as Offline (#5678)

Not a Breaking lead: no flag, wire format or API shape changes. In 0.14.0 a graceful agent disconnect deleted the device’s endpoint_state row, so the device vanished from /hardware instead of showing as Offline. A graceful disconnect now clears only the row’s presence session, so the device stays listed with its real last-seen time.

Behaviour change: the DEX cohort floor is one device, and the Apps list fails closed

Not a Breaking lead: no flag or route is removed and no response field is renamed. Three operator-visible behaviours change.

No config or data migration is required.

Behaviour change: a failed device-roster read is reported instead of shown as an empty fleet (#5531)

Not a Breaking lead: GET /api/v1/hardware and GET /api/v1/hardware/{id} already declared a 503 response; what changes is when it fires. (The new 400 on the record and sync routes, below, applies only to ids over 256 bytes or carrying a byte below 0x20, which agent registration never accepts.) Before this release a failed read of the persisted device roster (pool timeout or query error) was shown as an empty roster.

Informational: internal-CA CRL publish/audit/observability follow-ups (HA WS-6, #4828–#4830)

No schema, config, or wire-shape change — this is documentation of behaviour that already shipped with HA WS-6 slice 6.1 (ca_crl_versions migrations v3/v4, PR #4126) plus additive audit/metrics coverage.

Migration v3/v4 boot-time lock window (informational, applies to any upgrade crossing them). ca_crl_versions migration v3 (revoked_count column) and v4 (the ca_issued_keep_revoked append-only trigger) each take a schema-level lock guarded by a 30 s lock_timeout — long enough to outlast a legitimate in-flight CRL publish (which holds SHARE ROW EXCLUSIVE on the same table for at most its own 5 s lock wait plus signing time), but a rolling upgrade that has one replica already on the new schema while an older replica is still publishing can see up to ~30 s of the migrating replica’s readers/writers queueing behind the older replica’s lock. This is a one-time cost on first contact with each version and does not recur.

ca.crl.published is now audited on every publish path, not only operator revoke (#4829). Every CRL publish — the boot pre-publish, the leader-gated freshness re-publish, the count-compare self-heal re-publish, and the two operator-triggered paths (revoke, subordinate-CA import on REST/MCP/dashboard) — now writes a ca.crl.published audit row. The three paths with no live operator request (boot, freshness, self-heal) audit under principal=system; every row carries a reason= token (startup/freshness/self_heal/revoke/import_chain) so a self-heal republish that resolves an earlier revoke-triggered failure leaves its own result=success reason=self_heal row as the resolution, rather than the silence a prior gap left here. See docs/user-manual/audit-log.md.

New Prometheus counters (#4830): yuzu_server_ca_crl_publish_failure_reason_total{reason} (a bounded, closed cause breakdown alongside the existing yuzu_server_ca_crl_publish_failures_total) and yuzu_server_ca_unpublished_revocation_check_failures_total (the freshness pass’s own self-heal check failing — distinct from a publish failing outright). See docs/user-manual/metrics.md. No operator action required; both are additive.

⚠️ Breaking: --gateway-upstream now refuses to start without gateway peer authorization

Affects you if the server runs with --gateway-upstream (or YUZU_GATEWAY_UPSTREAM). The Linux package’s systemd unit passes --gateway-upstream unconditionally, so a Linux package install is affected even when you run no gateway: with your own certificates it will not start until you pin or drop the flag (see the table). The gateway-upstream service now requires an authorized gateway peer (an SPKI public-key pin, or an explicit acknowledgement). The decision is made at boot, before any listener is bound: a configuration that cannot be authorized makes the server exit non-zero with a message naming the flag to change. Nothing is silently downgraded, so check the steps for your deployment type before upgrading the server. Full reference: Gateway upstream peer authorization.

Check before you upgrade the server. Read the arguments the server will actually start with, on the host, and apply the decision rule below.

DeploymentCommand
Linux package (systemd)systemctl cat yuzu-server | grep -v '^[[:space:]]*#' | grep -E -- '--(management-)?(cert|key|ca-cert)|--no-tls|--gateway-upstream|YUZU_(CERT|KEY|CA_CERT|GATEWAY|INSECURE_GATEWAY)' and, for the environment file, grep -v '^[[:space:]]*#' /etc/yuzu/yuzu-server.env | grep -E 'YUZU_(CERT|KEY|CA_CERT|GATEWAY|INSECURE_GATEWAY)'. systemctl cat prints the unit and every drop-in, so a flag or an Environment= line added by a drop-in is covered. YUZU_CERT, YUZU_KEY and YUZU_CA_CERT are the environment names of --cert, --key and --ca-cert, so a unit with Environment="YUZU_CERT=..." plus --gateway-upstream refuses to start the same way a flag does. The grep -v drops comment lines: the shipped unit’s comment block mentions every one of these flags, so without it the output is mostly comments (on the shipped unit the filtered output is the single --gateway-upstream line, and on the shipped yuzu-server.env.example it is empty)
Docker / ComposeFrom the directory of the compose file you actually use: docker compose -f <file> config | grep -E -- '--(management-)?(cert|key|ca-cert)|--no-tls|--gateway-upstream|YUZU_(CERT|KEY|CA_CERT|GATEWAY|INSECURE_GATEWAY)', and the same two-stage grep (grep -v '^[[:space:]]*#' first) over any env_file: it names
Windowssc.exe qc YuzuServer (the flags are in BINARY_PATH_NAME: look for --ca-cert, --cert, --key, the --management-* overrides, --no-tls and --gateway-upstream), and dir "%ProgramData%\Yuzu Server\certs": a grpc-cert.pem or ca-cert.pem means your own gRPC certificates, a gateway-peer-pin file is a stored pin

Decision rule (--cert-san and --cert-group matches do not count as certificates; the --management-cert, --management-key and --management-ca-cert overrides exist only as command-line flags, so no environment-file check can find them):

  1. No --gateway-upstream and no YUZU_GATEWAY_UPSTREAM: not affected.
  2. --gateway-upstream and --no-tls: add --insecure-gateway-peer (or YUZU_INSECURE_GATEWAY_PEER=1), or the server will not start.
  3. --gateway-upstream and your own certificates (--cert, --key, --ca-cert or their YUZU_ variables), or any management override (--management-cert, --management-key, --management-ca-cert): a --ca-cert given without --cert and --key, and any management override, make the gRPC listener credentials operator-supplied, so the automatic pin does not apply and the server refuses to start without a pin. One corner: the server compares the path (server.cpp, agent_creds_are_default_files), so a --ca-cert given without --cert and --key whose value is exactly <ca-dir>/default-ca.pem is the generated default, not your own, and the automatic pin still applies. You need a pin (--gateway-peer-pin, --gateway-peer-pin-file or their variables) before the upgrade. If you run no gateway, remove --gateway-upstream instead.
  4. --gateway-upstream and none of the above (generated default certificates): nothing to set; the server pins its own gateway certificate.
  5. A pin and --insecure-gateway-peer together: refused. Remove one.

A literal dry run (a --check-config-style command that evaluates this decision without starting the server) does not exist; it is listed as a deferred follow-up in the decision record, so use the check above.

Back up the pin sources, and know what a restore does. The pin is configuration the server reads once at boot, and it lives outside the database. Add these to your pre-upgrade backup: any explicit pin files (wherever you keep them; under the packaged unit they must be under /etc/yuzu), /etc/yuzu/yuzu-server.env (or your compose file and its env file), and on Windows %ProgramData%\Yuzu Server\certs\gateway-peer-pin. Restoring the server without the pin file makes it refuse to boot (a missing explicit pin source never falls back to the automatic pin). Restoring an older certificate directory under the automatic pin pins the old gateway key, so a gateway that presents a newer one is refused until the server’s certificate directory and the gateway’s certificate match again (then restart the server).

What the service exposed before. The service was reachable on every port the server’s single gRPC server listens on, not only on --gateway-upstream. On the generated default certificates the agent port (--listen, default 50051) requests but does not require a client certificate, so a caller with no certificate could reach it. On the strict ports (management, gateway-upstream, and every port once you supply your own certificates) any holder of a client certificate signed by the install CA could, including an agent leaf. A caller whose ProxyRegister passed the handler’s enrollment checks was also recorded as a trusted gateway peer by IP address, and under --gateway-mode that relaxes the Subscribe peer-IP binding for that address. This is exposure, not a demonstrated end-to-end exploit. The change closes the service to anything but a pinned gateway key; it does not change what the agent listener itself accepts.

What it is now. Every call must come from a TLS-authenticated peer whose certificate lists serverAuth, whose public key (SHA-256 of the SubjectPublicKeyInfo) is in the pin set, and which is inside its validity window. Pins are fixed at boot: changing them needs a server restart. There is no pin reload and no revocation read on this path.

Your deploymentWhat happensWhat to do
Generated default certificates (no --cert/--key, no --management-* overrides), TLS on, gateway presents the server’s generated default-gateway.pem (the Docker reference gateway compose does; the Linux packages and the Windows installer with /GATEWAY also start the server on its generated certificates, so the same pin applies, but the gateway you deploy must present that certificate: the yuzu-gateway package’s shipped configuration dials the upstream in plaintext and does not connect to a TLS server at all, see the comment block in deploy/systemd/yuzu-server.service)The server pins default-gateway.pem automatically and boots. With --cert-group set it logs a warning that every process in the group holds gateway authority.Check key custody: the gateway key file is the credential that authorizes a gateway. For a Docker reference-compose install, the shipped compose file now mounts only the public CA certificate into the agent container (a one-shot ca-export service fills a ca-public volume), and pulling the new file changes nothing until the agent container is recreated: from deploy/docker run YUZU_ENROLL_TOKEN=<token> docker compose -f docker-compose.reference-gateway.yml up -d for the whole stack (the file refuses to start without YUZU_ENROLL_TOKEN; any value works for an agent that is already enrolled), not up -d server gateway, which leaves a running agent on its old mounts. If an agent container ever held the old mount, treat the gateway key as exposed: replace it and withdraw its pin (changing mounts cannot invalidate a copied key). A gateway on another host that does not present default-gateway.pem is denied until you pin its key.
Your own certificates (--cert/--key/--ca-cert), gateway connected over TLSRefuses to start until a pin is configured.Compute the pin with openssl x509 -pubkey -noout -in gw.pem | openssl pkey -pubin -outform DER | openssl dgst -sha256 | awk '{print $NF}' and pass --gateway-peer-pin <hex> (or --gateway-peer-pin-file <pem>; env YUZU_GATEWAY_PEER_PINS / YUZU_GATEWAY_PEER_PIN_FILE). On Windows, which has no awk, prefer --gateway-peer-pin-file <cert.pem> with the gateway’s leaf certificate: the byte pipeline is not documented for Windows PowerShell 5.1, whose text pipeline may corrupt the binary DER, and a wrong-looking hash is the commonest mistake (see “Lockout diagnostic” in the operator guide). The gateway certificate must carry serverAuth and clientAuth.
HTTPS on generated defaults but gRPC on your own certificatesTreated as your own certificates: no automatic pin.As the row above.
Plaintext (--no-tls) with a gatewayRefuses to start. --no-tls is not an acknowledgement.Add --insecure-gateway-peer (env YUZU_INSECURE_GATEWAY_PEER=1). Peer authorization is then disabled on every port the service is reachable on, with a boot error line and a boot audit row. Never combine it with a pin flag: that also refuses to start.
TLS on but no client CA (--insecure-skip-client-verify)Refuses to start: no pin could ever match.Supply --ca-cert, or use --insecure-gateway-peer on a development rig.
Linux packages (systemd unit)The shipped unit passes --gateway-upstream unconditionally and runs TLS on generated certificates, so the automatic pin applies and it boots. An install that gave the server its own certificates refuses to start unless a pin is set, even when it runs no gateway. The deb postinst only enables the service, so the failure surfaces at the next restart or reboot; the RPM restarts the service on upgrade, and systemd stops retrying after 3 failed starts in 60 seconds. The RPM %pre and the deb preinst of this release print an advisory warning on stderr during an upgrade when /etc/yuzu/yuzu-server.env or a yuzu-server.service override under /etc/systemd/system shows your own certificates with --gateway-upstream and no pin or acknowledgement; it never fails or changes the upgrade, it reads only those files, and the check above remains the authority.With your own certificates, set the pin in /etc/yuzu/yuzu-server.env. If you run no gateway, omit --gateway-upstream with a drop-in (the literal block is under the table); a package upgrade overwrites the unit file, a drop-in survives. A --no-tls drop-in now also needs --insecure-gateway-peer. Pin files must be readable by the yuzu user and live outside /home and /tmp (ProtectHome and PrivateTmp hide them): keep them under /etc/yuzu.
Windows installer/GATEWAY with /NOTLS adds --insecure-gateway-peer. /GATEWAY with generated certificates needs nothing, but a gateway on another machine must either present the server’s generated default-gateway.pem or have its key pinned. /GATEWAY with your own gRPC certificate and key needs a pin the first time: the installer refuses an upgrade or install that would start gateway mode on your own certificates with no pin, before anything is changed.Give /GATEWAY_PEER_PIN_FILE=<cert.pem> (preferred: the leaf certificate only, because a chain file pins every block) or /GATEWAY_PEER_PIN=<64 hex>; the two are mutually exclusive, and the interactive wizard has no pin page, so a pin is given on the command line only. The installer stores it as gateway-peer-pin in the certs folder under the data directory (%ProgramData%\Yuzu Server\certs) and, whenever gateway mode is selected, TLS is not skipped and that file exists, passes --gateway-peer-pin-file for it on every run, so the pin survives later upgrades. With /NOTLS the file is kept but not passed, because the server refuses the acknowledgement together with a pin, and /GATEWAY /NOTLS given together with a pin parameter is refused by the installer with nothing changed. A refused silent install exits with code 7 and the reason is only in the /LOG= file on a line starting PrepareToInstall: (deployment tools that report only the exit code do not show the reason); the existing service is left as it was. If you later revert the install to the generated default certificates, delete certs\gateway-peer-pin: the stored pin keeps being passed and replaces the automatic pin, so the generated gateway certificate would be refused. The installer checks that /GATEWAY_PEER_PIN is exactly 64 hexadecimal characters and applies only cheap checks to a pin file (it exists, is not empty, has no NUL byte); the server validates the contents when it starts. Giving either parameter again replaces the stored pin, so rotation with an overlap needs a multi-entry pin file. A service command line edited by hand is rebuilt by the installer on every run, so do not rely on it.
Compose wizard output (plaintext stacks)A stack generated by an earlier wizard revision refuses to start after an image pull: its server runs --gateway-upstream with --no-tls and no acknowledgement.Add the variable to the server’s environment: block in that block’s own style (the wizard’s list form - YUZU_INSECURE_GATEWAY_PEER=1, or YUZU_INSECURE_GATEWAY_PEER: "1" in a mapping block; compose rejects mixing the two), or regenerate the stack: the wizard now emits the list-form variable for a gateway stack without TLS.

The repository’s own plaintext rigs already carry the acknowledgement in this release. Four scripts and three compose files pass it as a flag: scripts/start-stack.sh, scripts/start-UAT.sh, scripts/win-start-UAT.sh and scripts/integration-test.sh (which passes it in its plaintext mode only: its --tls mode passes no acknowledgement and enforces the automatic pin of the generated default gateway certificate), and deploy/docker/docker-compose.full-uat.yml, docker-compose.sanitizer-uat.yml and docker-compose.viz-uat.yml. The two compose files that pull published release images, the root docker-compose.uat.yml and deploy/docker/docker-compose.demo.yml, set it as the environment variable YUZU_INSECURE_GATEWAY_PEER, as does the compose wizard’s plaintext gateway output. deploy/docker/docker-compose.reference-gateway.yml runs TLS on generated certificates and needs no acknowledgement. scripts/ci/qa-stack.sh launches no server of its own: it runs that reference-gateway compose on published release images, so the automatic pin applies and it needs no acknowledgement either (only a comment changed there). deploy/config/docker.env.example and deploy/config/yuzu-server.env.example document the new variables and deploy/systemd/yuzu-server.service carries a comment block for the unit. If you keep a modified copy of any of these files, carry the change over by hand. Release images that predate the acknowledgement ignore the environment variable, so with such an image the variable is harmless; to get the new behaviour set YUZU_VERSION to a release that includes it, or rebuild the images locally (a stale local image does not have it either).

Linux package, own certificates, no gateway. The shipped unit passes --gateway-upstream unconditionally. Run systemctl cat yuzu-server to see the unit you actually have (a local edit may differ), then run systemctl edit yuzu-server and enter the following, which first clears the inherited command and then repeats it without --gateway-upstream (this is the shipped ExecStart of deploy/systemd/yuzu-server.service minus that one line; keep any other arguments your unit carries):

[Service]
ExecStart=
ExecStart=/usr/local/bin/yuzu-server \
    --listen 0.0.0.0:50051 \
    --web-port 8080 \
    --data-dir /var/lib/yuzu

Then systemctl restart yuzu-server. If you do run a gateway, leave the unit alone and set the pin in /etc/yuzu/yuzu-server.env instead.

Rotating or withdrawing a pin is a restart. Pins are read once at boot. To rotate, list the old and the new pin together, in the same source (a command-line pin makes the matching environment variable ignored, not merged), on every replica, roll the replicas, switch the gateway (it presents a new certificate only after it redials; a gateway redials after its connection drops and reads its certificate and key from the configured files when it does, measured on the gateway’s own upstream client: see The upstream client), confirm yuzu_server_gateway_peer_denied_total{reason="not_pinned"} stays flat on every replica, then remove the old pin and roll again. To withdraw a key, remove its pin from every source and restart every serving replica. Revoking a certificate serial in the CA store does not remove a key pin. The procedure, including what to do after a refusal window, is in Gateway upstream peer authorization.

After a refusal window, work out which side the cause was on first. A gateway that includes #5695 keeps a NotifyStreamStatus refused with gRPC status 16 and sends it again (at once at first, then at each half open probe once the refusals have opened its circuit breaker), so the notice is delivered once the cause is fixed, unless the gateway restarted or its held store filled in the meantime. An older gateway does not retry a refused NotifyStreamStatus sent by an ordinary notification worker, so a session whose CONNECTED notice was refused can stay unplaced. The step-by-step runbook, with a verification after each step, is in Gateway upstream peer authorization; in short, there are three cause classes.

Certificate expiry is class B. A gateway whose connection outlives its certificate’s notAfter is refused on every call and does not redial by itself, so renew the certificate before notAfter, inside a maintenance window, and follow class B above to tell whether the gateway has already redialled.

Mixed replicas (HA). Give every replica the same pin set (on the generated defaults each replica pins its own default-gateway.pem, so if you ever give replicas different certificate directories, supply the same explicit pin on all of them). Compare the pin prefixes in each replica’s boot log line. Roll the replicas first; a gateway needs a restart afterwards only if the cause is on the gateway side and denials continue (class B above), because unless the agents run a build with the #5183 fix (shipped in 0.15.0) released agents stay wedged after one. While replicas run different versions an old replica admits calls that a new one refuses.

Rollback / temporary exception

Rolling back is a binary-only change: gateway peer authorization has no schema or store change. But the previous binary has no gateway peer authorization at all, so a rollback is itself a risk acceptance of the same kind as --insecure-gateway-peer. Record who accepted it, the date, and the date by which it will be remediated (a pin in place on the new binary), and schedule that work rather than leaving the old binary running.

Remove the new flags BEFORE rolling the binary back. An older binary rejects an unknown flag at startup (a command-line parsing error) but ignores an unknown environment variable, so a configuration that passes --insecure-gateway-peer, --gateway-peer-pin or --gateway-peer-pin-file on the command line fails to start on the old binary, while the same settings supplied as YUZU_INSECURE_GATEWAY_PEER, YUZU_GATEWAY_PEER_PINS or YUZU_GATEWAY_PEER_PIN_FILE are simply ignored. Remove the flags wherever they live:

To keep a new binary running while you fix a pin, --insecure-gateway-peer is an exception, not a fix: it disables peer authorization on every port the service is reachable on. Treat it as a time-boxed risk acceptance too: record who accepted it, the date and the remediation date, put the pin in place, and remove the flag. On a TLS server with a client CA the exception is reported as yuzu_server_gateway_peer_authz_mode{mode="insecure_ack_tls"} and raises YuzuGatewayPeerAuthzDisabledWithTls; every start with it writes a server.gateway_peer_authz_disabled audit row.

New flags and series: --gateway-peer-pin, --gateway-peer-pin-file, --insecure-gateway-peer; yuzu_server_gateway_peer_* (see Metrics).

⚠️ Breaking: gateways fronted by an HTTP/2-terminating proxy or mesh sidecar (#3869)

Affects you only if you run the Erlang gateway and agents reach it through something that terminates HTTP/2: an HTTP/2-aware reverse proxy (for example nginx grpc_pass), an L7 load balancer, or a service-mesh sidecar. Agents that connect to :50051 directly, or through an L4 / TLS-passthrough path that keeps one TCP connection per agent, are not affected by the proxy requirement; see Older agents and the registry restart note below.

Minimum agent version. An agent rejected by the gateway recovers by re-registering through its NOT_FOUND handling. That logic exists from v0.13.0, but the released v0.13.0 and v0.14.0-rc6 agents wedge in their reconnect path with default settings (bug #2182, fixed by PR #5183, which ships in 0.15.0) and recover only with --no-auto-update (observed with v0.13.0 and v0.14.0-rc6), or on a build that includes the fix. v0.12.0 never recovers by itself (observed; older versions were not tested; v0.14.1 does not contain the fix and was not tested). Upgrade the agents first, then the gateway, using 0.15.0 or later (the #2182 fix); until the agents are upgraded, restart an agent that stays rejected (restarting the agent service re-registers it). Agents that do not connect through the gateway are not affected by this requirement.

The gateway now admits an agent Heartbeat only on the HTTP/2 connection that opened the session’s Subscribe stream, and only for a session that gateway node holds. A heartbeat that arrives on any other connection is answered NOT_FOUND (unknown session), counted and not forwarded. An HTTP/2-terminating proxy is supported only when it is configured for connection affinity (each client connection gets its own backend connection, kept for as long as the client connection lives; see Keeping each agent on one backend connection). One that pools backend connections across clients is unsupported: it may cause repeated heartbeat rejection or share gateway-side connections across agents, removing the per-agent connection separation this check requires. Observed in a test with nginx grpc_pass in front of two agents: every heartbeat was rejected, agents still enrolled and received commands, no heartbeat reached the server and the server’s online count flickered; an L4 TCP forwarder (nginx stream) caused no rejections. Each rejection raises yuzu_gw_heartbeat_session_mismatch_total{event="security"} or yuzu_gw_heartbeat_rejected_total{reason} and appears in a rate-limited gateway summary log line (the connection_mismatch= count). The agent-facing message is the same unknown session for every reason, so diagnose from the counters and that log line.

Check before upgrading. Look for an HTTP/2-terminating hop between the agents and :50051 (reverse proxy, L7 load balancer, mesh sidecar). If there is one, move agents to an L4 or TLS-passthrough path first, exempt :50051 from the proxy, or configure the proxy for connection affinity and run the check in Client addresses behind a proxy (HAProxy 3.2.25 and the OKD 4.21 router were measured at 50 agents; nginx grpc_pass rejected every heartbeat in the one two-agent test). Also check the agent versions of your fleet (the Hardware list at /hardware shows each device’s agent version) and upgrade every agent older than 0.13.0 that connects through the gateway before you deploy the new gateway, and plan to restart any released 0.13.0 or rc6 agent that stays rejected; see Older agents below.

Restart requirement. Deploy the new gateway with a restart. The session index is created when the gateway registry starts and hot code upgrade is not supported for this change. A node that had the new code loaded without a restart has no index table, rejects every heartbeat as registry_unavailable, and reports sessions_index in /readyz (503 while the table is missing).

Rollback. Redeploy the previous gateway release. The only new state is the in-memory session index, and there is no wire, agent or server change, so nothing needs migrating; agents with the reconnect fix re-register on their own (released agents may need a restart, see the next paragraph for older agents), and a rollback removes the connection check. This path is derived from the change and was not run.

Older agents. A rejected agent recovers by re-registering through its NOT_FOUND handling (escalating cooldown, 2 s doubling to a 300 s cap). That logic exists from v0.13.0 (checked in the agent source), but in the released v0.13.0 and v0.14.0-rc6 agents it is blocked by bug #2182 (the update-check thread join wedges the reconnect teardown; inferred cause), fixed by PR #5183, which ships in 0.15.0. With default settings those agents log (#1894) and Heartbeat thread stopped and then never re-register (observed, 19 minutes, reproduced on a second agent); with --no-auto-update they re-registered 20 to 21 s after a gateway registry restart (observed with v0.13.0 and v0.14.0-rc6; it is a command-line flag with no environment variable). Agent v0.12.0 only logs Heartbeat failed and never re-registers by itself (older versions were not tested) (observed with v0.12.0, 29 failures in 14.5 minutes with default settings; a --no-auto-update run was watched for only about 2 minutes and behaved the same; older than v0.12.0 was not tested), so for persistent missing state they stay rejected until restarted or upgraded (such agents were still counted online by the server’s /health agents.online in the rig, so check the rejection counters and the agent log, not the online count; a heartbeat that falls in the short gap between the session leaving the pending table and its agent process registering can succeed later without re-registration). Upgrade the agents first, then the gateway, with a build that includes the #2182 fix (0.15.0 or later); until the agents are upgraded, restart an agent that stays rejected (restarting the agent service re-registers it). Agents that do not connect through the gateway are not affected. This matters only when heartbeats are rejected, which happens in four cases:

The registry-restart recovery in 17 to 37 s was observed with agents built from the branch tree (version 0.14.0, which includes the #2182 fix). Released agents were observed as described above.

See Heartbeat admission for the supported topologies, counters and runbook.

⚠️ Breaking: GET /api/v1/openapi.json now requires authentication (#2057)

The OpenAPI spec endpoint used to be public — any unauthenticated client could fetch it to learn the REST surface. It now gates Infrastructure:Read, the same permission the MCP yuzu://openapi resource twin already required, so the two surfaces agree.

Breaking for pre-auth tooling. A script, health-checker, or API-client codegen step that fetched the spec before logging in now gets 401 instead of the document:

{"error":{"code":401,"message":"unauthorized"},"meta":{"api_version":"v1"}}

What to do. Fetch the spec with a session cookie or an API token, same as any other /api/v1/* route:

# Session cookie (dashboard-style login)
curl -c cookies.txt -X POST http://localhost:8080/login \
    -d "username=admin&password=<password>"
curl -b cookies.txt http://localhost:8080/api/v1/openapi.json

# API token
curl -H "Authorization: Bearer <api-token>" \
    http://localhost:8080/api/v1/openapi.json

Agent OTA pulls are now bounded per peer (#913, #911) — behaviour change

DownloadUpdate is now admitted through a per-peer gate. A refused pull returns gRPC RESOURCE_EXHAUSTED; a transfer exceeding --ota-transfer-deadline-secs (default 900) or --ota-chunk-write-deadline-secs (default 30) returns DEADLINE_EXCEEDED. Agents retry on their normal update interval, so no update is lost — but the first fleet-wide update after upgrading can take substantially longer than before, and on one deployment shape it can take days.

Check this before your first post-upgrade fleet update. Admission keys on the peer’s certificate identity, falling back to source IP where no client certificate is presented. Every certless agent behind one NAT egress therefore shares one bucket: 2 concurrent transfers and a sustained 1 admission per minute for the entire site, after an initial burst of 20.

Certless devices behind one egressTime for a fleet-wide update
20minutes (absorbed by the burst)
500~8 hours
5,000~3.5 days

What to do. Run yuzu_ota_admission_key_mode_total and look at the mode="peer_ip" share. If it is material:

Watch yuzu_ota_download_admission_total{decision="rejected_rate"} during the rollout, and see the “Agent OTA pull bounds” runbook in server-admin.md.

Also new, and check this one before you upgrade a large fleet: --grpc-max-threads (default 8192) now caps the gRPC sync server’s thread pool. Previously nothing did. It is a fleet-size ceiling rather than a tuning dial — AgentService is synchronous and Subscribe holds one thread for the life of each connected agent’s command stream, so the value must exceed your concurrently-connected agent count. Below it, gRPC answers RESOURCE_EXHAUSTED to every RPC on every service sharing the quota, which is a fleet-wide outage rather than back-pressure. 8192 covers the fleets this release targets; if you run more than ~5,000 concurrent agents, raise it (expected_agents x 1.5) in the same change as the upgrade.

--ota-cert-reserve-pct (default 50) reserves half of --ota-max-concurrent-total for peers admitted on a certificate identity. If your fleet is largely unenrolled — no client certificates, so admission keys on source IP — the remaining half is the whole capacity those agents can use, and you may want to lower the reserve or raise the total for the rollout window.

⚠️ Security: rotate oidc_client_secret if it was ever set on this install

Every release up to and including v0.13.0 emitted the OIDC client secret in the clear to three places, and this upgrade closes all three:

Breaking for automation: closing the second path changed a response shape. GET /api/config now reports "is_set": true|false for a secret-valued key instead of value, so a client reading value for oidc_client_secret gets nothing where it previously got the credential. Full shape and the reason it is not a placeholder: rest-api.md.

Because the log path fired on every boot, historical logs — and anything that ingested them, such as journald, a Docker log driver, or a SIEM — may still hold the secret in plaintext. Purge or restrict those as your retention policy allows.

Audit rows written before the upgrade are now redacted when read, so the value stops being disclosed through that path too. The rows themselves are deliberately left intact: an audit row is compliance evidence, and rewriting history to conceal a mistake is a worse posture than declining to disclose it. So the plaintext remains at rest in those rows, and a direct database read or a restored backup still shows it. Do not assume a fixed expiry: audit_retention_days defaults to 365, but each prune pass is capped, and the retention guard deliberately declines a pass after a clock anomaly or a long outage, so rows can outlive the nominal figure. The secret also remains stored in plaintext in runtime-config; this fix closes disclosure paths and does not encrypt it at rest (see security-hardening.md, “Current encryption posture”).

Who this affects: any install where oidc_client_secret was ever set through Settings -> OIDC or PUT /api/config/oidc_client_secret on an affected release. All three paths above read from runtime-config, so a secret supplied only via --oidc-client-secret or YUZU_OIDC_CLIENT_SECRET, and never written through Settings or the API, was not disclosed by those three paths.

That is not the same as “not disclosed”. A secret passed as --oidc-client-secret sits in the process command line, world-readable through ps or /proc/<pid>/cmdline, and it is typically also in a systemd unit, a Compose file, or shell history. If you cannot rule those out, rotate anyway. YUZU_OIDC_CLIENT_SECRET is better against other local users — /proc/<pid>/environ is owner-only, unlike cmdline — but it is not better if the value is written into a Compose file or unit, since docker inspect exposes it to anyone in the docker group. Prefer a secrets manager that injects the environment variable at runtime.

Before you begin, check you have a local account you can sign in with — and on --auth-mode=sso-only, that it is the configured break-glass account, MFA-enrolled and armable. Rotating breaks SSO until Yuzu is re-pointed, and the way back in needs an account that already exists: break-glass names one rather than creating it. On an install with none, the only fallback auth-db-recovery.md documents is destructive.

What to do: rotate the client secret at your IdP, and delete the old one. The upgrade closes the disclosure paths but does not change the stored value — anything that already read it still holds a working credential, and only the IdP can invalidate it. Most IdPs let you add a second client secret without removing the first; adding one is not a rotation, because the disclosed secret keeps working until it is deleted.

Deletion does not reach what the secret already bought: access tokens minted with it are signed JWTs that stay valid until they expire, and IdP sessions can outlast them. Treat revoking those as part of the same job. Yuzu’s own sessions are separate — the IdP cannot reach them — and last at most 8 hours; with the Postgres substrate they are durable rows and survive a restart, so a restart does not end them. End them per operator via DELETE /api/v1/sessions?username=<name> (admin, UserManagement:Write; for an SSO operator the name is oidc:<iss>#<sub>). Neither reaches API tokens: one minted under a session that used the disclosed secret survives every step above. List them with GET /api/v1/tokens and revoke with DELETE /api/v1/tokens/{id}.

Re-pointing Yuzu at the new secret is a configuration task rather than a remediation one: once everything above is dealt with, SSO failing until you update it is an outage, not a continuing disclosure. It is also configuration-specific and easy to get subtly wrong, so it is being written up as a reviewed runbook rather than summarised here. Until that lands, see authentication.md (“OIDC Single Sign-On”) for the durable and non-durable ways to configure OIDC.

⚠️ Breaking: a multi-host --postgres-dsn now needs target_session_attrs=read-write, and load_balance_hosts is refused (HA WS-8)

Affects you only if the server’s Postgres connection names more than one host — host=n1,n2,n3, postgresql://n1,n2/yuzu, a PGHOST/PGHOSTADDR list in the server’s environment, or a host list in the pg_service.conf entry a service= DSN or PGSERVICE names — or sets load_balance_hosts (in the DSN, PGLOADBALANCEHOSTS or that service entry). A single host without load_balance_hosts, including a proxy or managed endpoint (RDS, Azure Flexible Server, Cloud SQL, the shipped HAProxy compose), is unaffected.

Check before upgrading — look for a host list or load balancing in the DSN and the server’s environment:

grep -rnE 'YUZU_POSTGRES_DSN|postgres-dsn|PGHOST|PGHOSTADDR|PGLOADBALANCEHOSTS|PGSERVICE' \
  /etc/yuzu/ /etc/systemd/system/yuzu-server.service* <your compose/env files> 2>/dev/null

If that shows service= or PGSERVICE, also read the named entry in the service file libpq uses (PGSERVICEFILE, else ~/.pg_service.conf of the server’s user, else pg_service.conf in PGSYSCONFDIR) for host, hostaddr, load_balance_hosts and target_session_attrs.

Behaviour change: /readyz now goes red when Postgres is unreachable, and shutdown can hold for a drain grace (HA WS-8, ADR-2002 §12)

/readyz gains a gating pg_reachable row, fed by a small probe on its own Postgres connection. Before this change, /readyz stayed 200 through a Postgres outage whenever the server had idle pooled connections or no traffic (the pool’s connect breaker only notices a failed new connection, and every store’s row only reports whether it opened at startup). A load balancer health-checking /readyz kept sending traffic to a server that could not serve it.

What you may observe after upgrading:

What to do: point liveness probes at /livez, readiness at /readyz; size max_connections by the formula and apply the reserved-slot grant on an existing database; set --shutdown-drain-seconds and the recommended health-check thresholds if a load balancer fronts the server.

Behaviour change: legacy /api/executions* routes are now management-group confined (#3789)

The legacy pre-v1 GET /api/executions (list), /{id} (detail), /{id}/summary, /{id}/agents, /{id}/children, POST /{id}/rerun, and POST /{id}/cancel routes previously carried NO management-group confinement at all — a bare Execution:Read/Execute grant, global or group-scoped, saw and could act on the entire fleet’s executions. They now gate on require_fleet_read, the same admit-then-filter mechanism GET /api/v1/executions/{id} already used.

Who this affects: operators holding Execution:Read/Execute ONLY through a management group (not a global grant) — these callers are newly ADMITTED to the five GET routes (previously flat 403’d outright) but see a narrowed, redacted view. A global grant holder sees no behavior change on read shape. Everyone sees the following response-shape changes regardless of scope:

What to do: if you have automation with a group-scoped (not global) Execution:Read/Execute grant, re-verify it still sees the executions it depends on — a confined caller now sees only executions it dispatched itself or that involve at least one agent it can see, rather than being denied outright or (previously) seeing everything. If you scripted around the old /cancel false-success or the old zero-filled /summary response for an unknown id, update to expect 404.

Verify: GET /api/executions/{id} for an execution outside your management-group scope should return 404, not the previously-unfiltered full record. docs/auth-architecture.md’s “Fourth migration (#3789)” section has the full design; docs/user-manual/rest-api.md’s “Executions” section has the per-route reference including the 503 shape and the rerun escape hatch above.

Behaviour change: Destructive-class dispatch now requires explicit agent_ids on REST and MCP (#3685)

17 plugin.action pairs are classified Destructive in the command catalogue: echo.purge_source, echo.rollup, filesystem.delete_lines, registry.delete_value, registry.delete_key, storage.clear, content_dist.stage, content_dist.execute_staged, content_dist.cleanup, content_dist.upload_file, tags.clear, script_exec.exec, script_exec.powershell, script_exec.bash, http_client.download, certificates.delete, and quarantine.quarantine (19 as of this writing — power_health.set_power_plan and printing.clear_queue have since joined them, under the same rule). Dispatching any of them without explicit, non-empty agent_ids — an omitted/empty target, or a scope (including "__all__", even alongside agent_ids) — is now refused with 400 ("destructive action requires explicit in-scope agent_ids; broadcast and scope fan-out are refused") on REST POST /api/command, and -32602 with the same message on MCP execute_instruction (refused before an approval ticket is minted or consumed, at both the supervised-tier pre-mint gate and the operator-tier handler).

MCP execute_instruction had no Destructive-targeting gate at all before this release — a scope-targeted or broadcast call to one of the 17 rows above dispatched normally and succeeded.

REST /api/command’s outcome for these 17 rows is unchanged — the prior guard already denied a scope-targeted or broadcast dispatch to any of them, both before and after this release. What changed on REST is the guard’s shape, not its outcome: the prior code was an if that collapsed “classified and not Destructive” and “failed to classify at all” (a classify-miss) into the same skipped branch, relying on a classify-miss being denied unconditionally further downstream (build_classified_command) rather than proving that locally. This release replaces it with an exhaustive switch with no default arm, so a future change to the verdict enum forces a compile-time decision here too, instead of silently falling through to a guard shape that was correct only because of a downstream backstop.

Who this affects: two surfaces change outcome in this release. MCP execute_instruction, for any operator or automation dispatching one of the 17 rows above with scope or an omitted/empty target. And the dashboard execute surface (POST /api/dashboard/execute), which gains the gate for the first time here — including the case most likely to bite, an omitted scope: that surface treated “no target named” as the whole fleet, so console automation that posted neither agent_ids nor scope for one of these rows previously dispatched fleet-wide and is now refused. A REST /api/command caller was already refused before this release and sees no behavior change. Dispatches to a row NOT on this list are unaffected on all three surfaces.

A second dashboard change, distinct from the targeting gate: an explicitly-targeted Destructive dispatch from the console is now confined to the agents visible to the calling operator, matching REST. With RBAC off this is a no-op — every enrolled agent belongs to the auto-created root “All Devices” group, so the visible set is the whole fleet. It bites only with RBAC enforcement on where an operator holds Execution:Execute as a global grant rather than through a management-group role: that operator has no management-group rows to be visible through, so the confined list is empty and the dispatch is refused with no reachable in-scope agent (audited reason=scope_violation). The fix is to grant the operator a management-group role covering the agents they administer. Note the message names in-scope reachability, not connectivity — an agent shown as online in the console can still produce it.

What to do — the field to send differs by surface, so do not apply one remediation to both:

In both cases, if RBAC is enforced, also confirm the calling operator’s management-group roles cover the agents named — see the confinement note above. POST /api/instructions/{id}/execute and MCP execute_bundle do not yet enforce the gate — a tracked follow-up, not a gap this release closes.

Verify: re-run the affected dispatch with an explicit single target — agent_ids on MCP/REST, scope=<agent-id> on the dashboard — and confirm it succeeds as before; a group-scope, broadcast, or untargeted call to one of the 17 rows should now return the refusal above rather than dispatching. To confirm the gate is exercising in your environment:

yuzu_server_dispatch_target_rejected_total{route=~"command|mcp|dashboard",reason="destructive_untargeted"}

Note the regex matcher (=~): a label matcher cannot take an alternation of quoted strings. The visible-agent confinement refusal is deliberately not on that counter — look for the command.dispatch|denied audit row carrying reason=scope_violation instead.

Behaviour change: SAML login gains a new availability coupling to rbac_store (#1832)

SAML SSO now reaches parity with OIDC’s group-to-role RBAC reconciliation (see authentication.md “SAML Fine-Grained RBAC”). This introduces a coupling that did not exist before: on a deployment that already has RBAC enabled and --saml-group-attribute configured, a SAML login now depends on rbac_store being reachable — a transient reconcile-store error denies the login outright (fail-closed), rather than granting a session under stale or unreconciled roles. Previously, a --saml-group-attribute-configured deployment’s SAML logins did not touch the RBAC store at all (only the coarse --saml-admin-group admin/user mapping ran), so this is a genuinely new dependency for that combination of flags, not a tightening of an existing one.

What to do: if you run SAML with both RBAC enabled and --saml-group-attribute set, ensure rbac_store (PostgreSQL) health and availability are covered by whatever you already monitor for RBAC-gated REST/MCP calls — a rbac_store outage now also takes down SAML login for that deployment, not just RBAC-gated reads. Deployments that leave --saml-group-attribute unset, or that run with RBAC disabled, are unaffected — this section is a no-op for them.

Also note: the --saml-group-attribute group-value cap (saml::kMaxGroupValues) is raised from 64 to 200 in this release, to align with RbacStore::kMaxIdpGroupsPerLogin. A SAML assertion carrying 65–200 group values, which previously had its excess values silently truncated (and the login still succeeded), now has all of them considered. An assertion carrying more than 200 group values, which previously also truncated-and-logged-in, now instead denies the login (mirrors OIDC’s group_count_exceeded behaviour) — see “SAML Fine-Grained RBAC” in authentication.md for the full detail, including the practical httplib form-body-size caveat on how large an assertion can realistically reach /saml/acs.

Behaviour change: InstructionStore moves to PostgreSQL — deleted content no longer resurrects (ADR-0058)

InstructionStore (InstructionDefinitions and InstructionSets — the operator- and build-time-shipped instruction/question catalog) moves from per-replica SQLite to the shared PostgreSQL substrate (ADR-0006/0007), like every other migrated store. No legacy-SQLite backfill (ADR-0009’s fresh-start-by-default class, following ResponseStore’s precedent): no production fleet has ever run a pre-Postgres build of any Yuzu store, so the pre-migration instructions.db is never read. On upgrade, the bundled catalog reseeds fresh on first boot the same way it always does, and any operator-authored content must be re-created via the normal API (POST /api/instructions, POST /api/instruction-sets) — there is no automatic carry-over.

Breaking, deliberate: an operator-deleted bundled definition/set no longer reappears. Since 0.12.x (see that row above), deleting a build-time-shipped definition and rebooting made it silently reappear — the old per-replica SQLite reseed logic couldn’t distinguish “never seeded” from “deliberately deleted”, so enabled: false was the documented workaround for permanent suppression. That workaround is no longer necessary but still works. Post-upgrade, a deleted bundled id stays deleted — across every reboot, every replica, and every future release — via a tombstone the reseed loop consults. Recovery path: the tombstone only gates the build-time-trusted reseed path; an operator can still freely bring the id back at any time via a normal POST /api/instructions create or POST /api/instructions/import.

A side effect worth knowing: delete-and-reboot was also, incidentally, the only existing way to force an existing bundled definition to pick up newer bundled content shipped in a later release (delete it, then the next boot’s reseed re-inserts the current version). That side channel is now closed along with the resurrection bug it depended on — an existing bundled row, edited or not, never automatically picks up newer shipped content. There is currently no other refresh mechanism; tracked in #2555.

REST contract change on the direct /api/instructions* / /api/instruction-sets* routes: DELETE on a missing id now returns 404 (was 200 {"deleted": false}); create/update/ import/create-set now return 503 on a genuine database failure instead of a misleading 400. See rest-api.md for the full per-route response-code reference. Any automation that specifically parsed the old 200 {"deleted": false} shape or treated every non-2xx response as a validation error should be updated.

No operator action is required to upgrade; the reseed loop runs automatically at boot. See docs/adr/0058-instruction-store-postgres-migration.md for the full design record.

Caveat (rollback direction): the above covers forward migration only. instructions.db still exists after the upgrade (it keeps backing the still-SQLite ExecutionTracker/ApprovalManager/ScheduleEngine siblings), but this binary never writes InstructionDefinition/InstructionSet rows to it. If you roll the server binary back to a pre-migration release, the outcome depends on your deployment’s history and is never an empty catalog either way — the pre-migration binary’s own boot sequence unconditionally CREATE TABLE IF NOT EXISTS-es the catalog tables and runs its own every-boot bundled-content reseed loop, the same way this store always has:

Behaviour change: token-rotation confirm now requires proof of possession (#3015)

confirm on a rotation — REST POST /api/v1/tokens/{id}/confirm and POST /api/v1/engine-principals/{id}/credentials/confirm, plus the MCP twins confirm_api_token_rotation/confirm_engine_rotation — previously admitted on caller identity plus the successor’s token_id alone. A caller who recovered an unknown successor’s token_id out-of-band (a support ticket, a log line) could confirm — and thereby revoke the predecessor for — a rotation whose secret they never actually received. All four confirm surfaces now additionally require the raw successor secret itself in the request body/args, verified with a constant-time hash comparison against the successor’s stored hash, checked LAST — strictly after ownership, pair-state, the token_id pin, tier, scope, and the initiator binding have all already passed — before the predecessor is touched.

Who this affects: any caller (human or automation) confirming a rotation with only the token_id, and any caller confirming with the wrong secret.

Correctly-implemented automation already holds the secret — it’s the same raw value the rotate response returns exactly once (REST data.token) — so a caller that installs the successor from that response and passes it straight through to confirm sees no change in behavior.

What to do if you lose the rotate response before confirming: you can no longer confirm — there’s no way to manufacture the secret from the token_id alone. Two recovery paths, and neither is new — this change makes the mechanism enforce guidance already documented in authentication.md “Rotating a Token” (don’t look the successor up and confirm blind):

  1. Wait for the automatic overlap-window sweep. Proof of possession gates this immediate, explicit confirm call only — the 60-second background sweep is unaffected by this change and still auto-revokes the predecessor on its own schedule with no secret required, provided the successor secret was actually installed and presented (used) at least once.
  2. Revoke the unknown successor and start a new rotation. This keeps the predecessor working immediately, at the cost of restarting the rotation.

See authentication.md and engine-principals.md for the full detail, and rest-api.md for the exact REST error bodies.

Behaviour change: MCP approval-store permanent failures now say don’t retry (#2786)

An MCP approval-ticket recall that hits a store fault has always returned -32603 with a retry_after_ms hint. Previously that hint was 5000 (retry after 5 seconds) for every store fault except a never-opened store, including one that was open but failing in a way no amount of retrying clears — SQLite corruption, not-a-database, a read-only filesystem, or a full disk. Those four cases now correctly get retry_after_ms: null (the same “escalate to an operator” response a never-opened store gets) instead of the transient retry hint.

What to do: if you have automation that blindly retries on -32603 without checking retry_after_ms, it now stops retrying sooner in this specific case — which is the correct behavior (the old retries were futile). If your automation already honors retry_after_ms per invariant A5, no change is needed. See mcp.md “-32603: Approval store unavailable” for the full response-body reference.

Behaviour change: four MCP tool input schemas got stricter (#2444)

revoke_certificate.serial_hex, confirm_engine_rotation.token_id, quarantine_device.reason/whitelist, and the eight approval-gated engine-principal tools’ principal_id (create/revoke_engine_principal, mint/rotate_engine_credential, transfer_engine_principal_owner, assign/unassign_engine_role) now carry pattern/maxLength bounds mirroring their handlers’ own checks (previously enforced only at the handler, after an approval ticket was already consumed). get_engine_principal and list_engine_roles also gained the same principal_id pattern, but per this codebase’s standing rule that input-schema validation runs only on the approval-gated path (mcp-server.md’s “Pre-approval input-schema validation”), those two are Read-tier and never approval-gated — the pattern is advertised metadata on tools/list only, not server-enforced; a malformed principal_id still reaches the handler and is rejected there exactly as before.

Who this affects: any operator or automation with an outstanding (pending or approved-but-unconsumed) approval ticket for one of these ten tools — revoke_certificate, confirm_engine_rotation, quarantine_device, create_engine_principal, revoke_engine_principal, mint_engine_credential, rotate_engine_credential, transfer_engine_principal_owner, assign_engine_role, unassign_engine_role — whose arguments don’t match the new bound: serial_hex must be ^[0-9A-Fa-f]{1,64}$; token_id must be ^[0-9a-f]{24}$; quarantine_device.reason/whitelist must be ≤1024/≤512 bytes (whitelist additionally charset-restricted); principal_id must match ^engine:[a-z0-9._-]+$ (or the bare-slug form on assign/unassign_engine_role). Full patterns: mcp-server.md’s “Pre-approval input-schema validation” section.

What to do: such a ticket becomes unrecallable fail-closed once the new server is running — the recall fails pre-consume just like any other schema-invalid call, and the ticket is never burned, but it also can’t be redeemed as originally minted. It stays valid until its normal 7-day expiry (pending or approved-unconsumed), then ages out and must be re-requested. To check exposure before upgrading, list your outstanding tickets for these ten tools (list_pending_approvals MCP tool or GET /api/v1/approvals) and compare each one’s arguments against the patterns above; no action is needed for a ticket that already matches, or if you have none outstanding.

Verify: re-request any ticket the check above flagged, confirm the new recall succeeds under the same tier, and confirm a deliberately malformed value (e.g. a non-hex serial_hex) is now rejected at mint time — -32602, no approval created — rather than being accepted and only failing later at the handler.

Behaviour change: discover_instructions/GET /api/v1/discover/instructions null out a non-object stored parameter_schema (#2986)

Both surfaces share one builder (build_instructions_catalog) that parses each InstructionDefinition’s stored parameter_schema text. It previously forwarded any value that parsed as JSON, even a non-object (an array, string, number, or boolean). It now forwards it only when the parsed value is itself a JSON object — a non-object value is reported as null instead, matching GET /api/v1/discover/plugins’ existing behavior for the same field.

Who this affects: an operator or integration that authored an InstructionDefinition with a non-object parameter_schema, written before the write-time schema check existed, or by a non-standard write (the store now refuses a non-object parameter_schema, and POST /api/instructions/import is the only REST route that can supply one). No shipped content sets parameter_schema to anything but an object or leaves it unset (defaults to {}), so this affects only a deliberately or accidentally malformed definition.

What to do: if you have such a definition and relied on the old raw-forwarding behavior, re-author parameter_schema as a JSON Schema object (see Replacing a stored parameter schema). No action is required otherwise.

Behaviour change: webhook and offload-target deliveries, and enrollment/execution-failure notifications, now actually fire (#3261)

A boot-ordering bug wired NotificationStore/WebhookStore/OffloadTargetStore into the agent-service RPC path (Register, Subscribe, process_gateway_response) before those three stores were constructed, so the wiring silently never took effect. Dashboard notifications for agent enrollment and execution failures, and configured webhook/offload deliveries for agent.registered and execution.completed events, have been dead for the life of the process since the stores were introduced — for NotificationStore/ WebhookStore, every tagged release from v0.10.0 through v0.13.0 (roughly four months); OffloadTargetStore didn’t exist until v0.12.0, so its affected window is v0.12.0 through v0.13.0 (roughly three months). Every use site double-guards ptr && ptr->is_open() with a silent skip, so nothing logged or alerted; the bug produced zero signal. They now fire on every boot.

What to do: if you configured a webhook (POST /api/webhooks) since v0.10.0, or an offload target (POST /api/v1/offload-targets) since v0.12.0, expecting live traffic and saw none, that integration has been silently non-functional the entire time, not merely quiet — there is no queued backlog to replay, because the events were never generated in the first place. If you use one of these integrations as part of your own monitoring or compliance evidence chain (a SIEM feed, an alerting pipeline), treat the gap as a documented monitoring-coverage lapse for that period, not just a product bug, when recording it in your own change or incident log. After upgrading, confirm your receiving endpoint still exists and can handle real traffic before you rely on it — deliveries start immediately, with no flag or opt-in involved. If your webhook fires to Slack or a similar noisy channel for enrollment alerts, note that agent.registered fires on every gRPC reconnect, not only first enrollment (this is unchanged, documented behaviour — see “Webhooks” / “Offload Targets” in rest-api.md — but was invisible until now); the “Agent Enrolled” dashboard notification specifically does not repeat on reconnect, so a server restart with N connected agents does not flood that feed.

Behaviour change: mcp.bridge.* audit rows can now carry result=failure (#2487 / #2506)

Audit rows for the MCP progress bridge (mcp.bridge.done_reap, session_dead, arming_reaped, pin_acked, forced_expire) were previously stamped result=success unconditionally, regardless of what actually happened. They now report the real outcome: result=failure when a background teardown could not release one of the resources it owns, or when the terminal-frame publish ladder poisoned the session, threw, or was never reached. The detail field names which, and no longer asserts a delivery or a poisoning that did not occur.

What to do: if you have a SIEM rule, dashboard or evidence query that treats result on this verb family as a constant success - for example “alert if any mcp.bridge.* row is not success” - update it before upgrading, or it will fire on legitimate rows. Note also that failure here is self-audited: a rule that surfaces failure branches by filtering on denied will not see these. Filter on result != "success" where you need both. mcp.bridge.cancel is unaffected and remains success.

These rows are background-actor events (principal=system), not operator actions, and a failure row does not mean a client lost a result - executions stay durably fetchable by execution_id. See audit-log.md for the verb family and ../ops-runbooks/mcp-bridge-teardown-recovery.md for what to do when the paired alert fires.

Behaviour change: MCP unknown-tool calls now audit result=denied, not failure (#2445)

An MCP tools/call request naming a tool that doesn’t exist has always returned -32601 (kMethodNotFound) with the same error message. The audit row for that rejection previously carried result=failure - the token this codebase reserves for server-side faults - while every sibling denial on the surface (tier, read-only, schema, input-bounds, per-submitter-cap) uses result=denied. It now audits result=denied too, matching those siblings. Nothing about the JSON-RPC response changed.

What to do: if you have a SIEM rule, dashboard or evidence query that classifies mcp.<tool_name> rows by filtering on result=failure to catch unknown-tool probes, re-point it at result=denied before upgrading, or it will stop matching. A rule that already filters on result != "success" sees both and needs no change. Note result=failure was never a clean isolator for unknown-tool probes specifically - it is also emitted for genuine server-side faults on known tools (store degraded, dispatch exception, etc.) and for several other client-caused-but-mislabeled rejections on the same surface (still open, tracked in #3176), so a rule that wants unknown-tool rows specifically should filter on detail="unknown tool", which this change does not touch. See audit-log.md “Result vocabulary” for the surface’s current token usage.

Behaviour change: a failed MCP progress-bridge teardown is now retried instead of being permanently retained (#2513)

A mcp.bridge.* teardown step (unsubscribe, releasing the streamed admission charge, or erasing the correlation record) that failed used to strand the record - and the bus channel, replay buffer, and per-session admission slot it held - for the rest of the process’s life. It now gets up to Config::teardown_retry_max retries beyond the first attempt (4 total attempts by default), one per later sweep tick, before falling back to the old permanent-retention behavior. New yuzu_mcp_bridge_teardown_retry_total{outcome="recovered"|"exhausted"} metric and mcp.bridge.teardown_retry audit action evidence which happened; see audit-log.md and ../ops-runbooks/mcp-bridge-teardown-recovery.md.

What to do: the shipped YuzuMcpBridgeTeardownIncomplete alert changed from a raw counter test (fires and never clears, since the underlying series is monotonic) to a windowed increase(...[15m]) > 0 test, because most of its movement now self-heals via retry within a few sweep ticks. If you copied the old rule into your own alerting config instead of re-importing docs/prometheus/yuzu-alerts.yml, update it the same way or it will stay permanently firing on transient, already-recovered blips. The new YuzuMcpBridgeTeardownRetryExhausted alert is now the actual permanent-retention signal - add it if you track these alerts individually rather than loading the bundled rules file wholesale.

If you have dashboard or SIEM logic reading the pre-existing yuzu_mcp_bridge_teardown_incomplete_total counter directly rather than through the bundled alert - it is unchanged in name and label shape, but its increment semantics changed: previously one increment per stranded record (ever), now up to Config::teardown_retry_max + 1 increments for the SAME record (once per failed step, per attempt) before it either recovers or exhausts. Arithmetic that read the counter as “count of stranded records” will now overcount under retry; read yuzu_mcp_bridge_teardown_retry_total{outcome="exhausted"} for that instead. Config::teardown_retry_max is a code-constant default, not an operator-configurable flag or env var - there is nothing to tune here.

Behaviour change: agent command replay protection is now durable, and a redelivered command replays its real result (HA WS-0, ADR-2002)

The agent’s command deduplication moved from an in-memory set (cleared on every reconnect) to a durable SQLite store (command_dedup.db in the agent data directory). Two consequences an operator or automation may observe:

Breaking for automation/SIEM. A redelivered/duplicate command no longer receives a REJECTED response with the fixed reason "command replay rejected: duplicate command_id". That string is retired. A duplicate now either replays the command’s original terminal frame (its real SUCCESS/FAILURE/REJECTED status, exit code, and structured error — the terminal frame carries the outcome, not the streamed stdout, which is not re-sent) or, while the first attempt is still running, returns a non-terminal RUNNING frame with output "duplicate command still in flight". Any detection rule, dashboard, or evidence query that matched the old rejection string on a redelivered command will silently stop matching — re-point it before upgrading (there is no longer a single “this was a replay” signal on the wire; the server correlates by command_id).

What to do: update or remove any rule keyed on "command replay rejected: duplicate command_id". If you tracked replays for security evidence, note the agent now emits yuzu.dedup_* heartbeat tags (dedup_degraded for an unopenable store; dedup_claim_errors/dedup_record_errors/dedup_release_errors for post-open write failures; dedup_replays) instead; fleet-wide yuzu_fleet_* aggregation + an alert land with the HA observability workstream (WS-11).

New on-disk artifact. command_dedup.db (plus -wal/-shm) appears in the agent’s --data-dir, bounded to the most recent commands. It is a regenerable safety net, not a backup target. Upgrade is fresh-start — no migration from the old in-memory scheme (there was no persistent state to carry); worst case on upgrade is that one already-delivered command could re-execute if redelivered during the restart, exactly as it could before this change. Downgrade is safe: an older agent simply ignores the file.

Degraded mode is fail-open. If the store cannot be opened or written (e.g. a full disk), the agent keeps executing commands undeduplicated rather than refusing them, and surfaces the condition on the heartbeat: yuzu.dedup_degraded=1 for a store that never opened, and the dedup_claim_errors/dedup_record_errors/dedup_release_errors counters for post-open write failures. A per-destructive-instruction-class fail-closed option is planned but not in this release.

Behaviour change: an MCP maintenance failure degrades instead of aborting the server (#2487)

A transient failure in the MCP progress-bridge sweep or the MCP session collector, which runs on the server’s maintenance thread, previously escaped that thread and terminated the whole process. It is now contained: the tick is skipped and retried on the next cycle.

What to do: be aware this is a genuine trade, not a pure win. The old failure was self-announcing - the process died, your supervisor restarted it, and uptime monitoring saw it. The new one is silent unless you are scraping metrics: a sustained failure means bridge teardown or session collection stops running, live MCP progress can degrade for the remaining life of the process, and /readyz will continue to report healthy throughout (MCP session state deliberately does not gate readiness). If you run Prometheus, scrape yuzu_mcp_bridge_teardown_incomplete_total, yuzu_mcp_maintenance_tick_failures_total and yuzu_mcp_bridge_records_active, and load the bundled rules in docs/prometheus/yuzu-alerts.yml - they are not applied automatically. If you do not, you will not see this condition at all.

Note this narrows the crash surface rather than closing it: other work on the same maintenance thread remains unguarded, so a sufficiently severe resource exhaustion can still terminate the process from a different call site.

Behaviour change: yuzu-agent no longer restart-loops forever (ADR-0021 rung 7.7a)

The yuzu-agent systemd unit now sets StartLimitIntervalSec=300 + StartLimitBurst=5 in its [Unit] section. Previously the unit was Restart=always with no start limit, so an agent that could not stay up restarted every 10 seconds indefinitely. With this change, if the agent exits 5 times within 300 seconds systemd stops retrying and the unit enters failed.

This matters because the agent’s Guardian engine can now hard_exit() when an I/O worker is wedged past a grace period (a permanently wedged watched target, e.g. a dead NFS mount or a hung service query); without the start limit that would be a silent 10-second crash loop.

What to do: if you alert on systemd unit state, add yuzu-agent entering failed to your alerts - it means the agent gave up after a crash loop and the device is now dark, which the old restart-forever behaviour hid. Recover with systemctl reset-failed yuzu-agent once the underlying cause (e.g. the wedged mount) is resolved, then systemctl start yuzu-agent. The Guardian spark detection path this rung wires is dormant (--spark-disable unchanged, legacy detection unaffected), so there is no detection-behaviour change to account for. See also server-admin.md “Stopping a wedged agent”.

Behaviour change: a Guardian rule that fails to arm no longer crashes the agent (ADR-0021 rung 7.7b PR-1a)

Independent of the dormant spark path above, the Guardian engine’s apply_rules (which runs on the live legacy detection path regardless of --spark-disable) now firewalls a per-rule reconcile failure. Previously, if arming a rule threw - e.g. a std::thread creation failure under resource exhaustion, or a bad_alloc - the exception escaped onto the agent’s dispatch thread and aborted the whole daemon. It now degrades: the failure is caught and counted, the rule stays persisted, and the agent keeps running. The full_sync teardown and the dispatch boundary carry the same firewall.

One behaviour to be aware of: when a rule fails to arm, the agent holds its policy generation (does not report “caught up”) so the server’s heartbeat reconcile keeps re-pushing and retrying - a transient resource failure self-heals within a reconcile cycle. A rule that fails persistently (rare - it requires a durable resource condition, not bad rule content) keeps the agent behind-generation and drives a re-push each reconcile interval; this is visible in the server’s guaranteed_state.reconcile audit rows for that agent. No operator action is required.

Behaviour change: Guardian event-drop metric narrows to genuine collisions (ADR-0021 rung 7.7b PR-Sv)

Guardian event ingest now distinguishes an idempotent redelivery (a matching-fields event_id PK conflict — the durable agent lifecycle journal’s expected crash-durable, duplicate-tolerant, bounded-retry redelivery, not a guarantee of eventual delivery - an un-acked record that ages out of the 7-day retention window is a rare, counted loss, not silent) from a genuine collision (a mismatched-payload conflict — a forged-id pre-claim or an event_seq_ reset carrying a different event). The eviction-loss counter (yuzu.guardian_journal_evicted_no_send_evidence) is in-memory only and resets on agent restart, and today has no dashboard, REST, or MCP surface at all, per-agent or fleet-wide - see Reconnect replay traffic for the full picture (tracked under #2298). A high redelivery count after an agent outage or reconnect storm is expected and is NOT a loss signal - genuine loss shows up only as a mismatched collision (..._events_dropped_total) or the rare journal-eviction case above. Three operator-visible consequences:

Verify: post-upgrade, confirm the new series are present and re-tune any custom copy of the drop alert:

yuzu_server_guardian_events_redelivered_total   # should appear and climb on reconnects
yuzu_server_guardian_events_ingest_errors_total # should stay at 0 in a healthy fleet

No config or data migration is required.

Behaviour change: a degraded file-change rule now reports itself unhealthy instead of staying silently green (#4748)

A file-change Guardian rule watches its target’s parent directory to detect the target being renamed or moved. If that watch’s own teardown repeatedly fails to confirm (normally only a wedged filesystem driver, or sustained packet loss on a network share), rename detection for that rule permanently disables until the rule is re-armed. Previously the only signal was a single agent log line — the rule kept reporting guard.compliant (the device page’s compliance census showed it green) for as long as the agent ran, even though it could no longer detect a rename of its watched directory.

After this upgrade, a rule that reaches this state reports itself errored on the device page (the same red state as an unresolved drift) instead, with an operator-facing detail explaining the cause and how to recover, and re-asserts every 5 minutes for as long as it stays disabled. The rule’s own file presence/content detection is unaffected either way — only rename detection is lost, and only its reporting changes here.

This is not a retroactive flip. The disabled state is in-memory only (never persisted), so it resets on the very restart that ships this fix — no rule that was showing compliant before upgrading suddenly shows errored at upgrade time. What changes is future behavior: if a rule reaches this degraded state after upgrading (e.g. from another bout of network-share packet loss), it now shows red where a pre-upgrade agent would have stayed silently green. If you see a previously-quiet file-change rule turn red after upgrading, that is very likely this fix correctly surfacing a pre-existing condition on your network share, not a new regression — see the degraded-mode section for the recovery steps (re-push the rule’s policy, re-deploy its Baseline, or restart the agent).

No config or data migration is required.

Behaviour change: dashboard YAML Save is schema-aware and stricter (#1993)

POST /api/instructions/yaml (the New Definition panel’s Save endpoint) and POST /api/instructions/validate-yaml now share one schema-aware parser that accepts the canonical nested InstructionDefinition schema (metadata.id, spec.execution.plugin/action — the documented format every shipped definition uses), which the old substring scanner rejected. In exchange, the endpoint is stricter about malformed input:

Who this affects: automation POSTing YAML to /api/instructions/yaml that (a) omits apiVersion:/kind: lines, (b) relied on a stray name: substring anywhere in the document being picked up as the definition name, (c) re-creates a definition whose metadata.id already exists (now 409 with a denied audit row instead of a silent second copy under a generated id), (d) supplies an explicit id outside [A-Za-z0-9._-]{1,128}, or (e) pastes multi-document (----separated) files (now rejected; save one definition at a time). Interactive panel users are unaffected — the structured form’s output stays accepted.

Verify: re-run your automation against a test server; failures surface as specific per-field messages (e.g. Missing metadata.id (or metadata.name) field), and successful saves now emit instruction.create / instruction.update audit rows.

⚠️ Reserved on-behalf-of headers rejected + principal_class metric label (ADR-1005 Phase 1)

Two operator-visible changes ship together:

1. Reserved headers now 403. Five header names are reserved and rejected on every HTTP endpoint before authentication (On-Behalf-Of, X-On-Behalf-Of, X-Yuzu-On-Behalf-Of, X-Yuzu-Delegated-Operator, X-Yuzu-Delegation-Artifact, case-insensitive; the equivalent gRPC metadata keys cause the call to be cancelled). Nothing previously consumed these headers, so a correctly-built integration is unaffected — but pre-flight check any proxy, service mesh, or SSO gateway in front of Yuzu: some API-gateway products stamp an on-behalf-of/OBO convention header on every upstream request (the name is Microsoft’s Entra OBO term), which would 403 all REST/MCP/dashboard traffic after upgrade. Health probes (/livez, /readyz, /health, /api/health) are exempt, so the pod stays in rotation — a green probe with a 100% 403 rate is the signature (see docs/operations/troubleshooting.md). Rejections are visible in yuzu_onbehalf_rejected_total and throttled [ADR-1005] warn lines.

2. yuzu_http_requests_total gains a principal_class label (human/agent/none; engine reserved). This is a Prometheus series-identity change on a long-shipped metric: old {method,status} series freeze at upgrade and new {method,status,principal_class} series start at zero. Plain selector queries and sum by (method, status) keep working; re-baseline any dashboard or alert that matches the exact label set or joins on series identity.

⚠️ Breaking: account lockout is ON by default

This release adds account lockout for failed local-password logins (SOC 2 CC6.3) and it is active by default (--auth-lockout-threshold=5, --auth-lockout-window-secs=900) on any deployment that runs with a persistent auth database — i.e. one started with --data-dir.

⚠️ --data-dir is required. Lockout state lives in the persistent auth.db, which only exists when the server is started with --data-dir. The shipped container images and compose files pass --data-dir /var/lib/yuzu, so lockout genuinely is on by default there. The systemd/.deb unit now also passes --data-dir /var/lib/yuzu (added in this release) — but if you run a custom invocation without --data-dir, the server falls back to in-memory auth and lockout (and session persistence) is silently off. The server logs a loud startup WARN in that state. Set --data-dir to make the control active.

Superseded by the AuthDB→Postgres migration (see below). Lockout state now lives in Postgres (users.failed_login_count / locked_until), not auth.db. It is active whenever --postgres-dsn (or YUZU_POSTGRES_DSN) is set — which the server now requires to boot at all (no SQLite fallback) — and --data-dir no longer gates it. Read the --data-dir requirement above as historical, for pre-migration builds only.

No further config change is required to opt in — a deployment that already runs with --postgres-dsn gains the behavior the instant you start the new build.

What changes on upgrade:

Highest-risk targets on upgrade: shared or service accounts that log in with a password, and any password-rotation / monitoring automation that may submit a stale password in a loop — these can now lock themselves out where previously nothing happened.

Before upgrading, do ONE of:

  1. Accept the default (recommended for most) — it closes a real credential-stuffing surface. Make sure operators know the recovery path below.
  2. Raise the threshold if you also rate-limit at the network layer (NIST 800-63B §5.2.2 suggests ≥10): --auth-lockout-threshold=10.
  3. Disable it (not recommended; constitutes a deviation from the CC6.3 hardened baseline): --auth-lockout-threshold=0 (the server logs a startup WARN).

Recovery if an account is locked out:

Behaviour change: operator/API tags now beat agent self-report (#1411)

An agent’s self-reported tags (scopable_tags, synced on every Register) can no longer overwrite an operator- or API-set tag for the same (agent_id, key) — the operator/API value is now authoritative. This closes a path where a rogue or misconfigured agent could self-assign into an operator-declared benchmark cohort.

Who this affects: only an operator who deliberately relied on agent-reported values winning over an operator/API-set tag for the same key. After upgrade those agent values stop overriding — silently (no error, no log line); an affected device simply drops out of the cohort the operator-set value defines.

Verify: audit the source field via MCP get_tags — GET /api/v1/tags does not expose source, only key/value pairs (see the read-side note below) — to see whether each key is api/mcp-sourced (operator/API) or agent-sourced.

Read-side completion (#3295): #1411 closed the write-time overwrite; scope-DSL tag:<key> evaluation (dispatch targeting, management-group membership) had its own, separate precedence — a currently-connected agent’s live self-report answered before the store was even consulted. That is now also store-first: an operator/API-set row always wins for scope-DSL purposes too, and an agent-claimed service value never answers from the live self-report at all (it is dropped at registration, not just at store sync). Who this affects: an operator who deliberately relied on a live agent’s tag value overriding an operator/API-set tag specifically for dispatch targeting or cohort membership while that agent stayed connected. See docs/asset-tagging-guide.md “Tag source precedence (read time, scope-DSL, #3295)”.

Caution: the /devices tag-chip display is unaffected by this change and can now visibly disagree with what actually governs targeting: it renders only the agent’s live self-report, never the TagStore row, so a store value that now wins for dispatch purposes — of any source, including a previously-synced agent value that’s now stale relative to the agent’s live report, not only an operator-set one — may not be what the dashboard chip shows. When a store row exists for the key, MCP get_tags (its source field) is the source of truth for what governs a device’s scope-DSL matching, not the dashboard; GET /api/v1/tags?agent_id=<id> does not expose source. For a gateway-proxied or not-yet-synced agent with no store row at all, the live self-report (what the dashboard shows) is what actually governs — an empty get_tags/GET /api/v1/tags result for a key means the live claim decides, not that nothing does.

Remediate: if an agent-reported value was the intended one, re-set it explicitly via the REST API (writes source=api) or MCP set_tag (writes source=mcp) — either is authoritative over an agent-sourced row. Keys the agent reports that the operator never set are unaffected.

Behaviour change: MCP approval-gated calls use ticket-then-recall (-32006) (#289)

Supervised-tier MCP tokens that attempt an approval-gated operation (e.g. delete_tag, quarantine_device) now run through a ticket-then-recall approval flow. The first call returns JSON-RPC error -32006 (ApprovalRequired) whose error.data carries an approval_id and a status_url (GET /api/v1/approvals/{id}). After a second principal approves the ticket, the caller re-invokes the same tool with the approval_id and the operation executes (the ticket is consumed exactly once, args-bound, replay-rejected).

An earlier development iteration returned -32004 (TierDenied) for these operations because the approval re-dispatch path was unbuilt. That intermediate behaviour never shipped in a release and is fully superseded by the flow above — there is no -32004 approval-denial in the released product.

Who this affects: any MCP client that matched on the approval-gated denial. Match -32006 and read error.data.approval_id / status_url to drive the approval workflow; operator-tier executions are auto-approved and are unaffected. On the REST transport an approval-gated operation by an MCP-tier token is denied (there is no JSON-RPC ticket channel on REST) — use the MCP /mcp/v1/ flow or the dashboard.

Three review-round hardenings in the same release (PR #1796):

Behaviour change: POST /api/v1/tokens now honors mcp_tier

The token-mint endpoint now applies the mcp_tier field from the request body (readonly / operator / supervised). Previous releases silently dropped it: a token requested with an mcp_tier was stored tier-empty (RBAC-deferred), which means it was over-privileged relative to the tier the operator intended — a readonly request behaved as a full RBAC-scoped token, not a read-only MCP token. The endpoint also now validates at mint time: an unrecognised tier, or a tiered/service-scoped token whose expires_at is missing or more than 90 days out, returns 400 (previous releases returned a misleading 503).

Who this affects: any operator who minted an MCP token via POST /api/v1/tokens before this upgrade and relied on the mcp_tier to constrain it. Those tokens are still tier-empty and therefore broader than the requested tier.

Remediate: rotate any pre-upgrade mcp_tier tokens — revoke the old token (DELETE /api/v1/tokens/{id}) and re-mint it; the new token stores the tier and is constrained as intended. Tokens minted with no mcp_tier are unaffected.

Additive (no operator action required): the discover_plugins discovery response (REST GET /api/v1/discover/plugins and the MCP tool) is now version: 2 — it adds an inline parameter_schema per action (where a published InstructionDefinition exists and the caller holds InstructionDefinition:Read) plus a top-level actions_enriched_with_schema count. The change is purely additive; any client that pinned version == 1 should relax the check to a minimum (>= 1).

⚠️ Breaking: API and MCP bearer tokens are invalidated on upgrade (ApiTokenStore → Postgres, ADR-0030)

ApiTokenStore — the store backing every Authorization: Bearer API token and every MCP token — moves from SQLite (api-tokens.db) to the server’s PostgreSQL substrate in this release (engine-principals PR 4.1; ADR-0006/ADR-0030). This is a fresh-start cutover with no data migration: the new server creates an empty Postgres api_token_store.api_tokens table and never reads the old api-tokens.db, so every API/MCP token minted before the upgrade stops working the instant the new server starts. Interactive cookie-session login (dashboard, OIDC/SAML SSO) is not affected — only bearer tokens.

The rationale for no backfill (convention with every prior server-store migration, plus the token store holds only verify-only hashes) is in ADR-0030.

Who this affects: every external automation consumer that authenticates with an API or MCP bearer token — CI/CD scripts, SIEM/webhook pollers, cron jobs, MCP clients. Because it is a fresh-start cutover, all of them break at once at the moment of upgrade; there is no rolling/staged token continuity.

Remediate: after upgrading, re-mint every API/MCP bearer token (POST /api/v1/tokens) and update the credential wherever it is stored (CI secrets, cron configs, MCP client configs). Plan the upgrade in a maintenance window and notify automation/integration owners in advance so a batch of 401s across every integration is expected, not a surprise incident.

Diagnostic: if the legacy api-tokens.db is still on disk, the server logs [auth] Legacy SQLite api-tokens.db found at … — API/MCP tokens now live in PostgreSQL and any prior tokens were INVALIDATED by the migration (ADR-0030) at boot. The old file is inert (never read again) and can be removed once you have re-minted; it contains only token hashes + metadata, no plaintext secret.

Multi-instance note: the Postgres substrate now permits running multiple server replicas against one database. Token validation uses a per-process 60-second cache, so a token revoked on one replica may remain accepted on another replica for up to that window — see ADR-0030 for the tracked hardening.

⚠️ Breaking: engine: namespace reservation — server refuses to start on a pre-existing collision (engine principals PR 4.2)

This release introduces the engine principal class — a distinct identity class for autonomous/agentic callers, stored in the new EnginePrincipalStore and resolved through RBAC/ApiTokenStore (design doc §3.1/§3.3). To make that resolution unambiguous, the engine: prefix is now a reserved namespace for both local usernames (users.username) and local RBAC group names (groups.name).

Pre-upgrade collision check (mirrors the plugin-trust-bundle filename collision check above). At boot, the server now scans for any pre-existing engine:-prefixed local user or local RBAC group and refuses to start if it finds one — silently coexisting with (or being shadowed by) a real engine principal is not an acceptable outcome, so this fails closed rather than booting into an ambiguous state (decision log #3). Before upgrading, run against both stores.

Superseded by the RbacStore → PostgreSQL migration (ADR-0041): the per-node sqlite3 rbac.db query below is a HISTORICAL artifact from the era when local RBAC groups were SQLite. RBAC config is now a single shared PostgreSQL rbac_store schema, so run one psql query, not per-node sqlite3. Both queries below target Postgres:

# Local users — Postgres.
psql "$YUZU_POSTGRES_DSN" -c \
  "SELECT username FROM auth.users WHERE username LIKE 'engine:%';"

# Local RBAC groups — PostgreSQL (shared; one query, not per-node — ADR-0041).
psql "$YUZU_POSTGRES_DSN" -c \
  "SELECT name FROM rbac_store.groups WHERE source = 'local' AND name LIKE 'engine:%';"

If either query returns rows, rename or remove those users/groups before upgrading — the new server will not boot until the collision is cleared (or, if the boot-time collision scan itself fails, e.g. a mid-scan database error, the server also fails closed rather than guessing). See docs/ops-runbooks/engine-principal-store-recovery.md for the full recovery procedure if you hit this at boot.

Scope of this release: PR 4.2 ships the engine-principal store, RBAC resolution, and attribution plumbing only — there is no operator-facing surface yet (no dashboard/REST CRUD for minting or managing engine principals). That ships in PR 4.3.

⚠️ Breaking: --mfa-enforcement now enforces

Releases before this one accepted --mfa-enforcement=admin-only and --mfa-enforcement=required but treated them as no-ops (the parser accepted the value for forward-compat and the server emitted a startup WARN). This release makes them enforce. If you staged the flag based on that prior documentation, enforcement goes live the instant you start the new build.

What changes on upgrade if the flag is set to admin-only or required:

Before upgrading with the flag set, do ONE of:

  1. Recommended: leave the flag at optional, upgrade, have all affected users enroll (Settings → Security and sign-in → Two-factor sign-in), then set admin-only/required and restart. or
  2. Upgrade with the flag set and accept that affected un-enrolled users will be walked through enrollment on their next login.

SSO / OIDC pre-flight (required reading if you use SSO): under required (and admin-only for admin SSO users), OIDC sessions are MFA-gated by the IdP’s amr claim — an SSO login the IdP did not MFA is blocked from high-risk endpoints (it must re-authenticate via SSO), symmetric with a local user being forced to enrol. Yuzu cannot mint a second factor for an external identity, so before enabling required/admin-only with SSO you MUST verify your IdP asserts an amr claim containing a recognized MFA method (Entra: mfa; others: otp/hwk/etc., RFC 8176). If it does not, affected SSO users will be unable to reach high-risk endpoints — recoverable by restarting in optional (see docs/ops-runbooks/auth-db-recovery.md § “Locked out by MFA enforcement misconfiguration”; that runbook is now Postgres-native throughout). Under optional, no IdP amr configuration is required (SSO sessions pass the gate).

Single-admin deployments: do not first-boot a fresh single-admin deployment straight into required. Enroll the admin under optional first, then switch. If you do start with required, the admin must complete login-time enrollment within --mfa-login-pending-secs (default 120s); if the token expires, restart with optional, log in, enroll, then re-enable.

Recovery if you get locked out (IdP doesn’t assert amr, or the sole admin can’t enroll): restart the server with --mfa-enforcement=optional (this re-seeds the in-memory config), log in, resolve enrollment, then re-enable. See docs/ops-runbooks/auth-db-recovery.md — that runbook is now Postgres-native throughout, so both the restart procedure above and its SQL steps apply as written to a Postgres-backed AuthDB.

Hardened auth mode (--auth-mode=sso-only) — opt-in (SOC 2 CC6.3/CC6.6)

No operator action on upgrade. Existing deployments default to --auth-mode=standard and behave exactly as before. This release adds the option to disable local-password login fleet-wide so only an SSO provider can mint a session.

If you intend to enable sso-only, do this first — the server fails closed (non-zero exit, refuses to serve) otherwise:

  1. Configure OIDC (--oidc-issuer + the related flags) and confirm SSO works in standard mode. sso-only without an SSO provider (OIDC, or a complete SAML SP config on Linux/macOS with HTTPS) refuses to start (it would lock every operator out).
  2. (Recommended) Create a single break-glass local account and enroll MFA on it (Settings → Security and sign-in → Two-factor sign-in). Point --break-glass-user at it. sso-only refuses to start if the named break-glass user doesn’t exist or has no MFA enrolled — a break-glass account must carry a second factor.
  3. Restart with --auth-mode=sso-only. Local-password login is now rejected for everyone except the break-glass account, and only while armed.

Breaking the glass (IdP outage). The break-glass account is dormant until armed out-of-band on the server host: yuzu-server --break-glass-arm --break-glass-user <name> --config … --data-dir … --postgres-dsn … arms it for --break-glass-window-secs (default 24 h, auto-expiring). It still requires the account’s MFA at login. Since the AuthDB→Postgres migration this one-shot opens the Postgres auth store, so --postgres-dsn (or YUZU_POSTGRES_DSN) is required — point it at the same database the server uses; without it the command fails closed with “requires the Postgres auth store” and does nothing. --data-dir is still needed for the audit record. Full runbook: docs/ops-runbooks/auth-db-recovery.md § Break-glass arm (that runbook is now Postgres-native throughout).

Migration. break_glass_armed_until is a nullable column on auth.users. It arrived as SQLite auth.db migration v4, but the AuthDB→Postgres cutover folded every historical migration into the born-on-PG schema, which starts at v1 — there is no v4 to apply and no rollback-past-v4 question. See the AuthDB→Postgres section above for the (fresh-start, no-backfill) upgrade contract that replaces it.

⚠️ Breaking: server generates default TLS certificates on first boot (v0.13.0)

Before v0.13.0 the server refused to start without operator-provided certs (or --no-tls/--no-https). From v0.13.0, when a TLS surface has no certs the server auto-generates a per-install ECDSA CA + leaf certs on first boot and serves encrypted with no operator action.

Impact by prior configuration:

Prior startup flagsAfter upgrade
--cert/--key (+ --https-cert/--https-key) suppliedNo change — operator certs always win; defaults are never generated for a supplied surface.
--no-tls --no-https (plaintext dev)No change — both surfaces disabled; no certs generated.
--no-tls only (HTTPS previously errored without certs)HTTPS now serves an auto-generated default cert instead of failing. The agent surface stays plaintext.
No cert flags (previously refused to start)Now starts, encrypted, on default certs, with a loud banner.

What to expect / do:

Generic inventory store no longer reads or scrubs a legacy inventory.db (ADR-0037)

InventoryStore’s one-time first-boot backfill from a legacy inventory.db SQLite file (migrate_from_sqlite()) has been retired (#3623) — no production fleet ever ran a pre-Postgres build, so there was no real legacy data to migrate. Two separate behaviors changed:

Verify: after the server reports ready, generic inventory reads (GET /api/v1/inventory or equivalent) behave exactly as before — this store’s runtime read/write/query behavior is unchanged, only the one-time backfill and the legacy-file erasure are gone.

Rollback caution. Rolling back to a build older than this release, against a database that has already run this store’s retirement migration, unconditionally fails that older binary’s boot with a Postgres undefined_table error on its own idempotency-marker check — not only when a legacy inventory.db happens to be present. This is a deliberate fail-safe (a clear startup error, never silent data loss), not a bug, but it means “just roll back the binary” is not a supported recovery path once this migration has run. The same is true for CaStore/ManagementGroupStore/QuarantineStore/WebhookStore below, each for its own dropped marker table. RbacStore is the one genuine exception (see its own section): its retirement migration poisons the marker rows to a sentinel value rather than deleting them, so an old binary’s own lookup finds them present (not 0 rows) and — reading a value it understands as “no verified source, but not a failure” — boots normally UNLESS a real, unmigrated legacy file is also present on that specific host. This is RbacStore’s own deliberate choice to tolerate rollback, not an artifact of a row count. AuditStore also removes marker rows rather than dropping a table, but do NOT assume it behaves the same way as either group — its own section explains why an old binary’s boot fails just as reliably as the DROP-TABLE group’s, via a CHECK constraint rather than a SQL error on a missing table.

⚠️ Behaviour change: internal-CA store moves to Postgres (ADR-0053)

CaStore (internal-CA root metadata, issued-certificate inventory, CRL version history — everything the mTLS-accept revocation gate and GET /api/v1/ca/* read) moves from the SQLite ca.db file to the server’s PostgreSQL substrate in this release (schema ca_store). The private CA root key is unaffected — it was never in ca.db and stays a local file behind KeyProvider (--ca-dir).

⚠️ Behaviour change: webhook store moves to Postgres (ADR-0057)

WebhookStore (operator-registered outbound webhooks and their delivery log — everything behind GET/POST/DELETE /api/webhooks and GET /api/webhooks/{id}/deliveries) moves from the SQLite webhooks.db file to the server’s PostgreSQL substrate in this release (schema webhook_store). The webhook HMAC signing secret is now envelope-encrypted at rest (SecretCodec, AES-256-GCM, ADR-0010) instead of a plaintext column.

⚠️ Behaviour change: response history resets on Postgres cutover (ADR-0039)

ResponseStore (agentic command/instruction results — the executions drawer and Echo read source) moves from the SQLite responses.db file to the server’s PostgreSQL substrate in this release (ADR-0006 Wave 1, schema response_store). Like the ApiTokenStore and AuthDB cutovers, this is a fresh-start cutover with no data migration — response results are expendable, TTL’d telemetry (ADR-0009 skippable backfill), so the legacy responses.db is never read on upgrade.

What happens on first PG boot:

Also in this release: non-UTF-8 bytes in a plugin’s output/error (which the old SQLite TEXT column tolerated but PostgreSQL TEXT rejects) are now replaced with the Unicode replacement character (U+FFFD) at ingest — the response row is still stored and still renders, defanged, rather than being dropped (governance #1593). Retention moved from an hourly background thread to a clock-guarded, capped reap on the maintenance tick (no operator-visible behaviour change beyond the same 90-day default).

⚠️ Behaviour change: directory sync state resets on Postgres cutover (ADR-0063)

DirectorySync (AD/Entra ID integration — synced users/groups, group memberships, and group→role mappings) moves from the SQLite directory-sync.db file to the server’s PostgreSQL substrate in this release (migration-programme PR 3, schema directory_sync). This is a fresh-start cutover with no data migration (ADR-0009) — the legacy directory-sync.db is never read on upgrade.

What happens on first PG boot:

Also in this release: directory_memberships gained a real foreign key (ON DELETE CASCADE on both user_id and group_id) — the SQLite era had none. /readyz and /healthz now report directory_sync — the SQLite-era store was never checked at boot or reflected in either probe.

Behaviour change beyond the storage cutover itself: a user or group deleted from Entra now actually disappears from DirectorySync on the next successful sync — the SQLite era kept every previously-synced user/group around forever, upserting but never removing one Entra had deleted. If your integration (or a saved report) relied on a removed identity remaining visible via GET /api/directory/users/GET /api/directory/status after leaving Entra, it will stop being visible starting with the first post-upgrade sync. Relatedly, a malformed/unexpected Microsoft Graph groups API response is now a hard sync failure (surfaced in GET /api/directory/status’s last_error) rather than a silently-empty group sync — this closes a gap the deletion behavior above would otherwise have introduced (a transient Graph glitch wiping every synced group).

⚠️ Behaviour change: schedules, approvals, and execution history reset on Postgres cutover (ADR-0065)

ScheduleEngine, ApprovalManager, and ExecutionTracker move from the shared SQLite instructions.db file (via InstructionDbPool) to their own independent PostgreSQL schemas in this release (migration-programme PR 5, the last of the components the postgres-migration ladder had never tracked — WorkflowEngine/ADR-0064 having already made the same move in PR 4 — schemas schedule_engine, approval_manager, execution_tracker respectively). This is a fresh-start cutover with no data migration (ADR-0009) — the legacy instructions.db is never read for data on upgrade. InstructionDbPool itself is deleted; instructions.db is retired and no Yuzu store writes to it again.

Before upgrading, capture what you’ll lose (none of it carries over): GET /api/schedules, GET /api/approvals, and GET /api/executions are safe, non-mutating reads — use them to record your active recurring schedules and any outstanding (pending or approved-but-unconsumed) approval tickets before you cut over, so you know what to re-create afterward rather than discovering gaps after the fact. GET /api/executions defaults to the 100 most recent rows (?limit=<N> to raise it — there is no offset/cursor parameter) — for a fleet with more history than that, raise the limit before relying on this as a full capture. This capture step runs on the pre-upgrade binary, which necessarily predates #3789 — the ADR-0065 Postgres cutover this section documents shipped before #3789 did, so any binary old enough to still need this procedure cannot yet carry #3789’s limit cap; raise ?limit as high as needed for this specific step. (A binary that already carries #3789, by contrast, caps GET /api/executions’ limit at 500 regardless of the value requested, with no offset/cursor to page past it — see docs/user-manual/rest-api.md’s “Executions” section — but that cap cannot bite on this already-completed cutover’s capture step.) The retired instructions.db file is left on disk, not deleted — if you need to recover consumed-approval audit evidence (the submitted_by → reviewed_by → consumed_by chain) after upgrading, that file is the most complete source (the audit store also carries approval.approve/approval.reject and mcp.<tool> consume events for the same tickets, but not the full row); back it up separately before any unrelated disk cleanup removes it, since Yuzu itself never will.

What happens on first PG boot:

Also in this release: advance_schedule’s locked select-then-compute- then-update collapses into one atomic UPDATE ... RETURNING, closing a two-statement race an app-level lock previously covered — no operator-visible behavior change. ExecutionTracker’s one sqlite3_changes() call (gating the terminal-transition SSE event on the shared SQLite connection) is closed via UPDATE ... RETURNING, fixing a latent race under concurrent callers — also no operator-visible change; the executions-history SSE contract (progress-before-terminal event ordering) is unchanged. The MCP approval error envelope’s permanent-vs-transient discriminator changes internally from a raw SQLite extended error code to a Postgres SQLSTATE string — the same two response shapes are still chosen by the same rule, so no client migration is needed.

Behaviour change: per-device dispatch can now refuse a target it always accepted before (ADR-1007)

ExecutionTracker gains a new table, concurrency_claims (same execution_tracker schema introduced above), backing real enforcement of the per-device concurrency mode — the shipped default and the only one of the five documented modes that was ever actually enforced (see docs/user-manual/instructions.md §8 and ADR-1007 for the full real-usage audit; per-definition/global/global-singleton/per-set remain accepted but unenforced, unchanged from before this release).

Operator-visible: dispatching a per-device definition (via POST /api/instructions/:id/execute or a ScheduleEngine run) to a device that already has an execution of that same definition in flight now excludes that device from the target list instead of sending a second, overlapping command as it always did before. This is the intended fix, not a regression — but any automation that dispatches the same definition repeatedly in quick succession to the same fleet (e.g. a tight retry loop) may see partial dispatch where it previously always saw full dispatch. Watch the new yuzu_server_dispatch_concurrency_skipped_total counter (or ExecutionTracker::claim_concurrency_slots’s spdlog::info line, which reports a count — how many of the dispatch’s candidates were already busy for the definition — not individual device ids) to see when this fires — the executions API’s agents_targeted field is the post-claim SENT count, not a pre-claim total, so there is no “targeted vs. claimed” comparison to read it against. Error code 3003 (ConcurrencyBlocked) is registered for this condition but not yet surfaced through the dispatch response.

Quarantining a device mid-flight (after it was claimed but before it reports a terminal status) behaves the same as a crash for this purpose — its claim ages out at the reconciler’s TTL rather than releasing immediately, so a device released from quarantine may still read as “busy” for a per-device definition for up to that TTL window.

The claim’s exclusion window is bounded by a flat one-hour timeout, renewed by a dedicated agent-core keepalive that runs independently of any given plugin — so a long-running script_exec.* action (a MUTATING instruction — runs an operator-named program or authored script that can hold real state) with an operator-configured timeout near the one-hour ceiling stays correctly excluded even if it produces little or no output. The keepalive sends one empty gRPC frame every 5 minutes per in-flight per-device command — negligible traffic even at fleet scale, but worth knowing about if you’re diffing agent-to-server wire volume before/after this release. See ADR-1007 for the full mechanism.

Rolling-upgrade behaviour: the keepalive lives in the agent binary, not the server — during a staged agent rollout, a per-device command running on an agent still on a build older than this release has no keepalive thread, so its claim relies on the same rarer, less reliable renewal signals this release replaces (a plugin’s own output volume crossing an internal threshold, or nothing at all for a quiet action). This is a narrowing of a pre-existing exposure, not a new one — an old agent’s long-running quiet command was always at risk of this race before this release; upgrading the server first only protects commands running on already-upgraded agents. Upgrade agents promptly to get full coverage; per the standing rule below, never upgrade agents before the server.

A claim is released when the excluded-from device’s prior execution reaches a terminal state; an execution that never gets a terminal response (agent crash or disconnect mid-run) has its claim force-released by a periodic reconciler, bounded by the claim’s own expires_at — worst case, a device can be excluded from a repeat per-device dispatch of the same definition for up to one hour (a claim’s lease is capped at one hour regardless of the originating command’s own expiry). Cancelling an execution does not release its claim — there is no way to tell the agent to stop, so a cancelled-but-still-running execution’s claim behaves exactly like a normal in-flight one until a terminal response or the reconciler’s TTL bound releases it. A per-device definition dispatched via a workflow step (rather than a direct POST /api/instructions/:id/execute call or a schedule) releases on the agent’s real terminal response and renews on its keepalive the same as every other dispatch path — workflow-step dispatch still does not correlate a real execution id for the executions drawer (so its progress won’t show there), but that gap no longer affects claim timing: it is not release-by-TTL-only. Raw MCP execute_instruction and any other dispatch call that does not resolve an InstructionDefinition are unaffected (no definition_id in scope, so this gate never fires for them), as is the fleet-broadcast dispatch arm. No operator action is required to upgrade.

Also in this release: DirectorySync::sync_entra (POST /api/directory/sync) now rejects a concurrent call while a sync is already in progress with 409 Conflict instead of racing — previously two overlapping sync calls could interleave writes. A caller that retries on 409 behaves correctly; one that treated every non-2xx response as a hard failure should special-case 409 as “try again shortly.” ConcurrencyManager (dead code with zero production callers, already flagged in docs/postgres-migration-ladder.md) is deleted rather than migrated to Postgres.

⚠️ Behaviour change: buffered analytics events reset on Postgres cutover (ADR-0049)

AnalyticsEventStore (the outbox spool behind /api/analytics/status and /api/analytics/recent — login/MFA/OIDC/SAML/role-elevation and agent/gateway command-lifecycle events, drained to any configured JSONL/ClickHouse sink) moves from the SQLite analytics.db file to the server’s PostgreSQL substrate in this release (ADR-0049, Wave 2 batch 3, schema analytics_event_store). Like ResponseStore, this is a fresh-start cutover with no data migration — the buffer is a transient spool, not authoritative or compliance evidence (ADR-0009 skippable backfill), so the legacy analytics.db is never read on upgrade.

What happens on first PG boot:

RBAC store moves to PostgreSQL (RbacStore → Postgres, ADR-0041)

RbacStore — the authorization substrate holding role definitions, role→permission grants, principal→role assignments, RBAC groups + membership, and the global rbac_enabled flag — moves from the SQLite rbac.db file to the server’s PostgreSQL substrate in this release (ADR-0041, Wave 2.1), schema rbac_store. No new flag or environment variable is added (it reuses the shared server PgPool).

No legacy-SQLite migration path. No production fleet ever ran a pre-Postgres build of this store, so there was no real rbac.db data to carry over — the mandatory backfill this section originally described (including the rbac_enabled-flag-first transfer and the move-aside/fingerprint-verify mechanism) was retired under ADR-0009’s fresh-start-by-default amendment (see ADR-0041’s Update). These are two SEPARATE behaviors, not one — do not conflate them:

If you see the legacy-row-count warning and the environment genuinely has real RBAC config to keep, there is no automated recovery path: re-author the equivalent roles/grants/groups against the new Postgres-backed store via the RBAC admin UI/REST API, and re-check rbac_enabled after boot (it seeds false on a fresh schema — it does NOT carry over from a legacy file, so an operator who had RBAC enabled must re-enable it explicitly).

Rollback caution. Do not roll back to a build older than this release while a legacy rbac.db file with real content still sits at its configured path on any replica. An older binary’s own (now-removed-from-current-code) backfill logic still runs on that replica and, seeing this database already past this migration, refuses to auto-migrate rather than silently doing anything to rbac_enabled — you’ll see a clear startup refusal, not silent corruption, but it IS a startup refusal. Move the legacy file aside first if a rollback is genuinely needed.

What to expect / do:

See docs/auth-architecture.md § “RbacStore — the authorization substrate” and ADR-0041 for the full design.

⚠️ Breaking: local accounts + MFA enrolments reset (AuthDB/ScimStore → Postgres, ADR-0006)

AuthDB (local user accounts, MFA enrolments, enrollment tokens) and ScimStore (SCIM-provisioned users/groups) move from the SQLite auth.db file to the server’s PostgreSQL substrate in this release (ADR-0006 Wave 3), schemas auth and scim_store. Like the earlier ApiTokenStore cutover, this is a fresh-start cutover with no data migration: on first boot against a Postgres database whose auth.users table is empty, the server seeds exactly the config-file admin and never reads the old auth.db. Sessions are (and always were) in-memory-only, so they are unaffected beyond the usual “log in again after a restart.”

Who this affects: every local-password operator, everyone with TOTP MFA enrolled, and every SCIM-provisioned user/group. This is not limited to password auth — SSO/OIDC/SAML session establishment itself is unaffected (the IdP still asserts identity), but any RBAC role/group mapping recorded only in the old auth.db local-account tables is gone until re-established.

What happens on first PG boot:

Remediate:

  1. Plan the upgrade in a maintenance window and notify every local-password operator, MFA-enrolled user, and SCIM administrator in advance — expect a batch of failed logins immediately after cutover, not a surprise incident.
  2. After upgrade, log in as the re-seeded admin using the credentials in yuzu-server.cfg.
  3. Re-create any other local operator accounts that are not SCIM-provisioned (Settings → Users), and have every user re-enrol MFA (Settings → Security and sign-in → Two-factor sign-in).
  4. For SCIM-provisioned users, either wait for the next scheduled IdP sync or trigger one manually if your SCIM setup supports it; do not manually re-create SCIM-managed users (they will be re-provisioned).
  5. Confirm the AUTH DATA RESET ON POSTGRES CUTOVER line appeared in the server log at the boot where the cutover ran.

Not affected: the PgTestTemplate config, engine principals (a separate store), and API/MCP bearer tokens (already migrated to Postgres in an earlier release, see “⚠️ Breaking: API and MCP bearer tokens are invalidated on upgrade” above — this AuthDB/ScimStore cutover is independent of that one and does not re-invalidate already-reissued tokens).

See docs/auth-architecture.md § “AuthDB — persistent authentication store” for the full design rationale, and docs/ops-runbooks/auth-db-recovery.md for lockout/break-glass recovery — that runbook has been rewritten for this cutover and is Postgres-native throughout (psql "$YUZU_POSTGRES_DSN" against the auth schema).

Notification feed moves to PostgreSQL (NotificationStore, ADR-0046)

NotificationStore — the dashboard toast/badge feed — moves from the SQLite notifications.db file to the server’s PostgreSQL substrate in this release (ADR-0006 Wave 2, ADR-0046), schema notification_store. No new flag or environment variable is added (it reuses the shared server PgPool).

No legacy-SQLite migration path. No production fleet ever ran a pre-Postgres build of this store, so there was no real notifications.db data to carry over — the one-time mandatory backfill this section originally described was retired under ADR-0009’s fresh-start-by-default amendment (see ADR-0046’s Update).

These are two SEPARATE failure/detection behaviors, not one — do not conflate them:

Not affected: /api/notifications* request/response behavior is unchanged — this is a storage-engine swap only, no API change.

Audit trail on PostgreSQL — legacy audit.db no longer imported (AuditStore, ADR-0040)

The audit log (AuditStore, the SOC 2 evidence chain) runs on the server’s PostgreSQL substrate (ADR-0006 Wave 1.3, schema audit_store). No new flag or environment variable is needed: the store reuses the shared --postgres-dsn / YUZU_POSTGRES_DSN connection the rest of the server already requires, and (like every server store) it fails closed at boot if Postgres is unreachable — there is no SQLite fallback.

No legacy-SQLite migration path. This release retires AuditStore’s one-time, mandatory, streamed backfill from a legacy audit.db (#3623, chore/retire-migrate-from-sqlite-auditstore, ADR-0009’s hard-cutover Update) — no production Yuzu fleet has ever run a pre-Postgres build, and this is a hard cutover with no migration path held open for any store, including this one, accepting permanent audit-trail loss as the failure mode if that premise is ever wrong for a specific host. AuditStore had been the one deliberate, permanent exception to ADR-0009’s fresh-start default — audit evidence cannot be regenerated the way config or cache state can — until this release withdrew it. These are two SEPARATE behaviors, not one — do not conflate them:

If you see the legacy-row-count warning and the environment genuinely has real audit history to keep, there is no automated recovery path: the pre-cutover trail stays exactly where it is, at its configured path (0600 now enforced on POSIX, #3593; content otherwise untouched) — and is your operator-managed record going forward (export it, archive it, keep it under your own retention policy; Yuzu will never import it). The generic “reapply manually” phrasing in the boot warning fits config-shaped stores below (a webhook subscription, a notification preference) where reapplying means re-creating the row through the API; it does not apply here in that sense — an audit trail is historical evidence, not something you recreate, so “reapply” for AuditStore means export-and-archive, never re-import.

Rollback caution. Do not roll back to a build older than this release. Unlike RbacStore above, this is not just a caution about a leftover legacy file — an older binary’s own (now-removed-from-current-code) backfill logic refuses to boot unconditionally, with or without a legacy audit.db present: seeing this database already past this migration (the marker rows are gone, but audit_retention_meta itself is not — see ADR-0040’s Update), its own idempotency-marker write hits the SAME migration’s CHECK constraint (audit_retention_meta_no_retired_backfill_markers) and fails with a Postgres 23514 check_violation, table empty or not, legacy file present or not. This is the same class of guarantee the DROP-TABLE-group stores above get from their undefined_table error — a schema-level rejection, not a live-row-count race — with one caveat: if a legacy audit.db is present, the old binary’s row inserts into audit_events still run and commit before this final write fails, so the refusal is loud and deterministic but not zero-data-movement the way a first-touch undefined_table error is. “Just roll back the binary” is not a supported recovery path once this migration has run, the same as every DROP-TABLE-group store above. If AuditStore runs as more than one replica, upgrade every replica and any --mfa-reset/--break-glass-arm CLI image together during a staged rollout — a replica or one-shot CLI still on the old binary that boots after this migration has run on the shared database will refuse to start, same as any other rollback attempt.

What to do: the YuzuAuditBackfillFailing alert is removed with this release. If you copied it into your own alerting config instead of re-importing docs/prometheus/yuzu-alerts.yml, remove it or it fires permanently via absent_over_time on a stale local copy — a CRITICAL page that never clears, pointing at boot-log remediation steps that no longer exist.

Reads deny-on-degrade. An audit-store or connection-pool failure makes GET /api/v1/audit* return 503 rather than an empty 200, so an infrastructure blip can never be mistaken for “no audit activity.”

Not affected: the audit event vocabulary and REST/MCP query surface are unchanged; SIEM export recipes keep working. One deliberate behaviour change: on a multi-replica deployment an identical-magnitude repeat clock step no longer re-emits yuzu_server_audit_clock_anomaly_skips_total on every pass (only a distinct anomaly does) — if you alerted on that counter’s cadence, alert on a sustained increase instead. See Audit Log.

Management-group confinement config migrates to Postgres (ADR-0042)

The ManagementGroupStore — the confinement hierarchy that backs operator scoping and the ADR-0017 authorize_list_read gate — moves from the SQLite management-groups.db file to the server’s PostgreSQL substrate in this release (ADR-0006 Wave 2.2), schema management_group_store. It reuses the existing shared connection pool, so no new connection flag or config is required — the same --postgres-dsn / YUZU_POSTGRES_DSN that every other server store uses.

No legacy-SQLite migration path. No production fleet ever ran a pre-Postgres build of this store, so there was no real management-groups.db data to carry over — the mandatory backfill this section originally described was retired under ADR-0009’s fresh-start-by-default amendment (see ADR-0042’s Update). These are two SEPARATE behaviors, not one:

If you see the legacy-row-count warning and the environment genuinely has real management groups to keep, there is no automated recovery path: re-author the equivalent groups/memberships/role assignments against the new Postgres-backed store via the REST API.

Rollback caution: see the InventoryStore section’s “Rollback caution” note above — the same unconditional rollback-refusal applies to this store (its own dropped marker table).

Operator-visible behaviour change (fail-closed reads). After cutover, a confinement-feeding read that degrades (store not open, pool-acquire timeout, or query error) now returns a distinguishable failure that the caller resolves to DenyAll — a scoped operator sees a reduced or empty list rather than the store silently returning an empty deny-set and under-restricting (the old fail-open hazard). A re-parent (PUT /api/v1/management-groups/{id}) whose hierarchy reads degrade now returns 503 rather than a misleading success. Watch the new yuzu_server_mgmt_group_read_degrade_total{reason} counter — a non-zero rate means scoped operators are seeing reduced/empty lists because the confinement substrate is degraded, not that groups actually shrank (see docs/user-manual/metrics.md § “Management group metrics” and the shipped YuzuMgmtGroupReadDegraded alert).

Not affected: the confinement hierarchy’s semantics, the REST/MCP surface, and dynamic-group scope expressions are unchanged — only the storage substrate and the fail-closed read posture change.

Custom properties migrate to Postgres (ADR-0045)

The CustomPropertiesStore — operator-authored per-agent metadata (properties and their optional type/validation schemas) used in scope expressions via props.<key> — moves from the SQLite custom-properties.db file to the server’s PostgreSQL substrate in this release (ADR-0006 Wave 2), schema custom_properties_store. It reuses the existing shared connection pool — no new connection flag or config is required.

No legacy-SQLite migration path. No production fleet ever ran a pre-Postgres build of this store, so there was no real custom-properties.db data to carry over — the one-time mandatory backfill this section originally described was retired under ADR-0009’s fresh-start-by-default amendment (see ADR-0045’s Update).

These are two SEPARATE failure/detection behaviors, not one — do not conflate them:

Operator-visible behaviour change (fail-closed reads). After cutover, a props.<key>-feeding read that degrades (store not open, pool-acquire timeout, or query error) now aborts the whole scope evaluation rather than silently resolving the property as absent — a props.<key>-scoped dispatch/policy/push rule now matches nobody while degraded, instead of (under a NOT/!= scope) silently matching everybody. GET /api/agents/:id/properties now returns 503 on a degraded read rather than an empty list. Watch the new yuzu_server_custom_properties_read_degrade_total{reason} counter — a non-zero rate means props.<key>-scoped rules may be silently matching nobody, not that operators removed the properties (see docs/user-manual/ metrics.md and the shipped YuzuCustomPropertiesReadDegraded alert).

Operator-visible behaviour change (fail-closed writes, 2026-08-14 follow-up). PUT /api/agents/:id/properties/:key and POST /api/property-schemas now return 503 (instead of 400) when the failure is a genuine database/store outage rather than caller-input or schema-validation error — previously every failure from either write, including a transient Postgres blip, surfaced as the same 400 a caller could not distinguish from their own bad input. A caller that branches specifically on 400 to mean “don’t retry” should treat the new 503s the same as any other transient server error (retry with backoff); a caller that already treats any 5xx as retryable is unaffected. See docs/user-manual/rest-api.md’s per-route notes for the exact response shapes.

Not affected: the props.<key> scope-DSL syntax and property/schema semantics are unchanged; the GET/DELETE property routes and GET schema route keep their prior response shapes — only the two write routes’ failure-mode status codes changed, as described above.

Network-discovered device data migrates to Postgres (DiscoveryStore, ADR-0044)

The DiscoveryStore — the network-discovered devices behind POST /api/discovery/scan and GET /api/discovery/results — moves from the SQLite discovery.db file to the server’s PostgreSQL substrate in this release (ADR-0006 Wave 2), schema discovery_store. It reuses the existing shared connection pool — no new connection flag or config is required.

No legacy-SQLite migration path. No production fleet ever ran a pre-Postgres build of this store, so there was no real discovery.db data to carry over — the one-time mandatory backfill this section originally described was retired under ADR-0009’s fresh-start-by-default amendment (see ADR-0044’s Update).

These are two SEPARATE failure/detection behaviors, not one — do not conflate them:

Operator-visible behaviour change (fail-closed reads). GET /api/discovery/results now returns 503 on a degraded read (store not open, pool-acquire timeout, or query error) instead of silently rendering an empty device list — previously, a local SQLite read essentially never failed short of file corruption, so this failure mode was not practically reachable. Watch the yuzu_server_discovery_read_degrade_total{reason} counter — a non-zero rate means the discovery view is degraded, not that no devices were found.

Breaking — POST /api/discovery/scan’s response contract changed. The endpoint no longer always returns 200 {"status":"ok",...}: the response gains a devices_failed count, status is "partial" when some but not all devices in a batch persisted, and the endpoint returns 503 when every attempted device failed to persist (previously this was silently reported as 200/"ok", with the discovery.scan audit row always saying "success" regardless of outcome — the audit outcome is now "success"/"partial"/"failure"). A caller that asserts a bare status == "ok", or that treats any 5xx from this endpoint as a hard failure needing operator escalation, should account for the new value and status code. Re-sending the exact same request body is safe — upsert_device is idempotent per ip_address, so a byte-identical retry cannot double-count or corrupt already-stored devices — but a fresh re-scan is not the same thing: mac_address/subnet overwrite unconditionally on every upsert, so a re-scan that fails to resolve a device’s MAC this time will blank a previously known-good value rather than simply retry the earlier failure. See the REST API reference’s Network Discovery section for the full response shapes.

Guardian quarantine records migrate to Postgres (QuarantineStore, ADR-0047)

The QuarantineStore — the Guardian device-quarantine bookkeeping behind POST /api/v1/quarantine, DELETE /api/v1/quarantine/{agent_id}, and the MCP quarantine_device tool — moves from the SQLite quarantine.db file to the server’s PostgreSQL substrate in this release (ADR-0006 Wave 2), schema quarantine_store. It reuses the existing shared connection pool — no new connection flag or config is required.

No legacy-SQLite migration path. No production fleet ever ran a pre-Postgres build of this store, so there was no real quarantine.db data to carry over — the mandatory, fingerprint-verified backfill this section originally described (all of the pre-upgrade checks, refusal conditions, and move-aside/fingerprint mechanics) was retired under ADR-0009’s fresh-start-by-default amendment (see ADR-0047’s Update). These are two SEPARATE behaviors, not one:

This is a stronger warning than it looks, and a deliberate choice despite the stakes. Unlike most other stores retiring their legacy-SQLite path in this same release, a genuinely non-empty quarantine.db here means a device an operator quarantined is not re-quarantined in the server’s view after upgrade — the server’s view of containment does not include whatever the warning names, though agent-side firewall enforcement (out of scope for this store) may still be independently in effect. This was reopened and explicitly reconsidered for this store given the containment stakes, not defaulted by uniformity with the rest of this retirement batch — see ADR-0047’s Update for the full reasoning. If you see the legacy-row-count warning and the environment genuinely has real quarantine records to keep, there is no automated recovery path: re-quarantine the affected device(s) via POST /api/v1/quarantine after confirming which devices need it.

Rollback caution: see the InventoryStore section’s “Rollback caution” note above — the same unconditional rollback-refusal applies to this store (its own dropped marker table).

Operator-visible behaviour change (fail-closed reads). GET /api/v1/quarantine can now return 503 for two DISTINCT reasons — the response body’s message distinguishes them, since only one is covered by the existing metric:

Verify: after the server reports ready, GET /api/v1/quarantine (quarantine has no dashboard surface — REST/MCP only) shows the expected active quarantine records — either already-live Postgres data on a redeploy, or an empty list plus the legacy-row-count warning above on a fresh install with a real legacy file.

Not affected: the agent-side quarantine firewall enforcement (WFP/nftables/pf block-all + exceptions) is untouched by this migration — only the server-side bookkeeping’s storage substrate changes. POST /api/v1/quarantine and DELETE /api/v1/quarantine/{agent_id}’s request/response shapes, and the MCP quarantine_device tool’s ticket-then-recall approval flow, are unchanged.

Device tags migrate to Postgres (TagStore, ADR-0050)

The TagStore — device tags behind GET/PUT/DELETE /api/v1/tags, the legacy /api/tags* routes, the MCP get_tags/set_tag/delete_tag/search_agents_by_tag tools, and every tag:<key> scope expression — moves from the SQLite tags.db file to the server’s PostgreSQL substrate in this release (ADR-0006 Wave 2 batch 3), schema tag_store, on the existing shared pool. Tags are dispatch-critical: scope expressions decide which agents a command reaches, and service-scoped API tokens are confined by the service tag — which is why the read/write failure modes below fail closed rather than degrading silently.

No legacy-SQLite migration path. No production fleet ever ran a pre-Postgres build of this store, so there was no real tags.db data to carry over — the one-time mandatory backfill this section originally described was retired under ADR-0009’s fresh-start-by-default amendment (see ADR-0050’s Update).

These are two SEPARATE failure/detection behaviors, not one — do not conflate them:

Operator-visible behaviour changes (fail-closed reads/writes).

Verify: after the server reports ready, GET /api/v1/tags?agent_id=<id> (or the device page) shows the expected tags (re-applied via the REST/MCP surface on a fresh install, or already-live Postgres data on a redeploy).

⚠️ Behaviour change: quarantine containment now covers IPv6, and reports honestly (#3282, #3283, #3284, #3285, #3286, #3260)

Six issues close against the quarantine agent plugin across Windows, Linux and macOS. Every one is the same class: an outcome reported as success when it was partial, failed, or unknown. Read §1 before quarantining a dual-stack Linux host — it needs operator action.

The server-side dispatch enforcement (#881) and the MCP tool’s honesty fixes (#3127) ship separately and have their own upgrade note.

1. Linux containment now covers IPv6 (#3282)

The Linux leg installed its yuzu-quarantine chain in iptables only. On a dual-stack or IPv6-capable host the device stayed fully reachable over IPv6 while reporting a clean status|quarantined — containment that was not containing. The plugin now mirrors the chain into ip6tables, and a failure on either family is reported (status|quarantined_partial), never counted as applied.

Action required for dual-stack fleets. Whitelist entries are routed to the chain matching their own family, so a v4-only whitelist now leaves IPv6 contained. There is no blanket “keep existing connections alive” rule on either family — only a whitelisted address survives, in any connection state. The agent automatically whitelists its own configured server address (an IP literal directly, a hostname resolved to its address(es) once, at agent startup — never at quarantine time, and using the endpoint’s own resolver), so most fleets need no manual action here. Prefer an IP-literal --server config for endpoints where containment integrity matters most, since that removes DNS from the equation entirely. If your agents reach the server over IPv6 through a path DNS resolution of the configured address wouldn’t reproduce (split-horizon DNS, a manually pinned route), or if the server’s address changes after an agent was last started, add the server’s address to the whitelist explicitly. See Security Hardening.

2. Reporting is more conservative again (#3283, #3285, #3286, #3260)

As with the Wave-2 argv migration before it, nothing got more broken — the reporting got more honest, so expect these to fire more often after upgrading:

No migration, no schema change

Rollback is data-safe. A mixed fleet is safe: an older agent simply keeps the older, less honest plugin reporting until it is upgraded.

Product packs migrate to Postgres (ProductPackStore, ADR-0054)

The ProductPackStore — operator-installed product packs behind POST/GET/DELETE /api/product-packs* — moves from the SQLite product-packs.db file to the server’s PostgreSQL substrate in this release (ADR-0006), schema product_pack_store, on the existing shared pool.

No legacy-SQLite migration path. No production fleet ever ran a pre-Postgres build of this store, so there was no real product-packs.db data to carry over — the one-time mandatory backfill this section originally described was retired under ADR-0009’s fresh-start-by-default amendment (see ADR-0054’s Update).

These are two SEPARATE failure/detection behaviors, not one — do not conflate them:

Operator-visible behaviour changes.

Verify: after the server reports ready, GET /api/product-packs shows the expected packs (re-installed via the REST API on a fresh install, or already-live Postgres data on a redeploy).

Guardian Baselines migrate to Postgres (BaselineStore, ADR-0055)

BaselineStore — Guardian’s deployable Baseline unit (/guardian → Baselines, GET /api/v1/guaranteed-state/device-compliance) — moves from the SQLite guardian-baselines.db file to the server’s PostgreSQL substrate in this release, schema baseline_store, on the existing shared pool. A Baseline is the only deployable unit in Guardian: what a deploy enforces across the fleet is read from each Baseline’s deployed_snapshot, never its live member set.

No legacy-SQLite migration path. No production fleet ever ran a pre-Postgres build of this store, so there was no real guardian-baselines.db data to carry over — the one-time backfill mechanism this section originally described was retired under ADR-0009’s fresh-start-by-default amendment (see ADR-0055’s Update).

These are two SEPARATE failure/detection behaviors, not one — do not conflate them:

Operator-visible behaviour changes (fail-closed reads). A degraded read on the enforcement-feeding path (deployed_member_rule_ids() — the source for the push fan-out, the heartbeat reconcile, and the per-device compliance view) now returns a distinguishable failure that the caller resolves to an explicit abort/503, never a silent empty/“fully compliant” enforced set. GET /api/v1/guaranteed-state/device-compliance returns 503 rather than a misleadingly-empty or false-compliant result when the underlying read degrades; Guardian deploy/delete dashboard actions show a degraded-modal rather than reporting success. There is no dedicated backfill-outcome metric for this store — a schema-migration failure is boot-fatal and never serves /metrics, so the boot log and /readyz are the channels to watch during an upgrade.

Verify: after the server reports ready, /guardian shows the expected Baselines (re-authored via the Guardian UI/REST API on a fresh install, or already-live Postgres data on a redeploy).

Guardian rules, events, and DEX observations retire their legacy-SQLite path (GuaranteedStateStore, ADR-0038)

GuaranteedStateStore — the live agent-side policy enforcement store (/guardian rule authoring, drift/remediation events, DEX observations) — no longer copies a legacy guaranteed-state.db file into Postgres on first boot.

No legacy-SQLite migration path. No production fleet ever ran a pre-Postgres build of this store, so there was no real guaranteed-state.db data to carry over — the one-time backfill mechanism this store originally shipped was retired under ADR-0009’s fresh-start-by-default amendment (see ADR-0038’s Update).

These are two SEPARATE failure/detection behaviors, not one — do not conflate them:

This is a stronger warning than it looks. Unlike most other stores retiring their legacy-SQLite path in this same release, a genuinely non-empty guaranteed-state.db here means real Guardian rules an operator authored are silently NOT loaded — enforcement resumes from an empty rule set, not the operator’s prior policy. (Guardian Baselines carry the same consequence — see the BaselineStore section above.) If you see the legacy-row-count warning above and the environment genuinely has real rules to keep, there is no automated recovery path: re-author the equivalent rules against the new Postgres-backed store via the Guardian UI/REST API before relying on it, and confirm the expected rules are actually present (see Verify below) rather than assuming they carried over.

Verify: after the server reports ready, /guardian shows the expected rules (re-authored via the Guardian UI/REST API on a fresh install, or already-live Postgres data on a redeploy) — do not rely on the absence of the legacy-row-count warning alone as confirmation that nothing was lost.

Compliance policy engine moves to Postgres (PolicyStore, ADR-0056)

The PolicyStore — compliance policy fragments/policies behind POST/DELETE /api/policy-fragments*, POST/DELETE /api/policies*, POST /api/policies/{id}/{enable, disable,invalidate,evaluate,remediate}, GET /api/compliance*, and every dispatched compliance check/fix/verify — runs on the server’s PostgreSQL substrate, schema policy_store, on the existing shared pool. Six operator-authored tables (fragments, policies, and their input/trigger/group associations, plus per-agent policy_status) plus a new operational-only seventh table, policy_dispatch_state, which coordinates fleet-wide compliance-check dispatch across replicas.

No legacy-SQLite migration path. No production fleet ever ran a pre-Postgres build of this store, so there was no real policies.db data to carry over — the one-time backfill mechanism this section originally described was retired shortly after this store’s Postgres migration merged, under ADR-0009’s fresh-start-by-default amendment.

These are two SEPARATE failure/detection behaviors, not one — do not conflate them:

Operator-visible behaviour changes (fail-closed reads/writes).

Verify: after the server reports ready, GET /api/compliance returns the expected policies and fleet compliance percentage.

Runtime configuration migrates to Postgres — OIDC client secret now encrypted at rest, config resets on cutover (RuntimeConfigStore, ADR-0060)

RuntimeConfigStore — the Settings-configurable overrides behind GET/PUT /api/config/:key (retention windows, log_level, DEX alert-routing knobs, and the oidc_* OIDC parameters) — moves from the SQLite runtime-config.db file to the server’s PostgreSQL substrate in this release, schema runtime_config_store. WebhookStore and InstructionStore have both since merged; OffloadTargetStore is now the only server store remaining on the Postgres migration ladder.

Before you upgrade, if you use OIDC/SSO configured through Settings (not --oidc-client-secret/the environment): record your IdP client secret now. Yuzu cannot recover it for you — GET /api/config never returns a secret’s real value — and this cutover does not carry it forward (see below). Without it on hand, restoring SSO after this upgrade means generating a new client secret with your identity provider.

⚠️ Behaviour change: offload targets reset on Postgres cutover (ADR-0059)

OffloadTargetStore — the response-offload control plane behind /api/v1/offload-targets — moves from the SQLite offload_targets.db file to the server’s PostgreSQL substrate in this release, schema offload_target_store, on the existing shared pool. Like ResponseStore/AnalyticsEventStore, this is a fresh-start cutover with no data migration (ADR-0009’s 2026-08-25 fresh-start-by-default amendment — no production fleet has ever run a pre-Postgres build of this store, so there is no legacy backfill to protect), so the legacy offload_targets.db is never read on upgrade.

Before you upgrade, if you have offload targets configured: the non-secret fields (name, url, auth_type, event_types, batch_size, enabled) are recoverable via GET /api/v1/offload-targets — list them now to speed up re-registration after the cutover. Credential values themselves were never GET-recoverable even before this migration (auth_credential has always been redacted from every response), so there is nothing new to lose there — only the non-secret configuration is worth capturing.

What happens on first PG boot:

Also in this release: auth_credential is now encrypted at rest app-side (ADR-0010) rather than a plaintext column; GET/list() responses gain a has_credential boolean alongside the already-redacted auth_credential. A delivery whose target’s credential fails to decrypt (KEK unavailable, a tampered/corrupted stored blob) is skipped entirely and logged with error=credential_unavailable — it is never fired unsigned. A degraded read on GET /api/v1/offload-targets or GET /api/v1/offload-targets/{id} now returns 503, distinguishable from a genuine “no targets configured” (empty list) or “no such id” (404).

Rolling back: because the legacy offload_targets.db is never touched by the Postgres-backed binary (not read, not moved, not deleted), reverting to a pre-cutover binary restores whatever was in that file before the cutover, exactly as it was. Any target registered through POST /api/v1/offload-targets on the Postgres-backed binary lives only in Postgres and is lost on rollback (re-register it after rolling forward again).

⚠️ Behaviour change: OTA package catalog resets on Postgres cutover (UpdateRegistry, ADR-0061)

UpdateRegistry — the OTA agent-update catalog behind the gRPC CheckForUpdate/ DownloadUpdate handlers and the Settings → Updates admin page — moves from the SQLite update_packages.db file to the server’s PostgreSQL substrate in this release, schema update_registry, on the existing shared pool. Like OffloadTargetStore/ResponseStore, this is a fresh-start cutover with no data migration (ADR-0009’s 2026-08-25 fresh-start-by-default amendment), so the legacy update_packages.db is never read for data migration/backfill on upgrade — the server does open it read-only once, at boot, to count rows and warn if it finds any (see “What happens on first PG boot” below), but never to carry any of them over.

Before you upgrade, if you have OTA packages configured: list them via the Settings → Updates page first so you know what to re-upload. Package binaries already on disk under your configured update_dir are untouched by this cutover — only the metadata row (sha256, rollout percentage, mandatory flag) is lost — so re-uploading the same file after upgrading reproduces the identical sha256/size.

What happens on first PG boot (OTA is on by default — see the note below if you’ve passed --no-ota):

Also in this release: construction is now fail-closed for OTA (on by default; disable with --no-ota, not an opt-in flag) — a reachable database whose update_registry schema fails to migrate/open now refuses to start the server, rather than silently serving with OTA dead (the pre-migration code had no such check). ca_store/scim_store’s own fail-closed construction is already unconditional on every deployment (only their probe checks are gated, on whether you’re using default certs / have SCIM enabled) — UpdateRegistry is simply the first of the three where that same construction-time posture applies with no opt-out flag set by default. Plan accordingly if you’re upgrading a fleet you haven’t audited for Postgres reachability — this now affects the ordinary default deployment, not an opt-in minority. UpdateRegistry now appears in /readyz’s check list and in /health’s stores object (previously absent from both). On /readyz, a deployment running with --no-ota sees no change — the check is vacuously true and never enters the failed list. On /health//api/health, a new stores.update_registry: "ok"/"error" key appears in the JSON body for every deployment, --no-ota or not.

Rolling back: because the legacy update_packages.db is never written by the Postgres-backed binary (opened read-only at most once, at boot, for the row-count warning above — never moved, never deleted, never modified), reverting to a pre-cutover binary restores whatever was in that file before the cutover, exactly as it was. Any package uploaded through the Postgres-backed binary lives only in Postgres and is lost on rollback (re-upload it after rolling forward again) — but its binary on disk under update_dir survives regardless, since the cutover never touches that directory.

⚠️ Behaviour change: quarantine is now enforced at instruction dispatch (#881, #3127)

Quarantine previously isolated a device’s network without stopping the control plane from sending it commands. This release closes that, and separately removes two paths on which the MCP quarantine_device tool claimed an isolation it had not achieved. Read this before upgrading a deployment that quarantines devices — the alert-rule item needs operator action.

The agent-side plugin containment fixes (IPv6 coverage, and honest reporting on all three platforms) ship separately and have their own upgrade note.

A quarantined device is now refused at dispatch (#881)

Quarantine previously isolated a device’s network without stopping the control plane from sending it commands: the feature was enforced at the endpoint and nowhere else. Instruction dispatch now consults containment state at the server’s single dispatch chokepoint, and a quarantined device is dropped from every targeting arm — by id, by scope, by management group, and by broadcast — before the command reaches the agent.

Fail-closed under a degraded store. If containment state cannot be read, the server serves a last-known-good snapshot for up to 60 seconds; past that, or when the store is durably unavailable, dispatch fails closed and refuses every target fleet-wide rather than guess who is contained. A background refresher re-reads containment state every 20 seconds, so the snapshot is warm even on a server with no operator traffic — without it the budget would only have absorbed anything on a busy server, which is not when you need it.

The snapshot window cuts both ways, and the other direction is the quieter one. While a stale snapshot is being served, a device quarantined after that snapshot was taken is absent from it — so dispatch to that device is permitted, and audited as an ordinary success. For up to the snapshot’s remaining budget, containment does not hold for a freshly-quarantined device. The refresher bounds that window to roughly the refresh interval in the common case, and YuzuQuarantineGateServingStaleState alerts on it; if you quarantine a device while yuzu_server_quarantine_read_degrade_total is non-zero, confirm with quarantine.status rather than assuming the dispatch gate is already holding.

There is no operator override. The 60-second budget is a fixed bound, not a tunable, and there is no flag to disable enforcement — a knob that let an operator turn the gate off under pressure would be the first thing reached for during exactly the incident it exists for. What survives a store outage regardless: the quarantine plugin’s own four actions (they never read the store), and the three server-internal pushes below. So containment, release and whitelist repair all remain possible while the store is down; what stops is other operator dispatch. Alert on yuzu_server_quarantine_gate_total{outcome="fail_closed"} — it means instruction dispatch has stopped, and it is an outage signal rather than a quarantine one. All four outcome values are pre-seeded at boot, so a zero is a measurement rather than an absent series — you can alert on fail_closed before it has ever fired. Absence of the family then means the server is not exporting it at all (an older build, or a scrape that is failing), not that the gate is idle: a booted server publishes all four from boot whether or not any dispatch has happened yet.

Alert-rule change (action may be required). YuzuDispatchTargetRejected now excludes reason="quarantined". Without that exclusion, correct enforcement fires the alert: any looping automation against one contained host clears its >3/15m threshold, and a fail-closed episode increments by the whole fleet at once. If you maintain a fork of docs/prometheus/yuzu-alerts.yml, apply the same exclusion — otherwise the rule gets silenced and the genuine #2500 near-miss signal goes with it.

MCP quarantine_device no longer reports phantom isolation (#3127)

Two independent bugs made the tool claim containment it had not achieved. Both are fixed, and both are wire-visible to an MCP client:

dispatch_confirmed:true means the plugin registry accepted the frame, never that the device is provably isolated (for a gateway-attached agent the frame is only queued). Confirming containment still requires a follow-up status read returning state|active.

Note the REST twin has diverged on one case. POST /api/v1/quarantine records only — it has never dispatched — so it still answers 400 on an already-quarantined device: there is nothing for it to re-drive. The 400-vs-503 split is otherwise unchanged.

No migration, no schema change

Rollback is data-safe. A mixed fleet is safe: the dispatch gate is server-side and applies regardless of agent version.

⚠️ Behaviour change: raw dispatch now enforces InstructionDefinition approval governance (#1398)

An InstructionDefinition’s declared approval.mode (role-gated/always) was previously enforced only on the governed POST /api/instructions/:id/execute path — raw dispatch (POST /api/command, MCP execute_instruction/execute_bundle/quarantine_device, the dashboard, workflow, schedules, and /auto Deploy) never resolved a definition, so a caller who held plain RBAC permission on an action could bypass its declared review requirement entirely by dispatching directly instead of through the governed path. This release closes that at the single shared dispatch chokepoint. Read this before upgrading if any automation, integration, or scripted workflow dispatches commands as a non-admin principal — an operation that previously succeeded may now be refused.

What changed

Every dispatchable plugin.action pair now carries a compiled ExecuteGate (None/AdminOrApproval/AlwaysApproval), derived strictest-wins from every shipped InstructionDefinition targeting that pair. 42 pairs are gated as of this release — including every script_exec.* action, filesystem.delete_lines/.write_content/.replace, registry.set_value/.delete_key/.delete_value, interaction.*, network_actions.*, wol.*, quarantine.quarantine/.unquarantine/.whitelist, rdp_control.set_state, certificates.delete, storage.*, tags.clear, discovery.scan_subnet, chargen.chargen_start, http_client.download, installed_apps.list_per_user, services.set_start_mode, echo.configure, and content_dist.* — the complete, authoritative list is tests/fixtures/1398_pair_gate_table.json (search for "AdminOrApproval"/"AlwaysApproval", excluding the server/server_internal/_server-prefixed rows, which are server-internal and never agent-dispatched).

A gated pair now denies a non-admin caller with no covering approval, regardless of which surface dispatched it:

How to check your exposure before upgrading

Query the audit log for command.dispatch rows (or your own automation’s recent activity) whose plugin/action appears in the gated list above, dispatched by a non-admin principal. If you find any, either grant the dispatching principal an admin role, or migrate that automation to dispatch through the governed path (POST /api/instructions/:id/execute, or an MCP supervised-tier token) so it goes through the approval workflow instead of raw dispatch.

Schedule approval tickets are now bound to their reviewed content

A separate hardening in this same release: a schedule’s approval ticket is now bound to the specific plugin/action it was approved for, not just the definition’s id. If a definition’s plugin/action is edited (PUT /api/instructions/{id}) after a schedule’s ticket for it was approved but before the schedule’s next fire, the stale ticket no longer redeems — the schedule correctly requests a fresh ticket for the new content instead of firing under review that was never given for it. This is self-healing (no manual database intervention needed): approve the new ticket the same way you approved the original. You would only notice this if you edit a scheduled definition’s target action between approval and fire, which is unusual but not unsupported.

Migration

ApprovalManager’s SQLite schema gains two additive columns (target_plugin, target_action, migration v8) — no backfill, no data loss, safe on a large existing approvals table (constant default, metadata-only ALTER TABLE). Rollback is safe: an older binary ignores the two new columns on read and simply doesn’t populate them on write, which the new binary’s fail-closed matching correctly treats as “must re-request” rather than a corruption state.

No config flag, no override

There is no flag to disable the new gate. An admin caller, or a caller dispatching via the governed path with a redeemed approval ticket, is not subject to it.

Behaviour change: a rule_id outside the documented charset is now rejected at creation (#4665)

POST /api/v1/guaranteed-state/rules and MCP create_guardian_rule now enforce the documented rule_id charset ([A-Za-z0-9._-]+, at most 256 bytes) at creation — a request whose rule_id violates it now gets 400 (REST) or an error (MCP), audited as guaranteed_state.rule.create denied, instead of being accepted on any non-empty string. This closes an operator-authored log-injection exposure (#4665): log lines across the Guardian/Spark subsystem now also neutralise a rule id or Spark key before printing it, but a rule_id that never conformed to the documented shape (a space, a newline, or any other disallowed byte) could previously still be created.

Who this affects: almost nobody — the charset was already documented before this release, just not enforced, so create requests generated by the dashboard, the REST/MCP tooling, or any client following the documented contract are unaffected. This only changes behaviour for a caller that was relying on the previously-permissive accept-anything-non-empty behaviour to create a rule_id outside [A-Za-z0-9._-]+ or longer than 256 bytes.

What to do — this is a hard cutover, not a migration, by deliberate decision: enforcement at the server is create-only, so an existing rule whose rule_id predates this release and doesn’t conform to the charset is never torn down by the creation check itself. But the server-side push builder now excludes any such row from every push it builds (logged at error with a sampled rate limit as Guardian push: rule ... has a rule_id that fails the .../256-byte charset check (#4665) — excluding it from this push, and counted in yuzu_guardian_push_rule_excluded_total{reason="invalid_rule_id"}), the same way an over-depth spec_json is already excluded — and the agent’s full_sync path unconditionally tears down and rebuilds its ENTIRE active rule set from whatever the push actually contains. Put together: the first full_sync an upgraded agent receives permanently disarms every rule with a non-conforming rule_id (logged at warn on the agent, matching full_sync is disarming N rule(s) with a rule_id outside the [A-Za-z0-9._-]+/256-byte charset (#4665) -- hard cutover, not preserved). This is not silent — the server-side exclusion is both logged AND metered (yuzu_guardian_push_rule_excluded_total); the agent-side disarm is logged only, with no metric of its own yet — but it IS a real, one-time loss of a previously-enforcing control if you don’t act first. Every OTHER rule in scope is unaffected either way — this only ever touches rows that were already unenforceable by the documented contract.

Finding a non-conforming legacy rule_id — do this BEFORE upgrading, not after: GET /api/v1/guaranteed-state/rules returns every rule regardless of its rule_id’s shape; filter the response for any rule_id that doesn’t match ^[A-Za-z0-9._-]{1,256}$ — that’s the same predicate the server now enforces at creation. For each one you find, either delete it (see below) and recreate it under a conforming rule_id, or accept the loss consciously — there is no third option that preserves it under its existing id.

Removing a non-conforming rule_id: the two delete surfaces have DIFFERENT restrictions, not a simple REST-restricted/MCP-unrestricted split. REST DELETE /guaranteed-state/rules/{rule_id} routes on the same charset regex the create path now enforces ([A-Za-z0-9._-]+, no length bound), so it can’t reach a rule_id containing a byte outside that charset, but it has no length cap of its own and so can reach one that’s charset-clean but over 256 bytes (a shape only possible pre-#4665, when creation had no length bound either). MCP delete_guardian_rule’s own input schema caps at maxLength: 256 with no charset pattern, so it has the exact opposite gap: it can reach a charset-violating id (any length up to 256), but can’t reach one that’s charset-clean and over 256 bytes — the request is rejected by schema validation before the handler’s own logic ever runs. A rule_id that violates BOTH (wrong charset AND over-length) is reachable by neither REST nor MCP; fall back to a direct database delete (guaranteed_state_store.guaranteed_state_rules) for that case.

Behaviour change: preview_scope_targets / scope/preview now resolve from_result_set:/props. atoms correctly (#4981)

POST /api/v1/scope/preview and MCP preview_scope_targets previously evaluated a scope expression against a bespoke attribute resolver that only understood ostype/arch/hostname/agent_version/tag:<key> — a from_result_set:<id> or props.<key> atom silently resolved to unset, so the atom’s comparison was always false, and NOT from_result_set:<id> inverted that to match every agent the caller could see regardless of the referenced set’s real membership. Both surfaces now route through the same fail-closed evaluation ladder a real dispatch uses.

Who this affects: anyone who has called either surface with an expression containing from_result_set: or props.. If you relied on the old (incorrect) match set for either atom kind, re-check any automation built on that response before upgrading — the direction of the correction depends on how the atom was used: a negated NOT from_result_set:<id> was, in practice, matching your whole visible fleet, and the corrected match set will generally be narrower and more accurate to what a real dispatch of the same expression would actually target; a plain (non-negated) from_result_set:/props.<key> atom always evaluated false under the old resolver, so its match set was previously stuck at 0 and the corrected set will generally be broader, now actually populated with real members.

New error responses a strict client should handle:

A preview call no longer extends a referenced result set’s TTL as a side effect — it is now a genuine read-only dry run, matching its documented readOnlyHint: true.

Fresh deployment only for 0.15.0. See the banner at the top of this page: the steps below are not a supported upgrade path from an earlier build’s database.

Upgrade Order

Always upgrade in this order:

  1. Server — new server versions accept connections from older agents. A server-only restart while a gateway stays connected recovers without operator action once the gateway is also at this version; see What happens when the server restarts (an older gateway keeps the previous behaviour described there)
  2. Gateway — updated to match server protocol changes
  3. Agents — can be upgraded via OTA or manually, in batches

Never upgrade agents before the server — the server must understand the agent’s protocol version.

Fresh deployment only for 0.15.0. See the banner at the top of this page: the steps below are not a supported upgrade path from an earlier build’s database.

Pre-Upgrade Checklist

Before upgrading any component:

Fresh deployment only for 0.15.0. See the banner at the top of this page: the steps below are not a supported upgrade path from an earlier build’s database.

Upgrading the Server

Linux (systemd)

# 1. Stop the server
sudo systemctl stop yuzu-server

# 2. Back up data
sudo cp /var/lib/yuzu/*.db /var/lib/yuzu/backup/
sudo cp /etc/yuzu/*.cfg /var/lib/yuzu/backup/
# ... and the Postgres database, if provisioned. The DSN lives only in the
# root-only env file (NOT in interactive shells) — load it, and keep the
# password off the argv via PGPASSWORD. Full recipe + restore procedure:
# server-admin.md § PostgreSQL Substrate.
sudo sh -c '. /etc/yuzu/yuzu-server.env
  export PGPASSWORD="$(printf "%s\n" "$YUZU_POSTGRES_DSN" | sed -E "s!^[a-z]+://[^:/@]*:([^@]*)@.*\$!\1!")"
  pg_dump --format=custom --file=/var/lib/yuzu/backup/yuzu-pg.dump \
    "$(printf "%s\n" "$YUZU_POSTGRES_DSN" | sed -E "s!^([a-z]+://[^:/@]*):[^@]*@!\1@!")"'

# 3. Replace the binary
sudo cp yuzu-server /usr/local/bin/yuzu-server
sudo chmod +x /usr/local/bin/yuzu-server

# 4. Start the server (schema migrations run automatically)
sudo systemctl start yuzu-server

# 5. Verify
sudo systemctl status yuzu-server
curl -s http://localhost:8080/livez

Docker

Fresh deployment only for 0.15.0. The pull-and-restart upgrade and rollback steps below are historical and are not a supported path for an earlier build’s database or backups (see “Deployment contract (hard cutover)” above). The backup recipe under “Back up before upgrading” is current: it backs up this deployment; restore only a backup taken from this deployment, dump and keys archive together.

The reference deployment template lives at deploy/docker/docker-compose.reference.yml — copy it into your deployment directory next to a .env file, set YUZU_VERSION, and harden per the inline TLS checklist in the file header before exposing the stack to any untrusted network. The compose file declares two named volumes for the server that survive container replacement: server-data holds the remaining mutable state kept on disk — yuzu-server.cfg, auto-approve.cfg, the NVD cache SQLite database, and OTA binaries; certs holds /etc/yuzu/certs, the internal CA and the secrets KEK files that the Postgres data (on postgres-data) depends on. Enrollment tokens and pending agents are PostgreSQL-authoritative (HA WS-6 6.2, auth.enrollment_tokens / auth.pending_agents) — a pre-6.2 volume’s enrollment-tokens.cfg / pending-agents.cfg are imported once, automatically, at the first 6.2+ container’s boot, then renamed to <name>.cfg.imported.

An upgrade is a pull-and-restart:

# 1. Back up first (see below) — the old data is the only recovery path
#    if a migration fails on your specific DB.

# 2. Pick the new release tag (use an .env file or export)
export YUZU_VERSION=0.10.1

# 3. Pull the new image and recreate the container
docker compose -f docker-compose.reference.yml pull server
docker compose -f docker-compose.reference.yml up -d server

# 4. Verify the new version came up and schema migrations ran
docker compose -f docker-compose.reference.yml logs server | tail -40
docker compose -f docker-compose.reference.yml ps server    # should be "healthy"

Schema migrations execute automatically during the first up with the new image — look for MigrationRunner: <store> migrated to v<N> lines in the log (one per store on first upgrade, silent on subsequent restarts). The healthcheck used by depends_on: service_healthy probes /readyz, which returns 200 only after every store in the readiness conjunction has successfully migrated — so a healthy server container genuinely reflects migration success, not just liveness.

Back up before upgrading. Compose prefixes each volume name with the project name, which is the name of the directory holding the compose file unless you set one (-p, COMPOSE_PROJECT_NAME, or a top-level name:); from deploy/docker the volumes are docker_server-data and docker_certs. docker volume ls prints the full names. Set P to your project name, then run the block from a dedicated backup directory so $PWD is predictable. It checks that both volumes exist first, because docker run -v <name>:... silently creates an empty volume for a name that does not. Postgres is dumped first, then the keys: key files only accumulate, so a keys archive taken after the dump holds every key the dump references (the KEK pairing rule). pg_dump is consistent against a live database, so no stop is needed for it. The keys archive holds the CA private key, so it is written owner-only and handed to you; keep it outside any git checkout.

P=docker
mkdir -p ~/yuzu-backups && chmod 700 ~/yuzu-backups && cd ~/yuzu-backups
STAMP=$(date +%F)
docker volume inspect "${P}_server-data" "${P}_certs" > /dev/null \
  && docker exec yuzu-postgres pg_dump -U postgres --format=custom yuzu > "yuzu-pg-$STAMP.dump" \
  && docker run --rm -v "${P}_server-data":/data -v "$PWD":/backup alpine \
       tar czf "/backup/yuzu-data-$STAMP.tar.gz" -C /data . \
  && docker run --rm -v "${P}_certs":/certs -v "$PWD":/backup alpine \
       sh -c "umask 077; tar czf /backup/yuzu-keys-$STAMP.tar.gz -C / certs && chown $(id -u):$(id -g) /backup/yuzu-keys-$STAMP.tar.gz" \
  && echo "OK: backup written to $PWD"

Note: this recipe is a cold-ish backup — SQLite is running in WAL mode and a filesystem-level tar of a live database may capture a torn snapshot. For strong consistency, docker compose -f docker-compose.reference.yml stop server before backup (seconds of downtime) and start after. A fully hot backup via SQLite’s online-backup API is tracked in the roadmap. The pg_dump half has no such caveat — logical dumps are transactionally consistent by construction. Never back up Postgres by tar-ing the postgres-data volume while the database is running; a torn copy of pg_wal/ is unrecoverable, which is why the procedure above dumps through the database instead.

Restore-pairing (ADR-0010): since 0.14.0 the Postgres dump contains envelope-encrypted secrets (ciphertext + wrapped DEKs only) and is unusable without the matching KeyProvider keys directory, the certs volume above. Back up and restore the two as a pair: capture order and the full procedure are in the KEK pairing rule; see also Server Administration § PostgreSQL Substrate.

Fresh deployment only for 0.15.0. The steps below are historical and are not a supported upgrade, rollback or restore path for an earlier build’s database or backups (see “Deployment contract (hard cutover)” in Upgrading).

Rollback if a migration fails (Docker). Set P to your compose project name, as in the backup above (docker volume ls prints the full volume names):

# 1. Stop the new server (KEEPING the named volume — do NOT use -v)
docker compose -f docker-compose.reference.yml down server

# 2. Restore the previous backup over the existing volume
docker volume inspect "${P}_server-data" > /dev/null && \
docker run --rm -v "${P}_server-data":/data -v "$PWD":/backup alpine \
  sh -c 'rm -rf /data/* && tar xzf /backup/yuzu-data-YYYY-MM-DD.tar.gz -C /data'

# 2b. Restore the Postgres dump (postgres container still running)
docker exec -i yuzu-postgres pg_restore --clean --if-exists --no-owner \
  --role=yuzu -U postgres --dbname=yuzu < "yuzu-pg-YYYY-MM-DD.dump"

# 3. Pin the previous release
export YUZU_VERSION=0.9.0

# 4. Start the previous version
docker compose -f docker-compose.reference.yml up -d server

Never run docker compose down -v unless you intend to delete server-data, the keys volume (certs in the reference composes, server-certs in the others: the CA and the secrets KEK), postgres-data (the PostgreSQL substrate — the server’s primary data store), and every bit of server state. down alone is safe; the -v flag removes named volumes.

Windows

# 1. Stop the service (if running as service) or kill the process
Stop-Service yuzu-server  # or: taskkill /IM yuzu-server.exe /F

# 2. Back up data
Copy-Item C:\ProgramData\Yuzu\*.db C:\ProgramData\Yuzu\backup\
Copy-Item C:\ProgramData\Yuzu\*.cfg C:\ProgramData\Yuzu\backup\

# 3. Replace the binary
Copy-Item yuzu-server.exe "C:\Program Files\Yuzu\yuzu-server.exe"

# 4. Start
Start-Service yuzu-server  # or start manually

Upgrading the Gateway

Breaking: the gateway’s {grpcbox, ...} configuration moved. The listeners are now configured under {yuzu_gw, [{grpc_listeners, [...]}]} and the channel to the server under {yuzu_gw, [{upstream_client, ...}]} (the value of the old {grpcbox, [{client, ...}]}, unchanged). A gateway config that still has a {grpcbox, [{servers, ...}]} or a {grpcbox, [{client, ...}]} block refuses to start: the gateway no longer contains grpcbox, so nothing would read either block. The gateway image and its sys.config are an atomic pair: an old image with a new config starts, reports healthy and serves nothing. Read “Gateway listeners move to grpc_listeners” and “The upstream channel moves to upstream_client” under Upgrade notes by release below (key mapping, the one edit that covers both, pre-flight check, roll-forward and roll-back) before you replace the image or the config.

If the server was restarted while this gateway stayed connected, see What happens when the server restarts first. The server is upgraded first, so the server restart that ships this fix meets the OLD gateway, and agents behind it can read offline until the gateway is upgraded. Upgrade the gateway and restart it: this version’s gateway then recovers a later server restart by itself. A gateway restart disconnects every agent that gateway holds, so it is a fleet-wide reconnect for it, and released v0.13.0 and v0.14.0-rc6 agents with default settings stayed wedged after a graceful gateway restart in testing (v0.12.0 never re-registered; bug #2182, fixed by #5183, shipped in 0.15.0): upgrade the agents to a build that includes #5183 first, or expect to restart the agent service on those agents (see “Agent dependency” in the section linked above). Whether restarting only the gateway recovers agents stranded by an earlier server restart was not tested.

Linux (systemd)

Breaking (#659): the gateway refuses to start without a non-default Erlang distribution cookie. .deb/.rpm installs auto-generate /etc/yuzu/gateway.env; for tarball/manual installs create it once (see “Gateway distribution cookie now required” under Upgrade notes by release below) before the restart step.

Breaking (#4555): the cookie must additionally be 32+ characters (every documented generation path already clears this), and the gateway’s Erlang node short name changed from yuzu_gw1 to a shared yuzu_gw — update any external tooling hardcoding the old full node name. See “Gateway distribution-cookie length floor + shared node identity” below.

The release directory holds the gateway’s only configuration file, /opt/yuzu_gw/releases/<vsn>/sys.config, so replacing the directory deletes your config. Save it first, restore it after, pre-flight it, and only then start (a config from before this release must be converted first: see Upgrade notes by release below):

sudo systemctl stop yuzu-gateway
# 1. Save the operator-edited config; it lives inside the directory removed next
sudo cp /opt/yuzu_gw/releases/*/sys.config /root/yuzu-gw-sys.config.saved
# 2. Replace the release directory
sudo rm -rf /opt/yuzu_gw
sudo tar xzf yuzu-gateway-linux-x64.tar.gz -C /opt/
# 3. Restore the config (converted if the release needs it)
sudo cp /root/yuzu-gw-sys.config.saved /opt/yuzu_gw/releases/*/sys.config
# 4. Pre-flight the NEW release against that config, with the production YUZU_GW_*
#    variables exported (for example: set -a; . /etc/yuzu/gateway.env). Go on only at exit 0.
R=/opt/yuzu_gw; V=$(basename $R/releases/*/); C=$R/releases/$V/sys.config
$R/erts-*/bin/erl -noshell -noinput -boot $R/releases/$V/start_clean -config "$C" \
  -pa $R/lib/*/ebin -eval '
Load = [application:load(yuzu_gw)], yuzu_gw_app:apply_env_overrides(),
Results = Load ++ [yuzu_gw_app:check_legacy_grpcbox_servers(), yuzu_gw_app:check_legacy_grpcbox_client(),
                   yuzu_gw_app:check_grpc_listeners(), yuzu_gw_app:check_upstream_client_config(),
                   yuzu_gw_app:check_upstream_channel_mix(), yuzu_gw_app:check_upstream_tls_posture(),
                   yuzu_gw_app:check_mgmt_listener_posture()],
io:format("PREFLIGHT ~p~n", [Results]),
halt(case lists:all(fun(X) -> X =:= ok end, Results) of true -> 0; false -> 1 end).'
# 5. Start, then verify
sudo systemctl start yuzu-gateway
curl -s -o /dev/null -w '%{http_code}\n' localhost:8081/readyz     # 200
ss -ltn | grep -E ':50051|:50063'                                  # both listening

Step 4 is the pre-flight described under Upgrade notes by release. Skip it and a config the new release refuses shows up as a restart loop under systemd instead.

Packages (rpm -U, dpkg -i). A package upgrade replaces the operator-edited /opt/yuzu_gw/releases/<vsn>/sys.config with the stock config, which is the plaintext dev config (upstream http to 127.0.0.1:50055), and then restarts the gateway on it: the gateway boots, /readyz answers 200, and every registration fails. This is how the package has always behaved (the file is neither marked as configuration nor kept as a package conffile, so no .rpmsave is made); the root cause is tracked in the repository issue tracker. Until it is fixed, do the same by hand around the package step: copy releases/*/sys.config somewhere safe before rpm -U / dpkg -i, copy it back after, run the pre-flight above (step 4) and systemctl restart yuzu-gateway, then verify as in step 5. The release version directory changes between releases, so copy the file to the new releases/<vsn>/ directory. /etc/yuzu/gateway.env (the cookie) is not touched by the package.

Docker

docker pull ghcr.io/<owner>/yuzu-gateway:<version>
docker compose up -d yuzu-gateway

docker compose up -d is not a rolling update: it recreates every container of the service that changed in one step, so a config the new image refuses (it exits 1 at once and restarts in a loop) takes all of them down together. Mounted config and image change together or not at all. Update one gateway at a time, as a service of its own, with docker compose up -d --no-deps <gateway-service>, and check curl -s localhost:8081/readyz and the logs before the next.

Kubernetes / OpenShift

Nothing here has been run on a cluster; every point below is reasoned from the image and the code and is unverified: test it on the OpenShift target before relying on it.

Upgrading Agents

If the server has OTA updates enabled (--update-dir):

  1. Place the new agent binary in the server’s update directory
  2. Agents will check for updates on their configured interval (default: 6 hours)
  3. Monitor the fleet dashboard for version rollout progress

Manual

Follow the same stop/replace/start pattern as the server, per platform.

Batch Rollout

For large fleets, upgrade in stages:

  1. Upgrade a pilot group (5-10 agents) first
  2. Monitor for 24 hours
  3. Roll out to remaining agents in batches of 10-20%

Plugin Code Signing (vNEXT, #80)

A new operator-managed plugin trust bundle ships in this release. Default behaviour is unchanged: agents that don’t pass --plugin-trust-bundle and operators that don’t upload a bundle through Settings → Plugin Code Signing see identical behaviour to prior releases.

If you turn the feature on, three things change:

  1. New on-disk artifact at <cert-dir>/plugin-trust-bundle.pem. Linux/macOS: /etc/yuzu/certs/plugin-trust-bundle.pem; Windows: C:\ProgramData\Yuzu\certs\plugin-trust-bundle.pem. Add this path to your backup procedure alongside the SQLite databases. A backup that captures the DBs but not the cert dir restores plugin_signing_required=true (in runtime_config) without the bundle, and require-mode agents reject every plugin until the bundle is restored. The Docker reference docker-compose.reference.yml mounts server-data and certs (the cert dir); include both in the backup script. A compose of your own must keep the cert dir on a named volume too.

  2. Cert-dir filename collision check. The server now treats plugin-trust-bundle.pem in the cert dir as authoritative. The filename was unused in prior releases, but if any deployment placed an unrelated PEM at that exact path for another purpose, it will be interpreted as the plugin trust bundle on first read after upgrade. Run ls <cert-dir>/plugin-trust-bundle.pem on every server host before upgrading and rename any pre-existing file.

  3. DO NOT enable “Require signed plugins” yet. The Yuzu release pipeline does not currently sign the 44 in-tree plugins under agents/plugins/. Enabling Require with an operator-only trust bundle will reject every Yuzu-shipped plugin on next agent restart — fleet-wide outage. Use the transitional mode (bundle uploaded, Require off) until you have signed every plugin your fleet uses, including the in-tree ones. The Settings card displays this warning inline.

The new audit actions (plugin_signing.bundle.uploaded / .cleared / .require.changed) and metric labels on yuzu_agent_plugin_rejected_total (signature_missing / signature_invalid / signature_untrusted_chain) are documented in audit-log.md and metrics.md respectively. SIEM and alert rules already filtering on the existing success/failure/denied audit vocabulary pick these up unchanged — no new vocabulary tokens were introduced.

Schema Migrations

Starting with v0.10.0, every server-side SQLite store is wired through a single MigrationRunner that tracks schema version per store and applies pending migrations in a transaction. Prior releases relied on CREATE TABLE IF NOT EXISTS plus silent ALTER TABLE ADD COLUMN statements, which made rollbacks opaque and left no audit trail of what had been applied.

How it works:

Indexes are normally migration entries. audit_store’s retention index is a deliberate, single exception (idx_audit_ttl_id, #2360) - every other index in the server is created inside MigrationRunner, and new code should follow that rule, not this one. Two conditions justify the exception here and both must hold before it is copied: (a) a failed migration closes the store, and for audit_store that means every audit write then fails, taking the SOC 2 trail offline; and (b) the build is O(N log N) over an existing multi-million-row table, unlike the v1 indexes which are created on an empty one. A best-effort object may silently not exist, so nothing with a CORRECTNESS dependency may use this path - the retention guard is correct without its index, only slower.

Cost on a large existing audit_events: a one-time build at first boot after upgrade, measured at ~81 MB and ~1.8-3.3 s at 5M rows on NVMe. At 50M rows the build reads a ~16 GB table - tens of seconds on local NVMe, and potentially minutes on container overlayfs or network storage.

This runs synchronously, before the server binds its listeners, so nothing answers /livez or /readyz until it finishes. If your orchestrator’s kill-before-ready budget is shorter than the build you get a CRASH LOOP rather than a slow boot: CREATE INDEX is one atomic statement, so every killed attempt rolls back and the next boot re-pays it in full, indefinitely. Budget at least 5 minutes before the first liveness kill for an audit_events above ~20M rows on non-NVMe storage. The shipped deploy/docker/docker-compose.reference.yml sets start_period: 30s, which is fine for a fresh install and NOT enough for a large first post-upgrade boot - raise it before upgrading such a deployment. The elapsed time is logged when it exceeds a second, and subsequent boots are a no-op. If the build fails, retention still runs, but each pass then scans the table AND sorts the whole expired backlog for its ORDER BY ... LIMIT (measured 2.0 s versus ~285-315 ms for the same capped pass WITH the index; the range reflects different benchmark runs, not two different operations) - still far better than the unguarded code it replaced, but the failure is logged as an error.

Upgrading from v0.9.x or earlier is data-preserving: the first 0.10.x startup stamps every database at schema v1. A small set of stores (api_token_store, instruction_store, patch_manager, policy_store, product_pack_store, response_store) also runs a one-time legacy compatibility shim that re-applies the historical ALTER TABLE statements before stamping, so databases from very old releases that never received those columns still converge to the latest schema. These shims are kept in code for one release cycle and can be removed after v0.11.

No manual migration steps are required. Just replace the binary (or pull the new image and up -d) and start the server. Migration progress is logged at info level as:

[info] MigrationRunner: rbac_store migrated to v1
[info] MigrationRunner: response_store migrated to v1
...

Expect a log burst on first startup after upgrade. 30+ MigrationRunner: <store> migrated to v1 info lines appear on the first run against a pre-v0.10 database — one per store. On every subsequent restart the runner is silent at info level. If your log-shipping pipeline has per-second rate limits, widen them for the upgrade window or filter this single line pattern.

Verifying migration state after startup, query the per-store audit trail directly:

docker exec -i yuzu-server sqlite3 /var/lib/yuzu/rbac.db \
  "SELECT store, version, datetime(upgraded_at, 'unixepoch') FROM schema_meta ORDER BY upgraded_at;"

Every store that has ever run through the migration runner has a row here with its current version and the wall-clock timestamp of the last stamp. This is the operator-side audit trail for schema evolution.

The Postgres-backed stores are not in that file. Stores migrated onto the PostgreSQL substrate — audit_store among them (ADR-0040) — run through PgMigrationRunner, log PgMigrationRunner: <store> migrated to v<N>, and stamp public.schema_meta in PostgreSQL. Query those there instead:

psql "$YUZU_POSTGRES_DSN" -c \
  "SELECT store, version, to_timestamp(upgraded_at) FROM public.schema_meta ORDER BY upgraded_at;"

For audit_store specifically, a legacy audit.db — if one still exists — is never renamed, moved, or modified: Yuzu only opens it read-only, at boot, to warn if it holds rows (see the Audit trail section above). sqlite3 /var/lib/yuzu/audit.db still opens it normally on a migrated server; there is nothing to restore or account for here.

If a migration fails:

  1. Check the log for MigrationRunner: migration v<N> failed for <store>: <sqlite error> and note both the store name and the SQLite error.
  2. The server will have closed the failing store’s database handle, so /readyz returns 503 with the failed store name in the failed_stores body field — the probe accurately reflects degraded state. Don’t rely on /livez for readiness; it only checks process liveness, not schema integrity.
  3. Stop the server and restore the affected database file from backup — not the whole data directory. Restoring all databases to fix one broken store wipes in-flight approvals, pending agents, and enrollment tokens. This step does not apply to a Postgres-backed store such as audit_store: there is no per-store file to restore, so use the Postgres dump procedure under “Rollback if a migration fails” above.
  4. Start the previous server version against the restored data.
  5. Open an issue with the full error line, the source/target version numbers, and the output of the schema_meta query above.

Upgrade notes by release

vNEXT — Decommission securable replaces the SLE/Inventory/GuaranteedState conjunction (breaking for custom roles)

DELETE /api/v1/sle/agents/{id} (the whole-device erasure cascade — see REST API § Software Licensing (SLE)) used to gate on a three-securable conjunction: SoftwareLicensing:Delete and Inventory:Delete and GuaranteedState:Delete, all required together. A companion Wave 7 PR7.2 package adds a sixth per-agent store (app_usage) to the cascade (and #5682 a seventh, offline_endpoint, for seven in total), and rather than grow the conjunction to a fourth securable, ADR-0024 Decision 9 was amended ahead of that landing to promote a single dedicated Decommission securable — the route now gates on Decommission:Delete alone, replacing the old conjunction outright (not adding to it).

Seeded roles are auto-preserved — the same INSERT OR IGNORE role-default seeding as the SLE securable above runs on every boot, so Administrator (full CRUD via the generic per-securable loop) and ITServiceOwner (a targeted Decommission:Delete grant) pick up the new securable automatically on upgrade, with no operator action required.

A previously revoked decommission ability is NOT silently regained. If you had used remove_permission() to revoke ITServiceOwner’s SoftwareLicensing:Delete, Inventory:Delete, or GuaranteedState:Delete specifically to strip its decommission ability under the old conjunction, the upgrade detects that revocation and carries it forward onto the new Decommission:Delete grant automatically — ITServiceOwner stays unable to decommission a device after upgrading, exactly as before. No operator action is needed for this case; it is the inverse of the custom-role gap above (a seeded role losing access it deliberately had revoked is checked for, not just a custom role gaining an unrelated 403).

This is breaking only for a custom role. Any custom role an operator built by hand-assembling the old three-securable conjunction (SoftwareLicensing:Delete + Inventory:Delete + GuaranteedState:Delete) specifically to reach this endpoint will start getting 403 Decommission:Delete on its next call — the three old grants no longer suffice, because the route no longer checks them at all. Audit your custom roles before upgrading:

SELECT pr.principal_type, pr.principal_id, pr.role_name
  FROM rbac_store.principal_roles pr
  JOIN rbac_store.role_permissions rp ON rp.role_name = pr.role_name
  WHERE rp.securable_type IN ('SoftwareLicensing', 'Inventory', 'GuaranteedState')
    AND rp.operation = 'Delete'
    AND rp.effect = 'allow'
  GROUP BY pr.principal_type, pr.principal_id, pr.role_name
  HAVING COUNT(DISTINCT rp.securable_type) = 3;

The old gate was a strict three-way AND — all three grants required together, effect = 'allow' on each. Matching on any ONE of the three (or including an explicit effect = 'deny' row, a legal role_permissions state) over-reports: a role holding, say, only Inventory:Delete for an unrelated purpose never had decommission ability under the old conjunction, and granting it Decommission:Delete now would be a NEW grant, not a preserved one.

Grant Decommission:Delete to any custom role in that list that must retain the ability to erase a decommissioned device’s data. SoftwareLicensing:Delete, Inventory:Delete, and GuaranteedState:Delete continue to gate their own unrelated surfaces exactly as before — only the erasure cascade’s gate moved.

Rolling upgrade (the seventh store, offline_endpoint). During a rolling upgrade, a DELETE served by a replica that has not been upgraded runs the old cascade, returns 200, and does not erase the presence row; a response whose stores map has no offline_endpoint key did not erase it, and the sle.agent.decommission success audit row that replica writes lists its own stores (no offline_endpoint), so it is not evidence the presence row was erased. Re-issue the DELETE once every replica runs this version.

vNEXT — Hardware CI list requires Inventory:Read, not the old Devices page’s Infrastructure:Read (breaking for custom roles)

/devices now 302-redirects to /hardware (the Hardware CI list) and /device?id= redirects to /hardware/ci?id= — bookmarked links and existing browser navigation keep working transparently, since browsers follow redirects. The permission the list itself checks changed, though: the old Devices page gated on Infrastructure:Read; the Hardware CI list gates on Inventory:Read (the same securable the rest of the Inventory/Software surface already uses).

Seeded roles are unaffected. Every built-in role that held Infrastructure:Read (Administrator, ITServiceOwner) also holds Inventory:Read, so no default-role regression on upgrade.

This is breaking only for a custom role granted Infrastructure:Read without Inventory:Read. Such a role could see the device list via the old /devices page pre-upgrade and will get a 403 reaching the redirected /hardware page post-upgrade. Audit custom roles before upgrading:

SELECT pr.principal_type, pr.principal_id, pr.role_name
  FROM rbac_store.principal_roles pr
  JOIN rbac_store.role_permissions rp ON rp.role_name = pr.role_name
  WHERE rp.securable_type = 'Infrastructure' AND rp.operation = 'Read' AND rp.effect = 'allow'
    AND NOT EXISTS (
      SELECT 1 FROM rbac_store.role_permissions rp2
      WHERE rp2.role_name = pr.role_name
        AND rp2.securable_type = 'Inventory' AND rp2.operation = 'Read' AND rp2.effect = 'allow'
    );

Grant Inventory:Read to any custom role in that list that must retain device-list visibility. Infrastructure:Read continues to gate the fleet-wide agent list (/api/agents) and other infrastructure-tier reads exactly as before — only the Hardware CI list’s own gate moved.

Retention clock guards (#2360 server audit store, #2361 Echo agent warehouse, #2964 rotation sweep)

Both retention paths used to issue an unbounded DELETE driven by the local wall clock, so one forward clock step could wipe a store in a single statement. They are now guarded and capped. Two operator-visible consequences on upgrade:

New Prometheus alert rules ship in docs/prometheus/yuzu-alerts.yml, which is the current set. The declined-pass and failed-pass counters must be alerted on separately: both leave rows undeleted, so an audit table that never shrinks looks identical either way. One rule, YuzuAuditRetentionNotRunning, fires on the reaper NOT running - the state in which none of the other counter-driven rules can fire, because they all key on a counter rising.

SLE — the SoftwareLicensing securable auto-grants on upgrade (ADR-0024)

This release adds the SLE (Software Licensing & Entitlements) discovery surface, gated on a new SoftwareLicensing RBAC securable. Per ADR-1005 the server ships the discovery mechanism only: the per-device GET /api/v1/sle/agents/{id} drill, its machine-scope MCP twin query_software_licenses, and the audited erasure DELETE /api/v1/sle/agents/{id}. The licence compliance / entitlement / reclamation views and the fleet posture reads (a /sle page and /api/v1/sle/summary · /licenses · the per-product device fan-out) interpret discovered facts and ship with the future SAM use-case-engine module — they are not in this release, so a reader who saw the earlier ADR expecting an in-server compliance page should not look for one yet. Because RBAC role defaults are seeded with INSERT OR IGNORE on every boot, an upgrading deployment silently auto-grants the new securable to the built-in roles the first time it starts the new build:

RoleGrant
Viewer, PlatformEngineerRead
OperatorRead + Write
ITServiceOwner, Administratorfull CRUD
ApiTokenManagernone

No action is required if that matches your intent — the securable gates the SLE discovery reads/erasure (GET/DELETE /api/v1/sle/agents/{id} and the MCP twin); the /inventory software catalog is unchanged and remains under Inventory:Read.

But the per-device drill exposes each endpoint’s detected-licence facts — product, vendor, channel, status, expiry, and, on per-user surfaces, the per-user user_ref identifier — to every Read holder, including Viewer. A deployment that must restrict this visibility should deny or remove the SoftwareLicensing Read grant from Viewer (and any other broad role) BEFORE enabling the SLE sources. A deny rule wins over the seeded allow (deny-override), so an explicit deny is the durable control — re-seeding on the next boot cannot re-open it. (Entitlement/purchase cost metadata is not served in-server this release — it is the SAM UCE module’s, governed by that module’s own RBAC when it ships.)

The SLE detection source also collects a new, suppressible per-user identifier (user_ref) on per-user licence surfaces. It defaults to a per-device keyed-HMAC pseudonym; see Software licence detection for the --license-scan-user-ref flag, the honest limits of that default, and the --inventory-disable source-level opt-out.

Gateway listeners move to grpc_listeners (the Cowboy edge) — BREAKING

Affects every operator who runs the Erlang gateway, from the first release containing the yuzu_gw grpc_listeners shape. The gateway’s two listeners (agent :50051, management :50063) are now served by the gateway’s own gRPC layer over Cowboy and Ranch instead of grpcbox, and they are configured under {yuzu_gw, [{grpc_listeners, [...]}]} instead of {grpcbox, [{servers, [...]}]}. The wire protocol, the protobuf contracts and the management-plane peer pin are unchanged. The channel to the server moves in the same release, with the same kind of refusal: see “The upstream channel moves to upstream_client” below, and make both edits in one pass (that section has the combined before and after). Every config and compose file shipped in this repository is converted. A config you wrote yourself is not: it has to be converted before the new gateway will start.

The gateway image and its sys.config are an atomic pair. Never change one without the other:

Key mapping. One listener at a time:

Old key (inside {grpcbox, [{servers, [...]}]})New key (inside {yuzu_gw, [{grpc_listeners, [...]}]})
grpc_opts => #{service_protos, services, auth_fun}Dissolved: service_protos, services and auth_fun are keys of the listener entry itself.
listen_opts => #{port, ip}listen => #{port, ip}
transport_opts => #{ssl => true, certfile, keyfile, cacertfile, verify, fail_if_no_peer_cert}tls => #{certfile, keyfile, cacertfile, verify, fail_if_no_peer_cert}. ssl => true is gone: the tls map’s presence turns TLS on. A leftover transport_opts refuses boot.
pool_opts => #{size => N}acceptors => N
(none)name: mandatory and unique.
(none)origin, http2, limits: new and optional.
{yuzu_gw, [{agent_listen_addr, _}, {agent_listen_port, _}, {mgmt_listen_addr, _}, {mgmt_listen_port, _}]}Dead keys that no code reads; remove them or leave them. The bind is listen.

grpc_listeners belongs in the yuzu_gw application block, not in grpcbox. A listener entry may carry only the keys the gateway reads (name, service_protos, services, listen, tls, auth_fun, origin, acceptors, http2, limits; limits takes max_connections, max_message_bytes and send_timeout_ms), and its tls, listen, http2, limits and origin maps only theirs: any other key, a legacy key above and a misspelled one alike, refuses boot and is named in the log. The legacy keys the gateway names are transport_opts, listen_opts, grpc_opts, pool_opts and ssl.

Not carried over. The new maps are closed, and several things the old shape could carry have no equivalent. Check your old entries for them before you convert:

Worked example. Before, the agent listener with one-way TLS:

{grpcbox, [
    {servers, [
        #{grpc_opts => #{service_protos => [agent_pb],
                         services => #{'yuzu.agent.v1.AgentService' => yuzu_gw_agent_service}},
          listen_opts => #{port => 50051, ip => {0,0,0,0}},
          transport_opts => #{ssl => true,
                              certfile   => "/etc/yuzu/certs/default-gateway.pem",
                              keyfile    => "/etc/yuzu/certs/default-gateway.key",
                              cacertfile => "/etc/yuzu/certs/default-ca.pem",
                              verify => verify_none,
                              fail_if_no_peer_cert => false}}
    ]}
]}

After:

{yuzu_gw, [
    {grpc_listeners, [
        #{name => agent,
          service_protos => [agent_pb],
          services => #{'yuzu.agent.v1.AgentService' => yuzu_gw_agent_service},
          listen => #{port => 50051, ip => {0,0,0,0}},
          tls => #{certfile   => "/etc/yuzu/certs/default-gateway.pem",
                   keyfile    => "/etc/yuzu/certs/default-gateway.key",
                   cacertfile => "/etc/yuzu/certs/default-ca.pem",
                   verify => verify_none,
                   fail_if_no_peer_cert => false}}
    ]}
]}

with the {servers, ...} block deleted. The management listener keeps its auth_fun => fun yuzu_gw_authz:check_mgmt_peer/1 and, in {yuzu_gw, ...}, its mgmt_peer_pins. To keep strict mutual TLS on the management listener, omit verify and fail_if_no_peer_cert from its tls map: the defaults are verify_peer and true, and copying the agent listener’s verify_none onto it would leave the command plane without transport authentication (the boot guard refuses a network-reachable management listener that has it). gateway/config/sys.config.prod is the full reference.

The refusals. With a {servers, ...} block still present the gateway logs, at critical level, and exits:

Refusing to start: {grpcbox, servers} is set. The gateway's listeners are configured in {yuzu_gw, grpc_listeners} (one entry per listener) and nothing reads {grpcbox, servers}: that block would configure no listener. Move the listener settings into grpc_listeners and remove the {servers, ...} block (see config/sys.config.prod and docs/user-manual/gateway.md).

The check runs before anything is bound, and the gateway contains no grpcbox to serve a servers block, so no listener is opened for it. It is refused rather than ignored because an operator who kept the block believes a listener is configured that is not. The gateway also refuses to start, with a critical log line that names the defect, when: grpc_listeners is unset, empty or [] (the old advice to delete the servers block and add nothing leaves a gateway that is healthy and accepts nothing; {yuzu_gw, [{allow_no_listeners, true}]} is the lab acknowledgement); an entry has a legacy or unknown key; a tls map lacks certfile, keyfile or cacertfile; an auth_fun is set on a listener that serves the agent service (agent_pb in service_protos, or the AgentService itself in services); no entry routes the AgentService (an entry counts only if its services map has the key 'yuzu.agent.v1.AgentService'; service_protos => [agent_pb] with an empty services does not); a listener has no name or reuses one, or reuses a fixed port; a proxy origin has no sources; or socket_opts tries to move the bind (ip, ifaddr, fd, port, local). An entry defect is printed as {Name, Term} (the entry’s name, or unknown when it has none) inside the Refusing to start: invalid {yuzu_gw, grpc_listeners} entry/entries line (the bad_grpc_listeners refusal); no_grpc_listeners and no_agent_listener are plain sentences in the log. All three names are what the pre-flight below prints. The next table says what to change for each term.

Term in the logFix
legacy_keysA key of the old shape (transport_opts, listen_opts, grpc_opts, pool_opts, ssl): move it as in the key mapping. The term is {legacy_keys, Where, [{Key, Hint}]}: it names the key and where it went.
unknown_keysA misspelled key, or one the gateway does not read: use only the keys listed in gRPC listeners (the term lists the allowed ones).
missing_keysThe entry lacks name, service_protos, services or listen.
tls_incompleteA tls map needs certfile, keyfile and cacertfile; omit the map for plaintext.
bad_tls_valuesverify is not verify_peer/verify_none, fail_if_no_peer_cert is not a boolean, or ciphers is not a proper, non-empty list of strings or tuples.
auth_fun_on_agent_listenerAn auth_fun on a listener that serves the agent service (agent_pb, or the AgentService itself in services). Remove it: it would cut agent enrollment, and only the management listener authenticates its peer.
listen_socket_opts_set_the_bindsocket_opts carries ip, ifaddr, fd, port or {local, _}; the bind is listen alone.
listen_socket_opts_set_tls_policysocket_opts carries client_renegotiation or secure_renegotiate (#5695). Remove them: the gateway fixes both on every TLS listener (client renegotiation refused, secure renegotiation required). When socket_opts also moves the bind, the bind term is reported instead.
bad_listen_iplisten.ip is not an IP address tuple.
bad_acceptorsacceptors is not a positive integer.
{unknown_keys, limits, [BadKey], {allowed, [...]}}A limits key the gateway does not read. Every unknown_keys term has this shape, with the map it was found in (entry, tls, listen, http2 or limits; the origin map has unknown_origin_options) as the second element and the allowed keys last; for limits they are max_connections, max_message_bytes and send_timeout_ms.
bad_limitslimits is not a map, or a key’s value is not valid (max_connections and send_timeout_ms take a positive integer or infinity, max_message_bytes a positive integer); the line lists the keys with bad values.
duplicate_nameTwo entries share a name.
duplicate_portTwo entries bind the same fixed port.
no_grpc_listenersgrpc_listeners is unset or empty; add the listeners to the yuzu_gw block.
no_agent_listenerNo entry routes the AgentService: no entry’s services map has the key 'yuzu.agent.v1.AgentService'. service_protos => [agent_pb] with services => #{} does not count, because nothing would answer an agent. No agent could connect and /readyz would never be ready. Only the literal {allow_no_listeners, true} excuses it (a lab or test).
entry_not_a_mapAn element of grpc_listeners is not a map.
grpc_listeners_not_a_listgrpc_listeners itself is not a list (the line prints it as {unknown, grpc_listeners_not_a_list}).
bad_service_protos, bad_servicesservice_protos is not a list of proto module atoms (a list with an improper tail included), or services is not a map of service name to module.
bad_auth_funauth_fun is set on a listener that does not serve the agent service but is not a one-argument fun.
bad_listen_socket_optslisten.socket_opts is not a list.
bad_routesThe services map could not be turned into a route table: the line gives {unknown_service, Name} (a service name that none of the service_protos defines) or {module_not_loaded, Module} (the module is not in the release).
{listen_failed, Reason}Not a config-shape defect: the entry validated, but the listener could not start (the port is already bound, eaddrinuse; the address is not on this host; or a tls file is missing or unreadable by the gateway user). It appears in the crash report, not as a Refusing to start line, and the pre-flight does not detect it: check ss -ltn and the file permissions.
bad_routes with {service_without_routes, Name}A service in the services map whose module exports none of that service’s methods (the arity-2, snake_case functions, register/2, heartbeat/2, and so on). This is nearly always a mistyped module name: the listener would start, /readyz would report ready, and every RPC of that service would answer UNIMPLEMENTED. A module that exports only some of the methods is accepted, and a request for one it lacks answers UNIMPLEMENTED. Name the module that implements the service (yuzu_gw_agent_service for the agent, yuzu_gw_mgmt_service for management).
origin_sources_required, bad_origin_sourcesA proxy origin (proxy_protocol or forwarded_for) needs a non-empty sources list; each element is an IP address or {Ip, Prefix} (a prefix of 0, or above the address width, is refused); a sources that is not a proper list reports {bad_origin_sources, improper_list}.
{ipv4_mapped_source_prefix_must_exceed_96, [...]}A sources element {::ffff:a.b.c.d, P} with P of 96 or less (counted over the 128-bit address); use a prefix of 97 to 128.
unknown_origin_mode, unknown_origin_options, origin_transport_takes_no_options, bad_originorigin.mode is not transport, proxy_protocol or forwarded_for; the origin map has a key its mode does not take; transport takes no other key; or origin is not a map with a mode.
bad_trusted_hopsorigin.trusted_hops is not an integer from 0 to 31.
auth_fun_without_tlsAn auth_fun needs a tls map; add it.
auth_fun_behind_terminating_proxyAn auth_fun on a forwarded_for listener: the check would run on the proxy’s certificate. Remove one of them.
bad_listen, bad_tls, bad_http2, reserved_http2_optionsA listen map without an integer port from 0 to 65535, a tls or http2 value that is not a map, or an http2 option the gateway owns.
bad_grpc_listenersThe summary line for any of the entry defects above; the entries and their defects are printed with it.

Environment variables. YUZU_GW_AGENT_PORT and YUZU_GW_MGMT_PORT changed nothing about the listeners before; they now rewrite listen.port of the entry that serves agent_pb and of the entry that serves management_pb, and only when exactly one entry serves the proto (otherwise a warning is logged and the variable is ignored). Check the value before you upgrade: a gateway container or unit that has either variable set to a number other than its configured port (a host-port number copied from an .env file, for example) moves its listener on the first start of the new image. The server’s settings page lists YUZU_GW_MGMT_PORT with default 50052; the management listener is on 50063.

Pre-flight (offline, before you touch production). On the host, with the NEW release unpacked beside the old one and the production YUZU_GW_* environment set, this runs the gateway’s own config checks without starting a listener. It covers both moves in this release: the legacy {grpcbox, [{servers, ...}]} and {grpcbox, [{client, ...}]} blocks, the grpc_listeners entries, the upstream_client channels and the management-listener posture. application:load(yuzu_gw) is required: without it the application environment is not read and the check passes falsely, and the script exits 1 if the load fails (a yuzu_gw that is not on the -pa path). A pre-flight script from the previous release does not work on this one: it also loaded grpcbox, which is not in the release any more, so it exits 1 with {error,{"no such file or directory","grpcbox.app"}} before it checks anything. Use the script below. It also runs the upstream TLS checks the boot runs: any https endpoint without {verify, verify_peer} and a cacertfile / cacerts of its own, and a channel that mixes http and https endpoints, exit 1 here (if YUZU_GW_ALLOW_UNVERIFIED_UPSTREAM=1 is exported the unverified case passes with a warning, as it does at boot). It does not check the distribution cookie, whether the endpoint carries a client certificate (the server demands that), that the certificate files exist, or that the ports can bind (a {listen_failed, Reason} in the crash report means one of those; see the table below). The config file you pass must be named *.config (a file called sys.config.prod is looked up as sys.config.prod.config and not found: copy it to a .config name). Proceed only on exit 0; a non-ok entry in the printed list is the refusal (legacy_grpcbox_servers, legacy_grpcbox_client, no_grpc_listeners, no_agent_listener, bad_grpc_listeners, no_upstream_channel, bad_upstream_client_config, mixed_upstream_endpoints, unverified_upstream_tls, or the management-posture defect), and the critical log line above it says why. On an image from before this change it fails with undef and exits 1, which tells you the image is old.

R=/opt/yuzu_gw; C=$R/releases/0.2.0/sys.config
$R/erts-*/bin/erl -noshell -noinput -boot $R/releases/0.2.0/start_clean -config $C \
  -pa $R/lib/*/ebin -eval '
Load = [application:load(yuzu_gw)], yuzu_gw_app:apply_env_overrides(),
Results = Load ++ [yuzu_gw_app:check_legacy_grpcbox_servers(), yuzu_gw_app:check_legacy_grpcbox_client(),
                   yuzu_gw_app:check_grpc_listeners(), yuzu_gw_app:check_upstream_client_config(),
                   yuzu_gw_app:check_upstream_channel_mix(), yuzu_gw_app:check_upstream_tls_posture(),
                   yuzu_gw_app:check_mgmt_listener_posture()],
io:format("PREFLIGHT ~p~n", [Results]),
halt(case lists:all(fun(X) -> X =:= ok end, Results) of true -> 0; false -> 1 end).'

Pre-flight for container operators. Run the same command in a throwaway container from the NEW image, with the candidate sys.config mounted and the production YUZU_GW_* variables passed with -e; bypass the entrypoint, which only prepares the node’s address. In the image the release is unpacked at /opt/yuzu_gw and the release version directory is releases/0.2.0 (deploy/docker/Dockerfile.gateway copies _build/prod/rel/yuzu_gw/ there, and the reference compose mounts the config at /opt/yuzu_gw/releases/0.2.0/sys.config; adapt R and the version to your release root if yours differs). Save this as preflight.sh, which is the command above with the config taken from its first argument:

#!/bin/sh
R=/opt/yuzu_gw; C=$1
$R/erts-*/bin/erl -noshell -noinput -boot $R/releases/0.2.0/start_clean -config "$C" \
  -pa $R/lib/*/ebin -eval '
Load = [application:load(yuzu_gw)], yuzu_gw_app:apply_env_overrides(),
Results = Load ++ [yuzu_gw_app:check_legacy_grpcbox_servers(), yuzu_gw_app:check_legacy_grpcbox_client(),
                   yuzu_gw_app:check_grpc_listeners(), yuzu_gw_app:check_upstream_client_config(),
                   yuzu_gw_app:check_upstream_channel_mix(), yuzu_gw_app:check_upstream_tls_posture(),
                   yuzu_gw_app:check_mgmt_listener_posture()],
io:format("PREFLIGHT ~p~n", [Results]),
halt(case lists:all(fun(X) -> X =:= ok end, Results) of true -> 0; false -> 1 end).'

then run it in the new image:

docker run --rm --entrypoint sh -e YUZU_GW_AGENT_PORT -e YUZU_GW_MGMT_PORT \
  -v "$PWD/preflight.sh:/preflight.sh:ro" -v "$PWD/gateway-sys.config:/pf.config:ro" \
  NEW_GATEWAY_IMAGE /preflight.sh /pf.config

The exit status of docker run is the script’s. The erl command was run on a compiled tree (exit 0 on the shipped sys.config.prod and deploy/docker/gateway-sys.config; exit 1, with the critical line, on a config carrying a {grpcbox, [{client, ...}]} block, on one carrying a {grpcbox, [{servers, ...}]} block, on one with no upstream_client, on one whose upstream_client is not a channels map, and on a -pa path without yuzu_gw) and, in the release layout above (erts-*/bin/erl, releases/0.2.0/start_clean, lib/*/ebin), on a scratch production release (exit 0 on its shipped sys.config). The scan for a legacy block reads the config files named by -config, so give the script the same file the node will start with. The two upstream checks added to the command (channel mix and TLS posture) were run on a compiled tree: exit 0 on the shipped sys.config.prod, exit 1 with the critical line on an https channel that has verify_peer and no cacertfile. The docker run wrapper (the entrypoint override, the mounts) was written from the Dockerfile and the compose files and has not been run.

Old and new gateways side by side is untested. Rolling forward replaces one gateway at a time (below), so for a while the fleet runs both shapes. Nothing in the code ties the two together beyond the protocol, but a mixed fleet has not been run against a single server, which is one more reason to drain each gateway from the load balancer before you replace it.

Rolling forward. Back up first: the tarball flow above (rm -rf /opt/yuzu_gw) deletes releases/*/sys.config with the old release and a package upgrade overwrites it (both described under Linux (systemd)), so copy the old sys.config out and keep both files. Convert and pre-flight the new config, then upgrade one gateway at a time: drain it from the load balancer, replace the image and the config together, and wait for ss -ltn | grep -E ':50051|:50063' to show both ports, curl -s localhost:8081/readyz to answer 200 (it now also requires each configured listener to be running) and yuzu_gw_agents_current to climb. A gateway restart disconnects every agent that gateway holds, and on agent builds that do not yet carry the #5183 reconnect fix (shipped in 0.15.0) a released agent may stay wedged after it: restart the agent service on the agents that stay rejected only if they do. Then upgrade every gateway before you rely on source_ip in the audit log: an older gateway still forwards the gateway_observed_peer the agent sent, so a forged address passes through it.

Rolling back. Restore the old image and the old config together, then verify as above. Restoring only the image leaves the old-image-plus-new-config pairing, which serves nothing and reports healthy. Restoring only the config leaves the new-image-plus-old-config pairing, which refuses to start.

Symptoms of a wrong pairing.

SymptomCauseFix
Exit 1 within seconds, Refusing to start: {grpcbox, servers} is setNew image, old configConvert the config (above), or restore the old image.
Exit 1 within seconds, Refusing to start: {grpcbox, [{client, ...}]} is setNew image, config with the upstream channel still under grpcbox (a listener-only conversion)Move the value to {yuzu_gw, [{upstream_client, ...}]} (next section).
Exit 1, Refusing to start: no upstream channel is configuredupstream_client is unset, empty, or has no channel named default_channel (channels under other names do not count)Add the default_channel (next section).
Exit 1, Refusing to start: the upstream channel configuration {yuzu_gw, upstream_client} is unusable (<Why>)upstream_client has a shape error (bad_client_config, bad_channel, bad_endpoint, bad_host, bad_socket_options)Fix the entry the shape names (next section).
Exit 1, Refusing to start: the upstream channel uses TLS (https) WITHOUT {verify,verify_peer} and a {cacertfile,...} of its ownAn https endpoint lacks verify_peer or a CA of its ownAdd {verify, verify_peer} and a cacertfile to every https endpoint (or use http on a lab rig).
Exit 1, Refusing to start: the upstream channel <name> lists both plaintext (http) and TLS (https) endpointsOne channel mixes schemesGive the channel one scheme.
VM exits at once with invalid config data: duplicate application config: yuzu_gw (before any gateway log line)The upstream_client or grpc_listeners value was added as a second {yuzu_gw, [...]} tupleMerge it into the existing {yuzu_gw, [...]} list; the pre-flight exits 1 on it too.
Boot succeeds but every upstream call fails and the channel logs connection <n> down (...); agents do not registerThe channel is unusable at run time: a CA or certificate path that does not exist ({options,tls} in the log, and no boot guard checks it, because the server may create the files after the gateway starts), or a name that does not resolveCheck the paths and the dialled name; see The upstream client.
Process up, /readyz 200, ports 50051 and 50063 refuse connections, yuzu_gw_agents_current 0, no listener logOld image, new configRestore the old config, or deploy the new image.
Exit 1, Refusing to start: {yuzu_gw, grpc_listeners} is unset or emptyThe servers block was deleted and nothing added, or grpc_listeners sits in the grpcbox blockAdd the listeners to the yuzu_gw block.
Listeners up, every TLS handshake fails with no_suitable_cipherstls.ciphers is set to a list with no suite for the certificate’s key type (ECDSA suites with an RSA certificate, say)Remove ciphers (the default follows the key type) or list suites that fit the key (below).
Works, but the audit log shows many agents with one source_ipA load balancer or router in front of the gateway and no origin configuredConfigure origin (below).
Exit 1, a critical line naming the management listener and a defectThe management listener lacks the required posture (plaintext, missing_cert_material, verify_not_peer, peer_cert_not_required, missing_or_wrong_auth_fun, no_mgmt_peer_pins (mode pin), no_mgmt_peer_identity (mode identity), no_mgmt_peer_pins_or_identity (mode identity_or_pin), proxied_origin)Fix the entry as in sys.config.prod; the lab-only override is allow_insecure_mgmt.

Bring-your-own certificates. The TLS cipher list is chosen by the certificate’s key type unless tls.ciphers is set: the ECDHE-ECDSA AEAD suites for an EC key (the certificates the server generates), ECDHE-RSA AES-GCM suites for an RSA key. TLS 1.2 only, as before. This is narrower than the OTP default list the grpcbox listeners used; an agent that needs a suite outside the list needs tls.ciphers. FIPS mode is not claimed.

TLS behaviour changes. The new listeners set reuse_sessions => false and honor_cipher_order => true; the grpcbox listeners took OTP’s defaults for both. With session reuse off, every reconnect is a full TLS handshake, so a reconnect herd (every agent behind a proxy that cut its idle connections, a gateway restart) costs more CPU than it did. The gateway’s own cipher order now decides the suite, not the client’s. TLS stays 1.2 only.

Connection ceilings. sys.config.prod and the reference configs now set limits => #{max_connections => 60000, max_message_bytes => 4194304} on the agent listener and max_connections => 256 on the management listener (before, neither had a ceiling). max_connections bounds file descriptors and is not protection against a hostile peer:

Size max_connections under the gateway’s file-descriptor limit (60000 assumes LimitNOFILE=65536). The compose files set no ulimits; add ulimits: nofile: 131072 to the gateway service and raise the agent listener’s limits.max_connections with it (keep about 5500 descriptors below the limit): the 60000 ceiling assumes about 65536 descriptors, so a larger descriptor limit alone does not move it. Check the hard limit in /proc/1/limits (the BEAM raises its soft limit to it).

A proxy without origin. A gateway behind a load balancer or router that has no origin configured attributes every agent to the proxy’s address and nothing flags it: the audit source_ip is the proxy and there is no origin_observed=false. Before this change the agent’s own (empty) value was forwarded, so the audit row carried origin_observed=false in the same situation. Configure origin for the proxy you run (see Client addresses behind a proxy).

Verify after the upgrade. ss -ltn | grep -E ':50051|:50063' shows both listeners; curl -s localhost:8081/readyz returns 200; yuzu_gw_agents_current climbs to the fleet size; behind a proxy, yuzu_gw_grpc_origin_ignored_total{reason=~"no_claim|bad_claim"} stays flat and the audit source_ip is each agent’s own address. The server writes the gateway origin to the audit log only on enrollment.token_consumed, so read source_ip from that row, for an agent that enrolled with an enrollment token after the upgrade; a Register without a token leaves no row to read.

The upstream channel moves to upstream_client — BREAKING

Affects every operator who runs the Erlang gateway; it ships in the same release as the listener move above, so convert both in one edit. The gateway’s connection to the server (GatewayUpstream, port 50055 by default) is now made by the gateway’s own gRPC client (yuzu_grpc_client) over gun, and the gateway no longer contains grpcbox (nor chatterbox, hpack, acceptor_pool or gproc), its patched vendored copy, or the grpcbox rebar3 plugin. The wire protocol, the protobuf contracts and the server are unchanged: no server or agent change. Every config and compose file shipped in this repository is converted; a config you wrote yourself is not.

UPGRADE ACTION. Move the upstream channel value from the grpcbox application to the yuzu_gw application. The value V does not change:

BeforeAfter
{grpcbox, [{client, V}]}{yuzu_gw, [{upstream_client, V}]}
{grpcbox, [{servers, L}]}{yuzu_gw, [{grpc_listeners, L2}]} (L2 is L converted, see the key mapping above)

and remove the {grpcbox, ...} block. Merge the values into your existing {yuzu_gw, [...]} list: a second {yuzu_gw, ...} tuple is a different mistake, and the VM refuses it before the gateway starts (invalid config data: duplicate application config: yuzu_gw). The log lines in the refusals below say config/sys.config, which is the file in the source tree; on an installed gateway the file is /opt/yuzu_gw/releases/<vsn>/sys.config (a container’s mount point, or the file your compose file mounts, otherwise). Before and after for a whole file, both moves at once:

%% BEFORE (the last release)
[
  {grpcbox, [
    {client, #{channels => [
        {default_channel, [{https, "yuzu-server.example.com", 50055, [
             {cacertfile, "/etc/yuzu/certs/default-ca.pem"},
             {certfile,   "/etc/yuzu/certs/default-gateway.pem"},
             {keyfile,    "/etc/yuzu/certs/default-gateway.key"},
             {verify, verify_peer}]}], #{}}]}},
    {servers, [ #{grpc_opts => ..., listen_opts => ..., transport_opts => ...} ]}
  ]},
  {yuzu_gw, [ ... ]}
].

%% AFTER
[
  {yuzu_gw, [
    ...,
    {upstream_client, #{channels => [
        {default_channel, [{https, "yuzu-server.example.com", 50055, [
             {cacertfile, "/etc/yuzu/certs/default-ca.pem"},
             {certfile,   "/etc/yuzu/certs/default-gateway.pem"},
             {keyfile,    "/etc/yuzu/certs/default-gateway.key"},
             {verify, verify_peer}]}], #{}}]}},
    {grpc_listeners, [ #{name => agent, ...}, #{name => mgmt, ...} ]}
  ]}
].

Why the old key cannot be left in place. OTP applies a sys.config entry only to an application that is loaded, and nothing loads one named grpcbox any more, so a block left under that name would be silently invisible: the file would read as configured while the gateway ignored its TLS client options. The gateway therefore refuses to boot instead, before any listener is bound, when it finds {grpcbox, [{client, _}]} or {grpcbox, [{servers, _}]}, either in the application environment or in any config file the VM was started with (a file named by another config file is followed, a few levels deep). It checks servers first, so a file that has both reports servers, and the client refusal appears only once that block is gone; the pre-flight above reports both at once. The critical log lines are:

Refusing to start: {grpcbox, [{client, ...}]} is set. The upstream channels are configured in {yuzu_gw, [{upstream_client, ...}]} and nothing reads the old key. Move the value across unchanged and remove the {grpcbox, ...} block (see config/sys.config and docs/user-manual/gateway.md).

The gateway also refuses to start when no channel is named default_channel (the registration and heartbeat code calls that name; a gateway without it would register no agent and forward no heartbeat while /readyz answered ready, because the upstream check only tests that a process is alive):

Refusing to start: no upstream channel is configured. Set {yuzu_gw, [{upstream_client, #{channels => [{default_channel, [Endpoint], #{}}]}}]} (see config/sys.config and docs/user-manual/gateway.md); without it the gateway can register no agent and forward no heartbeat.

Only the literal {yuzu_gw, [{allow_no_upstream, true}]} excuses it (a gateway under test), and a value that is not a channels map, or a channel, endpoint, host or socket_options of the wrong shape, is refused with Refusing to start: the upstream channel configuration {yuzu_gw, upstream_client} is unusable (Why), where Why is one of bad_client_config, bad_channel, bad_endpoint, bad_host or bad_socket_options and the shape of the offending term, never its value (SSL options name key files). The same boot also refuses, unless YUZU_GW_ALLOW_UNVERIFIED_UPSTREAM=1 (dev only) is set:

Production is https, verify_peer, a CA-pinned cacertfile and a client certificate and key (mutual TLS). The boot log says mutual-TLS only when every endpoint carries certfile and keyfile, and server-authenticated TLS (no client certificate) otherwise.

What else changes (the client is described in The upstream client):

Rolling forward and back are the listener move’s rules (above): replace the image and the config together, one gateway at a time. A new image with the old config refuses to start. An old image with the new config ignores both grpc_listeners and upstream_client: it starts, reports healthy and binds nothing (see “Old image + new config” above). It also has no upstream channel, because it reads no upstream_client (INFERRED from the old image’s use of {grpcbox, [{client, ...}]}; not tested). Rolling back restores the old image and the old config (which carries both grpcbox blocks) together.

Gateway distribution cookie now required (#659) — BREAKING

The Erlang gateway shipped a hardcoded default distribution cookie (yuzu_gw_secret_change_me). The cookie is the sole authentication for inter-node RPC, so a publicly-known value is unauthenticated remote code execution for anyone who can reach EPMD (TCP 4369). The gateway now refuses to boot with the default (or an empty/unsubstituted) cookie unless explicitly overridden.

Before upgrading:

Recovery — gateway won’t start after upgrade:

SymptomDiagnoseFix
systemctl status yuzu-gateway shows start-limit-hit / failedjournalctl -t yuzu-gateway | grep -i cookie shows “insecure distribution cookie” (for manual/foreground or container runs, check stdout / gateway.log instead)Create /etc/yuzu/gateway.env with YUZU_GW_COOKIE=$(openssl rand -hex 32) (see above), then systemctl reset-failed yuzu-gateway && systemctl start yuzu-gateway. Do not use YUZU_GW_ALLOW_DEFAULT_COOKIE=1 in production.
Gateway container restarts at boot on an AMX-capable Intel host (Sapphire Rapids+, AWS c7i/m7i/r7i)Container log has sys_sigaltstack(): Internal error: Failed to set alternate signal stack; image is 0.13.0 through 0.14.0-rc4 (#2150)Pull an image that carries the #2150 fix (0.14.0-rc5 or later). Until then, run the gateway on a host without AMX. A single-node gateway may use yuzu-gateway-chisel instead; a clustered one may not (it never joins the cluster). Do not roll back to 0.13.0 on that host: it is affected too.

The generated /etc/yuzu/gateway.env is intentionally preserved across apt purge / rpm -e (the /etc/yuzu directory may be shared with other Yuzu packages). To remove the cookie after uninstall: sudo shred -u /etc/yuzu/gateway.env.

Never set YUZU_GW_ALLOW_DEFAULT_COOKIE=1 in production — it disables the guard and restores the unauthenticated-RPC surface. It exists only for ephemeral dev/CI stacks.

Gateway distribution-cookie length floor + shared node identity (HA WS-4 #4555) — BREAKING

Extends the #659 cookie requirement above with two further changes, both shipped together as part of gateway multi-node cluster formation:

Recovery for the cookie-length case uses the same diagnostic path as #659 above (journalctl -t yuzu-gateway | grep -i cookie / stdout / gateway.log) — the log line names the specific failure (shorter than 32 characters) separately from the known-default rejection.

InstructionDefinition import signature enforcement now on-by-default (#1073 / W7.4 sibling-gap) — BREAKING

InstructionStore::import_definition_json (the storage path behind POST /api/instructions/import) previously accepted unsigned JSON envelopes without verification. After upgrade, unsigned imports are rejected with:

instruction-import is unsigned and signature enforcement is enabled (set --allow-unsigned-definitions / YUZU_ALLOW_UNSIGNED_DEFINITIONS=1 to bypass)

This is intentional and closes the sibling-gap to #802. Without the gate, any operator with InstructionDefinition:Write permission can import a definition that dispatches an arbitrary plugin invocation on every targeted agent — same fleet-RCE blast radius the pack-signing default closed.

Two migration paths, in order of preference:

  1. Sign your imports. Generate an Ed25519 keypair, sign the yaml_source field’s bytes with the private key, and wrap the envelope with signature: <hex> + publicKey: <hex> top-level fields. The wire format mirrors ProductPack: signature is hex- encoded over the verbatim yaml_source bytes; publicKey is the hex-encoded raw Ed25519 public key (32 bytes / 64 hex chars). Until the dedicated helper script lands (tracked as a follow-up issue covering both pack-signing and definition-signing tooling), use the raw openssl recipe directly:

    # One-time: generate a keypair (rotate periodically per your policy).
openssl genpkey -algorithm Ed25519 -out yuzu-signing.pem
openssl pkey -in yuzu-signing.pem -pubout -outform DER \
    | tail -c 32 | xxd -p -c 64    # → 64-hex publicKey

# Per definition: sign yaml_source bytes, hex-encode the output.
echo -n "$YAML_SOURCE" \
    | openssl pkeyutl -sign -inkey yuzu-signing.pem -rawin \
    | xxd -p -c 128                # → 128-hex signature

    Inject signature and publicKey as top-level string fields in the JSON envelope POSTed to /api/instructions/import. See the REST API reference (docs/user-manual/rest-api.md → POST /api/instructions/import) for the full signing-rules table and per-rejection error strings.

  2. Opt out temporarily (legacy environments only). Pass --allow-unsigned-definitions to yuzu-server or set YUZU_ALLOW_UNSIGNED_DEFINITIONS=1 in the service environment. The server emits an InstructionStore: signature enforcement DISABLED by configuration warning on every start AND a server.unsigned_definitions_allowed audit row at boot, so the relaxed posture is recoverable from both operator logs and the audit store. Remove the flag as soon as the signing migration completes.

Pre-existing imported definitions are unaffected. The gate fires only on the public import path. The bundled-content boot seed (the kBundledDefinitions baked into yuzu-server at build time) routes through an internal import_definition_json_trusted variant that bypasses the gate; its authenticity comes from binary linkage, not runtime signature.

Authoring surfaces are NOT gated. POST /api/instructions, POST /api/instructions/yaml, and PUT /api/instructions/{id} — the dashboard and CLI surfaces where operators author definitions in- session — continue to trust InstructionDefinition:Write as the author trust boundary (the operator IS the source; there is no supply chain to authenticate). The --allow-unsigned-definitions flag does NOT affect those surfaces; they have always accepted unsigned author-time input and continue to. The architectural question of whether authoring surfaces should ALSO require signed envelopes is tracked as a follow-up issue with operator-decision- required framing (UX trade-off: gating authoring would break in- browser definition authoring).

Audit-trail evidence chain. Every rejection emits an instruction.import / denied audit row with the store error string in detail (stable SIEM-keyable tokens listed in the audit-log.md reference). If the audit-store write itself fails (locked DB, disk full), the response carries Sec-Audit-Failed: true header AND audit_emitted: false in the JSON body, surfacing the SOC 2 CC7.2 evidence gap to the operator immediately rather than silently dropping the event.

Product pack signature enforcement now on-by-default (#802 / W7.4) — BREAKING

The ProductPackStore previously shipped with signature enforcement disabled by default and the setter to enable it was never wired to any operator-facing flag — the protection was effectively unreachable. After upgrade, calls to install a ProductPack without a signature: field are rejected with:

pack '<name>' is unsigned and signature enforcement is enabled (set --allow-unsigned-packs / YUZU_ALLOW_UNSIGNED_PACKS=1 to bypass)

This is intentional. Unsigned packs are a fleet-wide arbitrary-code- execution surface: any operator with Pack:Install permission, or a MITM on pack delivery, could install a pack containing InstructionDefinition or plugin payloads that would then execute on every enrolled agent.

Two migration paths, in order of preference:

  1. Sign your packs. Generate an Ed25519 keypair, sign each pack’s non-metadata YAML content with the private key, and add signature: <hex> + publicKey: <hex> fields to each pack’s ProductPack metadata document. The existing verify path (ProductPackStore::verify_signature) accepts the result. Pack install then succeeds and the verified column in the store is set to true so a future “show only verified packs” query has the data it needs.

  2. Opt out temporarily (legacy environments only). Pass --allow-unsigned-packs to yuzu-server or set YUZU_ALLOW_UNSIGNED_PACKS=1 in the service environment. The server emits a [SECURITY] product pack signature enforcement DISABLED by configuration warning on every start and writes a server.unsigned_packs_allowed audit row, so the relaxed posture is recoverable from both operator logs and the audit store. Remove the flag as soon as the pack-signing migration completes; it is not intended as a permanent configuration.

Pre-existing installed packs are unaffected. The check fires only on the install path (POST /api/product-packs). List, get, and uninstall paths do not re-verify, so already-installed unsigned packs remain queryable and uninstallable after upgrade.

vNEXT — Access-review CSV export gains a leading metadata line (breaking for fixed-column-index CSV consumers)

GET /api/v1/access-reviews/export?format=csv (SOC 2 CC6.2 evidence) now emits one new line before the existing header row:

# rbac_enforcement=enabled
principal_type,principal_id,display_name,owner_or_email,roles,effective_permission_count,last_activity_ms,last_activity_kind,classification,lifecycle_state,source
user,alice,...

This line is unconditional — present even when the grant population is empty — so it stamps whether RBAC was actually enforced when the file was pulled, travelling with the retained/offline copy an auditor keeps (see rest-api.md’s rbac_enforcement section for what the three values mean).

This breaks any consumer that assumes row 1 is the header. Verified empirically against this exact output shape (not a blanket claim about “CSV tools” — the actual split matters):

The fix is the same for every consumer class: skip exactly one line before treating the next line as the header, verified working against each tool above:

# csv.DictReader
with open(path, newline="") as f:
    next(f)                      # skip the metadata line
    reader = csv.DictReader(f)   # now reads the real header correctly
# awk (was NR>1 under the old format; now NR>2)
awk -F, 'NR>2' access-review.csv
# pandas
df = pd.read_csv(path, skiprows=1)

A consumer that already treats the file as free-form text and looks for the # rbac_enforcement= prefix, or that reads the rbac_enforcement field from the sibling JSON export (GET .../export without ?format=csv, or the frozen campaign row’s rbac_enforcement field), is unaffected either way.

Executions-history PR 2 — responses.execution_id exact correlation

PR 2 of the executions-history ladder closes a forensic-data correctness gap (UP-8) where two concurrent executions of the same definition to overlapping agent sets could show each other’s responses in the Instructions → Executions detail drawer. Operators upgrading to a release that includes PR 2 should expect the following:

Schema migration v2 on response_store. The migration adds an execution_id TEXT NOT NULL DEFAULT '' column to the responses table plus a partial index idx_resp_execution_ts ON responses(execution_id, timestamp) WHERE execution_id != ''. The migration is automatic and data-preserving — MigrationRunner::run wraps the ALTER + CREATE INDEX in a single transaction. Pre-upgrade rows are NOT deleted; they receive the empty-string sentinel value ''. The migration is idempotent: a pre-stamp probe at startup detects an already-altered DB and skips the duplicate ALTER (mirrors the precedent at instruction_store.cpp v2).

Observable after upgrade:

SELECT execution_id, COUNT(*) FROM responses GROUP BY execution_id;

will show ALL pre-upgrade rows under the empty string ''. This is expected — there’s no way to retroactively attribute pre-upgrade responses to executions because the dispatch path didn’t record the linkage. The Executions drawer detects empty-execution_id rows and falls back to the legacy timestamp-window-plus-agent-set join so they render correctly without operator action. No operator action required for the migration itself.

Dispatch-path coverage gap (PR 2.x follow-ups). Only executions dispatched via POST /api/instructions/:id/execute (the dashboard’s Execute Instruction form goes through this path) get exact correlation in PR 2. Three dispatch surfaces continue to write rows with execution_id='' until follow-up PRs close them:

For runs from these surfaces, the drawer’s responses section uses the legacy timestamp-window join. Cross-execution contamination (UP-8) is still possible if two such runs overlap on the same definition + agent set. Track via the executions-history follow-up issues.

Mixed-mode detail drawer behaviour. During an upgrade transition window (executions in flight at restart time, or in-progress executions that started pre-upgrade and finished post-upgrade), some responses for a single execution may carry execution_id='' while others are correctly tagged. The drawer prefers exact-correlation rows when they exist and only falls back to the legacy join when zero exact rows are returned — this means mixed-mode runs may show only the post-upgrade subset of their responses in the drawer. Pre-upgrade responses for those runs remain in the database and are queryable via SQL; the upcoming admin backfill CLI (PR 2.1) will stamp them with their correct execution_id once it ships.

Server restart caveat. The dispatch-time command_id → execution_id mapping is held in memory inside AgentServiceImpl. If the server is restarted while a command is in flight, the mapping is lost and any agent responses arriving post-restart will be tagged execution_id='' and use the legacy fallback. Avoid restarting the server during active executions where possible, or accept that the affected runs will use legacy correlation.

Admin backfill (planned in PR 2.1, not in this release). A yuzu-server admin backfill-responses CLI is filed as a follow-up. It will walk the executions table cross-store and stamp pre-upgrade responses with their best-effort execution_id (timestamp + agent set heuristic). Until it ships, pre-upgrade rows remain queryable via the legacy fallback in the drawer; no operator action is required.

Verifying the migration. After upgrade, the server log should contain MigrationRunner: response_store migrated to v2. To verify the column directly:

sqlite3 /var/lib/yuzu/responses.db ".schema responses"
# Expected: execution_id TEXT NOT NULL DEFAULT ''
sqlite3 /var/lib/yuzu/responses.db ".schema idx_resp_execution_ts"
# Expected: CREATE INDEX idx_resp_execution_ts ON responses(...) WHERE execution_id != ''

v0.12.0 — AuthDB persistent authentication (#618)

v0.12.0 replaces the in-memory + on-config-flush authentication model with a SQLite-backed auth.db that holds user accounts, sessions, and enrollment tokens.

First boot after upgrade:

role parameter ignored on POST /api/settings/users. New users are always created as user (security finding C1). To assign or change a role, use the dedicated POST /api/settings/users/{username}/role endpoint introduced in v0.12.0 (see REST API → Settings → User Management). The dashboard exposes a Change Role button on each user row that calls the new endpoint.

Live drawer updates via SSE. The Executions tab opens an EventSource to /sse/executions/{id} for in-progress runs. Reverse proxies in the request path must NOT buffer SSE — the server emits X-Accel-Buffering: no and Cache-Control: no-cache but a poorly- configured proxy can still chunk the stream. Verify with curl -N https://<server>/sse/executions/<id> from inside your network if the drawer freezes mid-execution.

LimitNOFILE=65536 on systemd units. v0.12.0 raises the systemd file-descriptor limit from the default 1024 to 65536. SSE connections + agent gRPC streams + SQLite WAL handles can saturate the default soft limit on busy fleets. The new value matches Docker compose’s ulimits.nofile so containerised and bare-metal deployments behave identically.

Restart-loop guard. StartLimitIntervalSec=60 + StartLimitBurst=3 in the systemd [Unit] section so a corrupt auth.db failing the integrity check at startup puts the unit cleanly into failed instead of spinning. Recovery procedure: docs/ops-runbooks/auth-db-recovery.md.

Rollback to a pre-AuthDB release (only if you have not yet written user state via the new dashboard):

  1. Stop the v0.12.0 server.
  2. Move auth.db aside (mv auth.db auth.db.v0.12.0).
  3. Restore yuzu-server.cfg from your pre-upgrade backup.
  4. Reinstall the prior release binary.
  5. Start the server. It reads yuzu-server.cfg as before.

If you HAVE written user state via v0.12.0’s dashboard, that state is in auth.db only — rolling back loses it. Export users via GET /api/v1/settings/users first if rollback is required.

v0.12.0 — Guardian PR 2

Guardian PR 2 ships the Guaranteed State control plane + agent skeleton. Two items require operator awareness on upgrade:

Stale *:Push RBAC grants on deployments that ran pre-hardening Guardian PR 2 code. Between commits 7c83911 and 1f39401, the RBAC seed granted the Push operation to Administrator and ITServiceOwner on every securable type, not just GuaranteedState. The H-4 fix (commit 21c0ba4, hardening round 2) restricted the seed to GuaranteedState only going forward — but because seed_defaults() uses INSERT OR IGNORE, the stale cross-type grants already written to role_permissions are not removed on upgrade. These grants are semantically inert today (only the Guardian REST handlers consult Push), but become a latent privilege for any future release that adds a non-Guardian handler checking perm_fn(..., "Push").

Manual remediation until the auto-migration in issue #485 lands. Run each step in order:

  1. Back up the RBAC database first. Destructive SQL with no rollback.

    docker exec yuzu-server cp /var/lib/yuzu/rbac.db /var/lib/yuzu/rbac.db.bak.$(date +%Y%m%d)
# or for a systemd install:
cp /var/lib/yuzu/rbac.db /var/lib/yuzu/rbac.db.bak.$(date +%Y%m%d)

  2. Preview the rows that will be deleted. This should return only Administrator and ITServiceOwner rows on a fresh Guardian PR 2 upgrade. If it returns rows for any other principal_id, you have custom RBAC grants that the bulk DELETE below would silently wipe — in that case stop and prune by hand.

    docker exec -i yuzu-server sqlite3 /var/lib/yuzu/rbac.db \
  "SELECT principal_id, securable_type FROM role_permissions \
    WHERE operation = 'Push' AND securable_type != 'GuaranteedState' \
    ORDER BY principal_id, securable_type;"

  3. Delete the stale grants. Scoped to the two seeded roles so custom grants are left alone:

    docker exec -i yuzu-server sqlite3 /var/lib/yuzu/rbac.db \
  "DELETE FROM role_permissions \
    WHERE operation = 'Push' \
      AND securable_type != 'GuaranteedState' \
      AND principal_id IN ('Administrator', 'ITServiceOwner');"

  4. Confirm cleanup: re-run the preview query from step 2 — zero rows expected.

Safe on fresh installs (no matching rows). If you are upgrading from a v0.11.x release directly to v0.12.0 or later, skip this entire sub-section — your RBAC database never carried the stale grants.

Retention changes do not take effect on a runtime PUT, nor on a restart (set the flag or environment variable). BL-2 wired --guardian-event-retention-days (default 30) through RuntimeConfigStore + PUT /api/config/guardian_event_retention_days, matching the existing response_retention_days and audit_retention_days pattern. However, all three retention-bearing stores (AuditStore, ResponseStore, GuaranteedStateStore) capture their retention value at construction time and never re-read it — the runtime PUT mutates cfg_ and RuntimeConfigStore but the running reaper continues using the startup value. An operator who PUTs a new retention value sees 200 {"applied": true} but the store behaviour does not change, and a restart does not change it either: the stores copy cfg_ when they are constructed, before stored overrides are applied, so the enforced window comes only from the --*-retention-days flag or its environment variable (corrected later; see the ‘administrator-only’ Breaking note above). This is a systemic limitation shared across all three stores, not a Guardian-specific bug; it is tracked as issue #483.

Runtime config PUT now rejects non-numeric and negative integer values with HTTP 400. Hardening round 4 (UP-R5) added std::from_chars validation to PUT /api/config/<key> for heartbeat_timeout, response_retention_days, audit_retention_days, and guardian_event_retention_days. The previous handler silently wrote invalid strings to RuntimeConfigStore and swallowed the stoi error, leaving cfg_ unchanged. If your automation relied on setting retention to a negative value (e.g., "-1") to disable retention — which the store then treated as “never reap” via the <= 0 sentinel — that automation will now receive 400 {"error":{"code":400,"message":"value must be a non-negative integer"}}. Use "0" instead only for guardian_event_retention_days, where it preserves the disable-retention semantic and passes validation. For heartbeat_timeout, response_retention_days and audit_retention_days the store applies its own > 0 check after the route’s, so "0" is also rejected - with a different, bare-shaped body, {"error":"value must be a positive integer"}. There is no supported way to disable retention for those three through this endpoint. Automation that previously set non-numeric strings (anything other than a base-10 integer) was silently a no-op before this release — the 400 now surfaces the configuration error that had been hidden.

v0.12.0 — A3 UX ladder (#620, #622, #624)

Three operator-visible behaviour changes ship in the v0.12.0 A3 ladder. None require code changes on the operator side, but two of them require action if you maintain a local compose override:

1. Container healthchecks now pass (#622). The shipped docker-compose.uat.yml healthcheck blocks were updated to use tools available in each runtime image (bash + /dev/tcp for the server; busybox wget --spider for the gateway). After upgrade, docker compose ps reports (healthy) instead of (unhealthy).

If you maintain a local copy of the compose file (e.g. docker-compose.local.yml or a pinned vendored copy), your override still uses the broken pre-fix healthcheck pattern and will continue showing (unhealthy) until you sync the change. Replace your server-service healthcheck stanza with:

    healthcheck:
      test:
        - "CMD"
        - "bash"
        - "-c"
        - "exec 3<>/dev/tcp/localhost/8080 && printf 'GET /livez HTTP/1.0\\r\\nHost: localhost\\r\\n\\r\\n' >&3 && grep -q '200 OK' <&3 ; rc=$? ; exec 3>&- ; exit $rc"

And the gateway-service healthcheck stanza with:

    healthcheck:
      test: ["CMD", "wget", "--spider", "-q", "http://localhost:8081/healthz"]

2. /api/health is restored as an alias of /health (#620). The pre-#401 endpoint path is back. Monitoring integrations that point at /api/health work without reconfiguration; both URLs serve identical JSON. Both are exempt from rate limiting (a follow-up hardening over the bare /health behaviour; since superseded: credentialed requests to these two paths are now rate limited per source). For load-balancer probes that should drain in-flight traffic before stopping, continue using /readyz. /health and /api/health are intentionally not draining-aware (Kubernetes pattern: liveness/health probes are not draining-aware).

3. File-logger boot messages are now quieter (#624). The previous WARN: Could not create log directory /var/log/yuzu + ERROR: file logger setup failed pair on every container boot is replaced by a single INFO-level line when the default path cannot be created. The Docker server image now pre-creates /var/log/yuzu (mode 0750, owned by yuzu) so the path is writable out of the box. If your monitoring previously alerted on the WARN/ERROR lines as a misconfig signal, those signals will no longer fire — the failure mode is now a single INFO line. Operators who require explicit on-disk logs should pass --log-file <path>; explicit-path failures still log at ERROR and are not silently degraded.

quarantine plugin: reporting is now more conservative (Wave-2 argv migration)

The quarantine plugin’s Windows/Linux/macOS netsh/iptables/pfctl invocations were migrated off shell-outs onto a bounded argv runner. Alongside the mechanism change, two reporting bugs were fixed: a Linux chain-flush could previously be miscounted as an applied containment rule (status|quarantined|rules_applied|1 reported with zero rules actually installed), and macOS whitelist mutations could silently proceed on a failed prerequisite read or report success despite pfctl -e failing to enable pf.

If your automation alerts on status|failed/status|release_uncertain/status|update_uncertain from quarantine actions, you may see these fire more often after upgrading — this is expected. Nothing got more broken; the plugin’s failure reporting simply got more honest about cases it previously reported as clean success. No action is required beyond expecting the change.

Fresh deployment only for 0.15.0. See the banner at the top of this page: the steps below are not a supported upgrade path from an earlier build’s database.

Rollback

If an upgrade causes issues:

  1. Stop the new version
  2. Restore the backed-up .db and .cfg files
  3. Start the previous version binary

Important: Rolling back after a schema migration requires restoring the database files from backup. The old binary cannot read the new schema.

Troubleshooting

SymptomCauseFix
Server won’t start after upgradeSchema migration failureCheck logs, restore backup, report issue
Agents can’t connectProtocol version mismatchEnsure server was upgraded first
Dashboard shows errorsBrowser cacheClear browser cache, hard refresh
Gateway disconnectsVersion mismatch with serverUpgrade gateway to match server version
Slow startupLarge database migrationNormal for first start; wait for migration to complete