Guaranteed State

Guardian is Yuzu’s real-time policy enforcement engine. A guaranteed-state rule is a desired-state assertion about an endpoint — a registry value, a service status, a file presence, a configuration key — plus optional automatic remediation when the endpoint drifts. Unlike the server-side Policy Engine, Guardian runs inside the agent, reacts to kernel-backed change notifications in sub-second time, and can repair drift without an operator in the loop.

Status — read this first. Guardian’s control plane, the agent guard host, and the Windows registry, file, and service (run-state) guards are live and verified end-to-end. Rules are authored/stored/pushed via the REST API; a push is delivered to in-scope agents over both the direct gRPC stream and the Erlang gateway, the agent arms the guard, and drift is detected sub-second (agent_id is server/gateway-asserted, never self-reported). The registry guard runs in two modes: enforce — on drift the agent writes the expected value back via a single in-process RegSetValueExW syscall (tens of µs; emits drift.remediated, or remediation.failed if the write is denied) — and audit — detect and report only, no write-back. A restarted agent re-arms enforce guards from its local cache and enforces pre-network (heal-on-restart; a rule whose boot re-arm fails is caught up by a server re-push after connect, see limit (3)); drift in that pre-network window is enforced but not reported until reconnect (durable event buffering is Guardian A3).

Boundaries you must know before enabling enforce: (1) enforce is HKCU-demo-grade today — HKLM writes need the agent service account to hold write access; a non-writable key degrades to a read-only watch that reports remediation.failed, it does not silently disarm. (2) The rule signature is not yet verified before a guard arms, so the integrity gate on what gets written where is Push RBAC + mTLS, not rule signing (deferred — contract G3). (3) Agent status is fail-closed: an armed guard reports errored/not-healthy, never a green “compliant”, until the self-test lands — do not read the agent status proto as proof of enforcement. (4) enforcement_mode defaults to enforce on rule create — see the upgrade note below. Still on the roadmap: non-Windows guards, server-side Prometheus metrics, fight-loop rate-limiting, rule signing, and durable event buffering (A3). Gateway drift forwarding is best-effort (yuzu_gw_guardian_forward_dropped_total). Design: docs/yuzu-guardian-design-v1.1.md. Rollout plan: docs/yuzu-guardian-windows-implementation-plan.md.

Concepts

Permissions

Guardian introduces a new securable type (GuaranteedState) and a new operation (Push). Seeded role grants:

Administrator and ITServiceOwner inherit Execute and Approve on GuaranteedState from the cross-type CRUD seed in rbac_store.cpp — neither operation has an active handler on Guardian today, but the grants exist in the database and will surface in any role-permission export. PlatformEngineer’s grants are explicit and narrower (Read/Write/Delete/Push only). Push is deliberately separated from Write so an operator can deploy a rule set authored by a platform engineer without being able to modify the rule body. See RBAC for the full securable-types matrix.

Rule YAML schema

Guardian rules use the DSL framework documented in docs/yaml-dsl-spec.md:

apiVersion: yuzu.io/v1alpha1
kind: GuaranteedStateRule
metadata:
  id: block-rdp-outbound
  name: "Block outbound RDP"
  severity: high
spec:
  os_target: windows
  enforcement_mode: enforce
  scope: "tag:env=prod"
  guard:
    type: windows_registry
    hive: HKLM
    path: "SYSTEM\\CurrentControlSet\\Services\\TermService"
    value_name: Start
    expected: 4   # Disabled
  remediation:
    action: registry_set
    hive: HKLM
    path: "SYSTEM\\CurrentControlSet\\Services\\TermService"
    value_name: Start
    value: 4
    value_type: REG_DWORD

Three guard types ship today: the registry guard (above, Windows-only), the file guard (below, Windows-only), and the service guard (below, Windows + Linux/systemd). On Linux the service guard is observe-only — it detects and reports drift but does not start/stop units; an enforce-mode rule on a Linux agent degrades to observe (drift is reported, never remediated), with no error. macOS (Endpoint Security / FSEvents), the remaining Linux event guards (inotify/netlink), and non-systemd Linux fallback are on the roadmap.

File guards — detect a file changed or deleted, in realtime

A file-change spark watches a file via ReadDirectoryChangesW on its parent directory — kernel-notified, no polling, so detection is realtime. The watch is resilient: it survives the directory holding the guarded file, or its nearest existing ancestor, being deleted and recreated anywhere in that ancestor chain. A rename or move of the watched directory itself is detected separately, via a watch on that directory’s own immediate parent — but nothing watches that parent’s own parent, so a rename or move of the parent itself, or of anything above it, is not detected in real time. Recovery in that case depends on the relocated directory’s own content next changing (not on any activity at the original path) — a clean substitution that recreates a new directory at the original path, while leaving the relocated original untouched, can go undetected indefinitely. A rename or move of the watched directory is evaluated like any other change to the guarded path: a guarded file that is no longer at its original path is reported as absent under a file-exists rule that expects it present and under a file-hash-equals rule, and changes made after the original path is recreated are detected. This behaviour is validated on local NTFS; it also functions over SMB in testing, via the plain name-match fallback used when the extended, FileId-bearing notify API is unavailable, but has not been validated against a production Windows/NAS file server.

Known degraded mode. The watch on the directory’s own parent is retained across ordinary re-arms and only rebuilt when the watched directory’s identity actually changes; if a rebuild’s teardown of the outgoing watch cannot be confirmed within a bounded wait (normally only a wedged filesystem driver, minifilter, or sustained packet loss on a network share), that watch generation is abandoned rather than reused. This condition has been reproduced in testing: holding a watch on an SMB share and inducing packet loss on that connection, a frozen SMB server or a fully severed connection both still confirmed cancellation well under the bound (both fail fast, locally), but sustained mid-path packet loss — a blackhole window on an otherwise-established connection, not a crash — reliably exceeded it, consistent with the client falling back to TCP’s own retransmission timeout with no faster local signal available. After 3 such abandons for one rule, this parent-level rename detection is permanently disabled for that rule — logged once, at error level, naming the rule and path ("parent-directory watch for {rule} ... permanently disabled after {n} drain failures") — until the rule is next re-armed (a policy re-push or an agent restart). Each level of a directory chain created one level at a time beneath the guarded path, and each replacement of the watched directory that is observed mid-change, performs one bounded teardown of the outgoing watch; where cancellation takes longer than the bound (sustained packet loss on a network share is the realistic case, per the measurement above), each one counts toward the limit, so it can be reached within a single such episode rather than only over three unrelated ones. This does not affect the guard’s primary detection (the file’s own presence or content, evaluated on every wake); only the specific ability to detect a rename or move of the watched directory itself is lost. On the transition into this state the rule reports itself unhealthy — the device page’s per-rule compliance census shows it errored (the same red state as an ordinary unresolved drift), with an operator-facing detail explaining the cause and the fix (“Rename/move detection for this rule is disabled after repeated failures to reset its file-system watch (often caused by a slow or unreliable network share). This rule’s file presence/content checks are unaffected. To restore rename detection: re-push this rule’s policy, or restart the agent.” — also retrievable via GET /api/v1/guaranteed-state/events), and the agent re-sends that report every 5 minutes for as long as the agent keeps running and the rule stays disabled, so a single lost report does not leave the census stuck showing an earlier, stale state (an agent crash or restart of this specific watch is a separate condition, corrected the same way any other agent restart is — a fresh policy sync). While disabled, a compliant evaluation of the file’s own presence/content is never published as guard.compliant — it is replaced by the same unhealthy report — so the rule cannot show 100% compliant while rename detection is dead; ordinary drift on the file’s own presence/content is still reported normally. To recover: re-push the rule’s policy (POST /api/v1/guaranteed-state/push), or re-deploy the Baseline it belongs to from the dashboard, or restart the agent — the rule’s first verdict after any of these clears the state.

Other known limitations. A forced dismount of the volume holding the watched directory, and a change to a mount point/reparse point along the watched path that does not itself rename any watched directory, are not guaranteed to be detected by the mechanisms above; both are pre-existing, architectural gaps distinct from the rename-detection behaviour described above.

Two assertions decide what counts as drift:

file-hash-equals uses a size pre-filter then a bounded SHA-256, and waits for the file to quiesce before hashing (a write is not atomic), so a no-op rewrite of identical bytes is not reported as drift — only a real content change is. Absent / oversize / unreadable states are reported (<absent> / <oversize> / <unreadable>), never a silent “compliant”. File guards are detection-only (file-content remediation is deferred); enforcement_mode still gates whether the rule is active, but there is no write-back.

A baseline captured on arm (empty expected_hash) is persisted per rule on the agent and re-asserted on every later arm — a fleet-wide policy change elsewhere, or an agent restart, does not re-capture the target’s then-current content as a new baseline (#4021 for the legacy detection path, #4045 for the Spark path). The original baseline holds until the rule’s path is deliberately changed (which is treated as a genuinely new target). Authoring an explicit expected_hash shadows the persisted baseline rather than replacing it — the rule compares against the authored value while it’s set, but the original captured baseline is neither deleted nor recomputed; removing the authored expected_hash later restores that original persisted baseline, not a fresh capture of whatever the target holds at that point. If the agent’s local KV store is unreadable at the moment a rule arms, that single arm falls back to capturing current content as if no baseline existed (logged as an error). The persisted record itself is protected against that fresh capture overwriting it if the store is readable again by the time the capture completes — the agent re-checks before writing and refuses to replace an existing record for the same target. If the read of the record fails at arm time AND the re-check read fails again when the capture is written, the fresh capture is written anyway (also logged), since the alternative would leave the rule permanently without a persisted baseline. That takes two failed reads (the arm-time read and the re-check just before the write), after which the write itself succeeds. A brief read fault can cause that, not only a sustained outage, so a local KV fault can still record a new baseline over a genuinely drifted one.

Spark path (#4045). SparkEngine (the prefer_spark backend, off in every released agent today) writes the same per-rule record by a different route: the detection runtime only notes the capture in memory, and the agent’s own persistence step then writes it to the local KV store. That step runs on the agent’s background delivery worker, which does not depend on the server connection (so a baseline captured while the agent boots before the network is up is saved too), before a later policy push tears down and re-arms the rule, and once more when the agent stops. It is driven by the capture’s own wake-up rather than a clock: it typically lands within milliseconds (except during a policy push, where it waits for the push to finish plus up to a 200 ms re-check), a capture that produced no wake-up waits for the worker’s 5-second backstop, and after a failed pass the worker waits 5 seconds, doubling up to 60, before trying again (a policy push does not wait for that delay, and neither does the flush at shutdown, which is skipped in the cases listed under “What can still go wrong”). Each pass is bounded: it starts at most 64 writes, gives up after 3 failed writes, and starts no new write after 2 seconds (a write already in flight is not interrupted, so a pass can run past 2 seconds by that one write). At shutdown the flush has no cap on the number of writes and allows 1 failure and 1 second between writes; a write costs about a third of a millisecond on a healthy store, so it writes roughly two to three thousand waiting captures before the second runs out. A large backlog therefore drains over several back-to-back passes, and one failing capture cannot hold up the ones behind it, although one capture that fails every time keeps the worker at its 60-second retry delay, so a capture noted after that waits up to about 65 seconds. The background worker also steps aside for a policy push: it does not start a pass while a push is being applied and ends its pass after the current write when a push is waiting for it, so a push is not held up by the worker’s backlog. There is no time bound while the write keeps failing. A capture that is still waiting to be written when a new arm seeds (for example one that an in-flight evaluation of the old arm records just after that push began) is handed to the new arm directly from memory, so the new arm is seeded with the captured baseline either way. The rule enforces against its in-memory baseline throughout. A failed write is a deliberate fail-open, never silent: the capture stays waiting and is retried, the failure is logged as an error, and it is counted in the yuzu.guardian_baseline_persist_failures heartbeat tag (see Metrics). A write the overwrite guard refuses because a record for the same target already exists is logged as a warning and is not a failure.

Where the baseline lives. In the agent’s local KV store, kv_store.db in the agent’s data directory, namespace __guardian__, key baseline:<rule_id>. The record is a schema number, a fingerprint (the text file-hash-equals|<authored path>) and the SHA-256 of the file’s content; no file content is stored. It is not signed or MAC’d. __guardian__ is a reserved plugin name: a plugin cannot claim that namespace through the plugin KV API, but the reservation is hygiene, not isolation (native plugins run inside the agent process as the agent account and can open the SQLite file directly). Changing or deleting a record re-baselines the rule. Two actors can change a baseline. A local actor who can write the agent data directory (--data-dir) can change or delete the record: that boundary is the permissions on the directory, and what was verified is narrower than “locked down”, below. A control-plane actor who can push a rule can re-baseline it without touching the endpoint, by changing the rule’s path or authoring an expected_hash; the server audits the rule create, update and push calls (audit verbs guaranteed_state.rule.create, guaranteed_state.rule.update and guaranteed_state.push, see Audit log), those rows record who acted and when; the create and update rows name the rule (the REST and MCP rows carry the rule id and its name, the dashboard create row adds trigger, mode and action) and the push row records the rule count, the scope and the number of agents pushed to, not which rules; none of them records the path or expected_hash, so the audit trail shows that a rule was changed and not what the change was. The file-hash-equals guard runs only on Windows under both detection backends, so a baseline record exists on Windows agents only. There the service is registered with --data-dir C:\ProgramData\Yuzu, and the installer’s only grant on that directory is admins-full system-full (yuzu-agent.iss, [Dirs]); that entry adds permissions and cannot disable inheritance, so the directory also keeps what C:\ProgramData grants by inheritance, which the installer’s own comments say includes create rights for ordinary users. The installer removes inheritance only on the agent-certs subdirectory. The Windows data directory is therefore not shown to be restricted to administrators, and the file mode of kv_store.db is not set by the agent (it follows the process umask). On Linux, where no file-hash guard runs, the rpm spec and the deb post-install create /var/lib/yuzu-agent as 0750 owned by the service account, but the shipped systemd unit declares StateDirectory=yuzu-agent without StateDirectoryMode=, so while the packaged service runs systemd applies its own default mode (0755, per systemd.exec(5)); this page does not rely on 0750 there. There is no reset action: to accept an intended change to a monitored file, change the rule’s path to a new target (changing it back later re-baselines from the current content) or author an explicit expected_hash. An authored expected_hash only shadows the old baseline: removing it later restores the OLD persisted baseline, not the file’s current content, so an operator who authored a hash to accept a change should keep it authored. Baselines are not tied to the detection backend (a backend flip reads the same record, so does a downgrade to an agent build that has #4021, and an upgrade needs no step); only the fingerprint (rule type and path) decides whether a record applies.

What can still go wrong (the known cases, eight). (1) A crash or hard kill between a capture and its write loses that capture, and the next start captures whatever the file holds then. The window is wider than on the legacy path, which writes at the moment of capture. It includes captures that a policy push’s own pass left unwritten because it ran out of its budget: the new arm is handed them from memory and the worker writes them next, but a crash before that loses them. (2) A stop flushes first, and also stages any capture the agent could not note (case 3). On a healthy store it writes every waiting capture that fits in its one second between writes, roughly two to three thousand. It can still lose waiting captures in five ways: the flush’s single allowed write fails (added to the tag, but the process is exiting, so the count is lost with it and the capture is not tried again, not even by the second stop call the exit teardown makes after the store has recovered); the flush’s one second between writes runs out before every capture is written (a slow store, or more than roughly two to three thousand captures waiting on a healthy one; logged, not counted); the flush is skipped, because the earlier shutdown steps had already used 18 of the 20 seconds the agent allows itself and that same shutdown had shown a slow store: either one of those steps had run slow, taking at least 0.75 seconds (the 1-second shutdown busy budget minus a quarter-second measurement slack), or a baseline write by the delivery worker had failed slowly during that shutdown (logged, not counted; a step that failed quickly is not such a sign, a slow failure with an earlier start does not skip the flush, and a late start with no slow sign still runs the flush, because on a healthy store it takes well under a millisecond per capture, so a shutdown that waited behind a long policy push does not lose its captures; a flush that late can use its whole one second plus one write, so one that starts in the last second can run into the watchdog); an allocation or lock failure cuts the flush, or the staging of captures the agent could not note, short (logged; any count dies with the exiting process); or the agent’s 20-second shutdown watchdog ends the process while the stop is still running, which cuts the flush and the remaining shutdown steps and makes a Windows service restart (exit code 4; nothing is logged and nothing is counted). Against a store that another process holds locked, each lock wait in a shutdown step that touches it normally lasts about the shutdown busy budget: when a stop begins the agent lowers the store’s busy wait from 5 seconds to 1 second for the rest of the process (every other user of the shared store runs at 1 second too: plugin actions and a plugin’s own shutdown-time writes, the sync scheduler and the license scan), and the lowering also cuts a write that is already waiting. The target is at most seven such waits in series (7 seconds): six enumerated waits (the two lifecycle-journal flushes, the loss-ledger write, the final baseline flush, which can make two writes, a slow success followed by a failing one because the flush allows one failure, and a delivery-worker write already in flight) plus one second of allowance for retry-sleep overshoot and a queued caller. Every other single-shot writer to the store that is already queued when the shutdown begins adds about one more second (measured on Linux, one run per point: 9.0 seconds with three extra writers and 11.0 with five, with a lifecycle record pending and a worker write in flight; extrapolated, not run: about 20 seconds, the watchdog, at about fourteen), at one extra writer the measurement is about 7.0 to 7.1 seconds in the latest quiet reruns, which meets the 7-second target only to within sleep overshoot (the cause is likely retry-sleep overshoot, not isolated); earlier runs under heavy load gave 7.12 and 8.14 seconds, so the target is not met under load at one extra writer. The one second of allowance is spent twice if a delivery-worker pass in flight makes two failed writes and a writer is also queued. A writer that loops costs more than one second: in a standalone run of six serial one-second writes, other threads looping a write with a 50 millisecond pause took 8.7 to 9.8 seconds with one and 20.8, 20.8 and 23.8 seconds with three over repeat runs, against 6.0 with none (the cost is not linear in the number of loopers because the store’s lock is not fair, and three loopers pass the 20-second grace), and a writer that loops with no pause at all can starve a shutdown step indefinitely because the store’s lock is not fair (with one such writer a step was still unserved after more than three minutes); no production caller that does this was identified. All of this predates the change and is not fixed by it. Measured on Linux with every write truly busy, at the production values (one second budget, with the final busy handler; one run per point, except the lifecycle-record rows: the emulated one was run twice and agreed within 3 milliseconds, the real-write one 16 times, below): a shutdown with captures waiting and nothing else pending took about 2.0 seconds; one with a lifecycle record also pending took 4.0, 5.0 or 6.0 seconds (16 runs, with 3 or 5 captures waiting: 4.0 seconds seven times, 5.0 six times, 6.0 three times; which of the delivery worker’s own writes line up with the stop decides how many of the waits occur); one that found the delivery worker in the middle of writing a capture took 2.0 to 2.7 seconds across four timings; one that found it in the middle of another write (emulated by a write issued from the worker thread) took 2.0 to 2.7 seconds, and 5.3 to 6.0 seconds when a lifecycle record was also pending (6.03 seconds, the largest figure measured, on both that emulated row and the real-write row above). A slow failed worker write used to skip the final flush even when the store had healed a moment later; it no longer does, which costs one or two more seconds when the store is still locked. Before this budget the same shutdowns took 21 to 29 seconds on earlier revisions of this code (not re-run on the final code) and could pass the 20-second watchdog. Trade-off: a different process that holds the store locked for between 1 and 5 seconds at the instant of shutdown now makes the shutdown-time writes fail where a 5-second wait would have outlasted it; the cost is one recapture of the affected baselines at the next start (any drift between a lost capture and that start is accepted as the baseline, case 1), a loss-ledger record that is stale but never corrupt, and, once Spark detection is enabled, lifecycle-journal records (audit evidence) that were still pending when the last shutdown flush failed: they are held only in memory and are lost when the process exits (logged once, as lifecycle-journal record(s) were still unpersisted after the stop-time flush, naming how many; the records themselves are not recoverable). The loss of those journal records is a condition to resolve before Spark detection is enabled by default (tracked in the Spark flip-gate) and has not been accepted. The budget bounds only waits for the lock: a stalled disk write is not such a wait, so it can still run into the watchdog, as can the DEX observer’s shutdown and a plugin’s own shutdown step, which this budget does not cover. SQLite’s own internal retries (for example on its WAL index) are not covered by that cap: on Linux they were measured at up to about 2.3 seconds against a 1-second budget (phasing dependent, a contrived trigger), and the bound is per statement step, not per call, on every platform (a lock wait that continues within one statement step shares one budget). On Windows, SQLite also retries failing file operations itself (about 1.45 seconds each, derived from the SQLite source and not measured), outside the busy wait. A shutdown that is queued behind a policy push is not cut short by the lowered wait when that push is forced (up to two failed writes of one second each) or is stuck in a watcher; this predates the change. A supervisor that stops the agent sooner than 20 seconds ends it first: Docker agent services stop at 10 seconds, so exit code 4 never appears there. The check is a sample taken by the earlier shutdown steps: a store that turns busy only after them, or whose steps each stay under 0.75 seconds, when the shutdown is already 18 seconds late, still gets the flush attempted and the watchdog may end it (the capture is lost either way, with no log line; two such shapes were reproduced before the busy budget existed, at 21 and 20.1 seconds, and have not been re-run since). A slow failure that ended before the shutdown began is not a sign of a slow store for this shutdown. (3) The agent can fail to note a capture (an allocation failure, counted in the tag once per attempt). The rule keeps the captured baseline, so a later change is still reported as drift against it, and the agent retries noting it at every later evaluation, when the rule is re-pushed, when it is withdrawn and once more at shutdown, before the final flush; the capture is written once a retry succeeds. A retry at shutdown that also fails loses the capture (logged as committed Spark baseline capture(s) could not be staged for the final flush at stop, a count only). Until then it is not saved: a crash or hard kill in that window loses it (case 1). If the rule is re-pushed in that window, the new arm is handed the capture when a retry made at the re-push succeeds (a delta push makes two; a full_sync makes one for a rule it omits, and a rule it keeps and rebuilds is retried as a delta push is); when they all fail the new arm captures afresh and a file that changed in between is accepted as the baseline. The same happens if a retry succeeds but copying the staged hash afterwards fails: the saved record keeps the original hash while the live arm is judged against the fresh capture until its next re-arm. (4) A rule’s path re-authored while its first capture was still waiting to be written loses the old target’s capture (counted). Re-authoring a path from A to B and back to A also re-baselines A from its current content even with a healthy store, because there is one record per rule and a path change is a new target; the legacy path behaves the same way. (5) The two-failed-reads case in the paragraph above, which a transient read fault can trigger. (6) The refused window: if the read of the record fails at arm time while a valid record exists, the rule is judged against freshly captured content (which may already have drifted) and reports compliant until its next re-arm or an agent restart. The record itself stays intact, because the overwrite guard refuses the fresh write (logged as a warning, and counted in an agent-side diagnostic counter that no heartbeat tag or metric exports). (7) A record that is found malformed (not valid JSON, a schema number this agent build does not know, or an invalid hash) is discarded at arm and overwritten by the next capture. This predates #4045 (#4021) and applies to both detection paths. (8) A record for a rule that is later removed stays in the store, dormant; if the rule returns with the same path it is seeded from that old record. The tag covers the Spark path only: a legacy-path write failure is logged, not counted, so an absent tag is not evidence that baselines were saved, and no server gauge or alert exists for it yet, so the agent’s error log is the primary signal. A loss at shutdown leaves evidence only in the agent’s error log: the final Spark baseline flush at stop was incomplete, skipped the final Spark baseline flush at stop (two wordings, both for a flush that could not start in time, one after a slow persist pass and one after a slow shutdown step), committed Spark baseline capture(s) could not be staged for the final flush at stop, the final Spark baseline flush at stop was cut short by an allocation or lock failure, staging the committed Spark baseline captures for the final flush at stop failed and, for a write that threw, persisting a Spark baseline capture for rule '<id>' threw, and, for lifecycle-journal records, lifecycle-journal record(s) were still unpersisted after the stop-time flush. Every shutdown loss except the watchdog route leaves one of these lines (a capture waiting with no local store is logged once, at the first pass that finds it); the shutdown watchdog leaves no log line at all, and its only evidence is the agent’s exit code 4.

# Alert if a sensitive config is modified OR deleted, in realtime.
curl -X POST https://yuzu.example.com/api/v1/guaranteed-state/rules \
  -H "Authorization: Bearer $YUZU_TOKEN" -H "Content-Type: application/json" \
  -d '{
    "rule_id": "watch-hosts", "name": "hosts file integrity",
    "enforcement_mode": "audit",
    "spark":      {"type": "file-change", "params": {}},
    "assertion":  {"type": "file-hash-equals", "params": {"path": "C:\\Windows\\System32\\drivers\\etc\\hosts"}},
    "remediation":{"type": "alert-only", "params": {}}
  }'

The full type catalog — including these file types and the expected_hash format — is discoverable at GET /api/v1/guaranteed-state/schemas (see Schema discovery).

Service guards — keep a service running (or stopped), in realtime

A service-status-change spark watches one service in real time, no polling: on Windows via NotifyServiceStatusChange (kernel-notified by the Service Control Manager, ~0 ms); on Linux via the systemd PropertiesChanged D-Bus signal on the watched unit’s ActiveState (sd-bus, event-driven). The watch is resilient: it survives the service being deleted and recreated, re-arms automatically, and on Linux reconnects after a systemctl daemon-reexec / dbus restart. The assertion type encodes the desired run state; the only param is the service / unit name:

Linux (observe-only, v1). On a systemd host, service-running / service-stopped detect and report drift exactly as audit mode does; the guard does not start, stop, or mask units — enforcement is gated behind a forthcoming polkit-grant change. An enforcement_mode: enforce rule on a Linux agent still detects drift and emits drift.detected (with platform=linux); a drift.remediated event is never emitted on Linux today. The unit name follows systemctl ergonomics — a bare name defaults to .service (so ssh means ssh.service). On a non-systemd host (no system D-Bus) the guard does not arm and the rule reports unarmed rather than compliant. systemd’s richer ActiveState maps onto the two run-state tokens: active → running; inactive, failed, or an absent unit → stopped; activating / deactivating / reloading / maintenance are transitional and held (no event), mirroring the Windows guard’s *_PENDING hold.

Like every Guard, a service Guard reaches an agent — and therefore enforces — only as a member of a deployed Baseline; authoring one alone does nothing on any endpoint.

In enforce mode the guard drives the service back to its desired state via the Windows service-control API (StartService / ControlService — never sc.exe or net start), gated by the same per-rule resilience policy as registry guards. In audit mode it only detects and reports. This is the canonical way to keep a security service alive — Microsoft Defender must always be running; restart it if it stops — or held down — the Print Spooler must stay stopped. Transitional SCM states (START_PENDING, STOP_PENDING, …) are held: drift is reported only once the service settles into a terminal state, so a normal restart doesn’t trigger a spurious enforce loop.

# Keep the Print Spooler stopped (PrintNightmare hardening), enforcing.
curl -X POST https://yuzu.example.com/api/v1/guaranteed-state/rules \
  -H "Authorization: Bearer $YUZU_TOKEN" -H "Content-Type: application/json" \
  -d '{
    "rule_id": "spooler-stopped", "name": "Print Spooler held stopped",
    "enforcement_mode": "enforce",
    "spark":      {"type": "service-status-change", "params": {}},
    "assertion":  {"type": "service-stopped", "params": {"service_name": "Spooler"}},
    "remediation":{"type": "enforce", "params": {}}
  }'

Enforce-stop is denied for protected services. So that Guardian — a security-enforcement tool — cannot be turned into a security-control disabler (or be made to stop itself), the server rejects an enforce + service-stopped rule (HTTP 400, A4 error envelope) targeting these SCM key names, and the push backstop downgrades any that slip through to audit: WinDefend, WdNisSvc, Sense, WdFilter (Defender’s filesystem minifilter — stopping it blinds real-time scanning while WinDefend still reads as up), wscsvc, mpssvc, BFE (the Base Filtering Engine the firewall sits on — stopping it tears down the firewall while mpssvc looks protected), EventLog (security controls); RpcSs, DcomLaunch (critical infrastructure — stopping them strands the agent and the host); and YuzuAgent (the agent’s own service). These services can still be observed in audit mode. service-running enforce (keeping a service up) is always allowed. The set covers Windows built-in services + the agent — it does not cover third-party EDR/AV; protecting those is on the roadmap (an operator-extensible denylist).

service-disabled (start-type) has no enforce path yet. A service’s startup type fires no SCM notification, so it is not a native assertion. You can detect start-type drift with a registry guard in audit mode on HKLM\SYSTEM\CurrentControlSet\Services\<name>\Start (4 = disabled), but you cannot enforce it that way — …\Services\… is on the registry enforce denylist (it is a privilege/persistence key), so an enforce registry write there is rejected by design. To set a service’s start type, use Group Policy or your MDM; pair it with a service-stopped audit guard to alert if the service runs. Like file guards, service guards are authored via the REST API or the seed rig today; a dashboard authoring form follows.

Workflow

1. Author a rule

curl -X POST https://yuzu.example.com/api/v1/guaranteed-state/rules \
  -H "Authorization: Bearer $YUZU_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "rule_id": "block-rdp-outbound",
    "name": "Block outbound RDP",
    "yaml_source": "apiVersion: yuzu.io/v1alpha1\nkind: GuaranteedStateRule\n...",
    "severity": "high",
    "os_target": "windows",
    "scope_expr": "tag:env=prod"
  }'

Requires GuaranteedState:Write. Response: 201 Created with data.rule_id. Audits guaranteed_state.rule.create.

2. List or fetch rules

curl -H "Authorization: Bearer $YUZU_TOKEN" \
  https://yuzu.example.com/api/v1/guaranteed-state/rules

Requires GuaranteedState:Read.

3. Update a rule

curl -X PUT https://yuzu.example.com/api/v1/guaranteed-state/rules/block-rdp-outbound \
  -H "Authorization: Bearer $YUZU_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"enabled": false}'

Version is auto-incremented on every successful update. Requires GuaranteedState:Write. Audits guaranteed_state.rule.update.

4. Push to agents

curl -X POST https://yuzu.example.com/api/v1/guaranteed-state/push \
  -H "Authorization: Bearer $YUZU_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"scope": "tag:env=prod", "full_sync": true}'

Requires GuaranteedState:Push. The example uses an API token; a call authorised by a session cookie must instead carry an Origin or Referer header naming this site (or an entry from --csrf-trusted-origin) and is otherwise refused 403 cross-site request refused, audited as csrf.denied. Returns 202 Accepted. Audits guaranteed_state.push with result=success and detail=rules=<N> full_sync=<bool> scope="<expr>" agents=<count>, where agents is the number of in-scope agents the push was dispatched to (across the direct gRPC and gateway transports).

The push is dispatched to in-scope agents; the agents=<count> field records how many were targeted. Delivery is best-effort for gateway-connected agents under an upstream outage (durable buffering is Guardian A3), so a non-zero agents count records dispatch, not a per-agent delivery receipt. SIEM correlation should treat the count as “push fanned out to N agents,” and corroborate actual enforcement against the drift/sync records at /api/v1/guaranteed-state/events.

What a full push does on the agent. A push with full_sync=true replaces the active set, and the agent applies it per rule. On the Spark path (not enabled in production today, prefer_spark is off, see the Spark flip gate) a rule the push still carries that is healthy and unchanged is left alone: it is not torn down or re-armed, gets no new guard.armed or lifecycle record, and does not re-send guard.compliant; a rule the push omits is removed, and a rule that changed, is not yet healthy, is in an Unknown episode, sits on a faulted or dead watch (reported by the mechanism, or found so by the agent’s own health check when that report was lost), or whose baseline hash differs from the live one, is rebuilt (flip gate AC-18). Each full push that reaches the end of its rule loop, on both detection paths, logs one agent line naming the rules it left alone, Guardian: full_sync preserved N rule(s) (skipped re-arm) ids=[...] more=M (more counts ids beyond the 128-entry / 4096-byte list cap; N is 0 when every rule was built, and a legacy-path agent always logs preserved 0; the line is best-effort under allocation failure and is not logged when the push stops early on a rule-persist failure). A rule whose identical arm was still in flight is not counted as preserved: the line continues with pending_reused P ids=[...] more=Q naming it, because that arm has not confirmed yet. The legacy guards are still stopped and rebuilt wholesale. On both paths the agent’s stored copy of the rules is updated in place: only the rows for rules the push omits are deleted, and a row whose content is unchanged is not rewritten. The line that used to read full_sync cleared N prior rule(s) now reads full_sync removed N prior rule(s) absent from the push and is not logged when no stored row is removed. Accepted consequences: an identical re-push no longer repairs a lost recovery event or a server-side state loss (a Postgres restore or an outbox drop) or restarts the demotion clock of an initial observation; a crash after the omitted rows are removed and before the carried rules are rewritten leaves the previous rows for carried rules, i.e. OLD content, including a rule the push disabled or changed, until the next push (a restart then re-arms those; this window is live on the legacy path too); a faulted or dead watch shared by several rules is repaired by a full push, not by an identical full_sync=false push; and a watch that the mechanism reports healthy but that is deaf without ever being marked faulted is no longer rebuilt by an identical re-push, so it is found only by the periodic convergence sweep (the flip gate’s AC-18, loss 6, accepted by the operator on 2026-10-12; a per-mechanism deaf-watch test is a flip precondition, tracked in #5719).

Store degrade is fail-closed (ADR-0038). Since the Guardian state store moved to PostgreSQL, a push (and the heartbeat reconcile) aborts rather than fan out if the rule store cannot be read (a 503 on the REST path, and no push on the reconcile path) instead of silently pushing an empty rule set (which would disarm every in-scope agent). If a push returns 503, the store is unreachable/degraded, not “you have no rules”; retry once the store recovers. This is a behaviour change from the SQLite era, where a store read error could surface as an empty result. Since #4856, every rule-store read abort ALSO increments yuzu_server_guardian_read_degrade_total{reason,source="guardian_rules"} (this same counter’s rule/status-read source, below), so a sustained run of 503s/aborted reconciles caused by the rule store now pages via YuzuGuardianReadDegraded, not just via the REST error path. That counter and alert cover the rule store (list_rules) only. A reconcile abort caused by an unreadable policy generation or by the Baseline deployed-set read is not counted there: it is visible in yuzu_server_guardian_reconciles_total{result="degraded"}, in the ABORTED audit rows (except the first generation read, which happens before admission and writes no audit row), and in the server log, and the shipped alert YuzuGuardianReconcileDegraded (warning) is the one that fires for it: it watches sum(increase(yuzu_server_guardian_reconciles_total{result="degraded"}[10m])) > 0, or sum(increase(yuzu_server_guardian_reconcile_exceptions_total[10m])) > 0, held for 10 minutes. The same abort applies to a degraded tag or property store during the reconcile’s per-agent scope evaluation (#5513): the reconcile sends nothing, rather than pushing a rule set narrowed to “not in scope”. That scope abort is not covered by YuzuGuardianReadDegraded: it increments only yuzu_server_guardian_reconciles_total{result="degraded"} and writes the ABORTED audit row. The alerts that fire for it are YuzuGuardianReconcileDegraded and the tag-store and custom-properties read-degrade alerts, YuzuTagStoreReadDegraded and YuzuCustomPropertiesReadDegraded (docs/prometheus/yuzu-alerts.yml). The agent is retried at its first heartbeat after the per-agent 25 s slot (an abort keeps the claim, so a heartbeat inside the slot is rate limited). A reconcile dependency that throws is not an abort of this kind: the exception is caught at the heartbeat-ingestion call site, logged as a warning and counted in yuzu_server_guardian_reconcile_exceptions_total (see Metrics; YuzuGuardianReconcileDegraded also fires on a rising count), the rest of that heartbeat’s ingestion still runs, and a throw after admission keeps the claim like an abort, while a throw before admission (the first policy generation read) writes no claim, so the agent is retried at its next heartbeat. The REST and dashboard policy-bump fan-out (which Baseline deploy and delete, REST push and MCP push use) now fails closed the same way (#5610): it resolves the explicit target scope and the scope of every enabled rule in a deployed Baseline before it sends anything, and a degraded tag or property read aborts the whole push and nothing is sent to any agent (REST answers 503 with retry_after_ms: 5000, MCP returns a JSON-RPC internal error with a retry hint, and the dashboard’s Baseline deploy and delete answer 200 with a warning toast and a degraded audit row; only REST returns 503), instead of omitting the scoped rule from every agent’s full push and recording the new generation as delivered. A scope that can never resolve (it names a result set or property with no store available) or does not parse still leaves its rule out for every agent, as before. One limit on this fail-closed behaviour: an agent that has nothing armed yet, such as a fresh install, receives no rules from an aborted reconcile, unscoped rules included, until the store recovers. The same abort applies to a degraded management-group membership read when a push is addressed to a group: scope. For a dashboard Baseline deploy or delete, the abort covers any store read the fan-out depends on: the rule store, the Baseline deployed set, the tag store, the custom-property store or a management-group membership. When a Baseline deploy or delete hits this abort, the Baseline change is already saved and the generation bump has been attempted, but the bump is best effort: bump_policy_generation() returns nothing, and a failure is only logged (GuaranteedStateStore: policy_generation bump failed: ..., or GuaranteedStateStore::bump_policy_generation: no database connection: ... when no pooled connection was available). The audit row is recorded as degraded (not success) and the dashboard shows a warning toast. The heartbeat reconcile re-sends the rule set to each agent once the store recovers ONLY if the bump succeeded; if it failed, agents stay at the old generation and nothing reconciles until the next bump. An agent that refuses a lower-generation full push (below) logs it on the agent; the server does not record the failed push command, but it counts an agent that reports a generation above its own (see the restore runbook below).

An agent refuses a full push labelled below its held generation (#5611). The agent’s held policy generation only ever advances. A full_sync push whose policy_generation is lower than the generation the agent holds is refused before anything is changed: no rule is torn down or re-armed, the agent logs a warning (Guardian: refusing full_sync policy_generation=N below the held generation M (nothing was changed)), and the push command fails with a message containing below the held generation. A full_sync at the held generation is still applied (the boot re-arm catch-up is a same-generation full_sync), and a delta push (full_sync=false) is not covered. An agent binary from before this check applies an older full push and keeps reporting the higher generation, so the server sees it as converged and the older rules stay in force until the next policy change. Servers have never sent a lower label on purpose, so no compatibility step is needed.

Limit: an agent that holds a higher generation than the server. This can happen after the server lost its generation state, for example a database restored to an earlier state. Because the refusal is a strict “lower than”, the agent accepts a full push again as soon as the server’s generation EQUALS the one it holds. Until then:

• Every full push to an agent that carries the check is refused, and the agent keeps enforcing its current rules (an agent from before this release applies an older full push).
• Rule and Baseline edits that travel as full pushes are silently NOT delivered to it. The heartbeat reconcile treats an agent that reports a generation at or above the server’s as converged and sends it nothing, while the REST push, the MCP push and the dashboard still report the push as sent (the audit agents=<N> counts dispatches, not applications). A REST or MCP push with full_sync=false is a delta push, which the agent applies (the refusal covers full_sync pushes only); it merges rules rather than replacing the rule set.
• The failed push command is not visible on the server: the reply to a solicited __guard__ command is dropped (agent_service_impl.cpp, the Subscribe response loop). The signals are the agent’s own log line above, the server’s yuzu_server_guardian_agent_generation_ahead_total counter and its once-a-minute warning (see the restore runbook below).

What raises the server’s counter (the complete list in this release). Creating, updating or deleting a rule, from any surface (REST, MCP or dashboard; enabling or disabling a Guard is an update), raises it by one, inside the rule store. On the dashboard, deploying or re-deploying a Baseline, and deleting a Baseline that was deployed, each raise it by one. Creating or editing a Baseline, deleting a draft Baseline, and a REST or MCP push do not raise it. The one operation that sets it to a chosen value is the audited raise (POST /api/v1/guaranteed-state/policy-generation/raise and the MCP tool raise_guardian_policy_generation; see “Restoring the policy generation” below), which only ever moves it up; no agent command resets the held generation.

Restoring the policy generation

When it happens. The server’s policy generation is one counter in the Guardian state store. It goes backwards relative to the agents when that store is restored to an earlier state, when Postgres is rebuilt or re-provisioned while agents keep the generation they persisted, or when a deployment moves from the retired SQLite store to Postgres (the legacy guaranteed-state.db counter is not carried over: ADR-0038 “Update (2026-09-02)” and ADR-0009’s fresh-start-by-default amendment). Agents also hold a generation above the server’s when an agent that was enrolled with another Yuzu server (or restored from a backup of another server’s state) is re-enrolled against this one: it carries the other server’s higher counter. Agents then hold a generation above the server’s and refuse every full push, as described above.

How to detect it.

• yuzu_server_guardian_agent_generation_ahead_total (see Metrics) rises. A non-zero value means at least one agent has held a generation above the server’s; it does not count agents. It is sampled at most once a minute per server replica, only after a fresh store read confirms the agent is ahead. The shipped alert YuzuGuardianAgentGenerationAhead (warning) fires on sum(increase(yuzu_server_guardian_agent_generation_ahead_total[15m])) > 0 held for 5 minutes, and keeps firing for about 15 minutes after the cause is fixed.
• The server logs one warning per minute per replica: Guardian heartbeat reconcile: agent_id=<token> reports policy generation N above the server's M; full pushes to it are refused by the agent until the server generation reaches N (see Guaranteed State, restoring the policy generation). Read “the agent” there as “an agent that carries the check”: an agent from before this release applies an older full push. The line names one sampled agent with both numbers; other agents may hold a higher generation.
• Each agent reports its held generation in the yuzu.guardian_generation heartbeat tag. This release has no operator page or API verified to list that tag per agent, so take the figure from the warning above or from the agent’s own log line (held generation M).
• Read the server’s current value with GET /api/v1/guaranteed-state/policy-generation or the MCP tool get_guardian_policy_generation (see “Reading the server’s value” below). It is a read-only call that changes nothing. The gauge yuzu_server_guardian_policy_generation shows the generation of the most recent push on a replica and is not set until a push has run after boot. The same value is the value column of the row key = 'policy_generation' in the table guardian_meta of the guaranteed_state_store schema; a SELECT of that row reads it without changing it, which is the fallback when the read route cannot be used.
• Detection limits. The counter and the warning see only an agent that REPORTS a generation above the server’s. An agent whose boot re-arm is unresolved reports generation 0 in yuzu.guardian_generation (with the companion heartbeat tag yuzu.guardian_boot_rearm_unresolved = 1) while it still refuses a lower-labelled full push, because the refusal compares against the generation it holds, not the reported 0. The server therefore does not count it as ahead; it re-pushes to it about every 25 s and counts each push as yuzu_server_guardian_reconciles_total{result="sent"} (audit rows generation 0 -> N for one agent id). For that case the signals are the companion tag and the agent’s own log line, not the counter. The same reporting means a generation read from heartbeat tags can be understated, so take H from the agent log lines (held generation M).

Reading the server’s value. GET /api/v1/guaranteed-state/policy-generation and the MCP tool get_guardian_policy_generation return the generation the server holds and the largest target a raise accepts:

curl -H "Authorization: Bearer $YUZU_TOKEN" \
  https://yuzu.example.com/api/v1/guaranteed-state/policy-generation

{ "data": { "generation": 4820, "ceiling": 9007199254740991 }, "meta": { "api_version": "v1" } }

generation is the stored value; 0 is a real value (a store that has never changed its rule set). ceiling is the largest target_generation the raise accepts. The call needs GuaranteedState:Read (not the administrator check the raise uses), and a service-scoped API token is refused with 403 because the generation is a fleet-wide scalar. On MCP it is readable from the readonly, operator and supervised tiers and needs no approval ticket. A degraded Guardian store is a 503 with retry_after_ms on REST and a retryable error on MCP, never a generation of 0; a server with no Guardian store wired answers a 503 with no retry hint. The REST call writes no audit row on success (an MCP call is recorded only by the generic mcp.get_guardian_policy_generation tool-call row); a service-scoped token refused on REST is audited as guaranteed_state.policy_generation.read with result=denied, and one refused on MCP as mcp.get_guardian_policy_generation with result=denied (the generic service-scope default-deny gate acts before the tool handler; see Audit Log).

How to compute the gap. H is a lower bound, not a known number. The warning names one sampled agent per replica per minute, and any one agent may hold less than another. Take H as the highest held generation over the refusal log lines (below the held generation M) of several agents, never a single sampled value, and compare it with the server’s current value from the read call above (or, as a fallback, from guardian_meta).

An agent that came from another server. An agent re-enrolled against this server after being enrolled with another one, or restored from a backup of another server’s state, keeps the generation it persisted: the agent stores it in its local KV store (kv_store.db in the agent’s data directory, namespace __guardian__, key meta:policy_generation) and reloads it at start. If that number is above this server’s counter, the agent refuses every full push labelled below it and is stuck behind the same refusal as above, whatever rules this server holds. This page does not describe a supported agent-side way to lower the held generation (no agent command resets it); the supported recovery is the raise below, to H + 1 where H is the highest generation any such agent reports, which is read from its refusal log line (below the held generation M). The raise affects the whole fleet, not only the re-enrolled agent: every agent behind the new value converges to this server’s rule set, so step 0 applies in full.

Recovery. Use the supported raise operation: POST /api/v1/guaranteed-state/policy-generation/raise (REST) or the MCP tool raise_guardian_policy_generation. It sets the server’s counter to the target you give, in one atomic step that can only move the counter up, and it records an audit trail. It never pushes anything: the agents behind the new value are converged by the heartbeat reconcile (see “What you will see afterwards”). It is irreversible for the agents that receive the new value, because an agent keeps the highest generation it has seen and no command lowers it.

0. Before you raise, check what rule set the server holds. Raising the counter makes every agent that is behind converge (full_sync) to exactly the rule set the server holds and has deployed NOW. After a rebuild, a re-provision or the move from the retired SQLite store, the rule and Baseline tables are empty (ADR-0038 “Update (2026-09-02)”: rules are never loaded and enforcement starts from an empty rule set; with nothing deployed a full_sync converges agents to zero guards), so the raise would tear down the guards the agents still enforce. After a restore to an earlier state it rolls every agent back to the restored, older rule set. So first confirm that the rule store (GET /api/v1/guaranteed-state/rules) and the deployed Baselines (the Baselines section of /guardian) hold the policy you want enforced fleet-wide, and re-create the rules or re-deploy the Baselines first if they do not. The request cannot be made without the field acknowledge_rule_set_replaced set to the JSON boolean true, which is your statement that you have done this check; the operation does not do it for you. A second acknowledgement applies to an empty rule set. When the server holds no rules at all (the rule count is 0) or no deployed Baseline (the deployed Baseline count is 0), the raise is refused (REST 400, MCP -32602, a denied audit row, counted refused, nothing changed; on MCP the approval ticket is already spent, so the retry needs a fresh one) unless the request also carries acknowledge_empty_rule_set set to the JSON boolean true. After a Postgres rebuild or the move from the retired SQLite store the store is empty, so the raise is refused until the rules are re-created and deployed. Sending acknowledge_empty_rule_set: true means: “I intend every agent behind the new value to converge to zero guards.” Send it only when that is what you want; the usual action is to re-create the rules and re-deploy the Baselines first and then raise without it. Its response reports rules, the number of rules in the rule store (not the number of guards agents will receive: agents converge only to enabled rules that are members of deployed Baselines), and deployed_baselines, the number of deployed Baselines, both read just before the raise, so a deployed_baselines of 0 there means the agents behind are now converging to zero guards; that can appear only because you sent acknowledge_empty_rule_set. Check earlier rather than relying on this response.

1. Choose the target: exactly H + 1. H alone stops the refusals, but an agent that already holds H is then treated as converged and receives nothing, so an edit made before the restore would still not be re-sent. A larger value works too but is permanent for every agent that receives it, so do not pick a round or large number for convenience. target_generation must be an integer from 1 to 9007199254740991 (2^53 - 1, the largest integer a JSON number carries exactly; the response field ceiling repeats it). A float, a string, a boolean, 0, a negative number or a larger number is refused with 400 and changes nothing.

2. Send the request.

   curl -X POST https://yuzu.example.com/api/v1/guaranteed-state/policy-generation/raise \
     -H "Authorization: Bearer $YUZU_TOKEN" \
     -H "Content-Type: application/json" \
     -d '{"target_generation": 4821, "acknowledge_rule_set_replaced": true}'

   Field	Type	Required	Description
   target_generation	integer	Yes	The generation to raise the server to, 1 to 9007199254740991.
   acknowledge_rule_set_replaced	boolean	Yes	Must be true. Missing, false or any other type is refused with 400.
   acknowledge_empty_rule_set	boolean	Only when the server holds no rules or no deployed Baseline	Must be true in that case, otherwise the call is refused with 400 (MCP -32602) and nothing changes. Optional otherwise; when sent it must be a boolean.

   For the empty-store case (after a rebuild, with the rules and Baselines not yet re-created) the same request needs the second field, and means every agent behind the new value converges to zero guards:

   curl -X POST https://yuzu.example.com/api/v1/guaranteed-state/policy-generation/raise \
     -H "Authorization: Bearer $YUZU_TOKEN" \
     -H "Content-Type: application/json" \
     -d '{"target_generation": 4821, "acknowledge_rule_set_replaced": true, "acknowledge_empty_rule_set": true}'

   The MCP tool takes the same arguments and returns the same fields. Who may call it:

   • An administrator, and only an administrator. The gate is the same durable is_rbac_administrator check as PUT /api/v1/rbac/enforcement, not an ordinary permission check, so GuaranteedState:Push does not suffice. With RBAC enforcement off, the caller’s own local account must hold the admin role; with it on, the caller must hold a user-principal Administrator grant. A JIT elevation or a group-held grant does not count.
   • REST: MFA step-up applies when the account is enrolled and its proof is stale; the route then answers 401 with the step-up envelope (see REST API). A SAML session gets 403 at the step-up gate. An SSO (OIDC) session is judged on the identity provider’s MFA assertion and the enforcement mode instead of a local enrolment. An API-token session (api_token or mcp_token) is exempt from step-up, as it is for the RBAC enforcement toggle, and --mfa-step-up-window-secs 0 disables the gate. Limit: with --mfa-enforcement optional (the default) an administrator who has not enrolled in MFA is not stepped up, and neither is an SSO (OIDC) administrator whose identity provider sent no MFA claim, and any administrator API-token session is exempt, so on this route such administrators are not MFA-gated. A service-scoped API token, an engine session and a bearer token that carries an MCP tier are refused with 403. A browser (cookie-session) call must also carry a same-site Origin or Referer (or a --csrf-trusted-origin entry); a cross-site call is refused with 403 and audited as csrf.denied. API-token callers are exempt, and the curl example above, which uses a token, needs no Origin. After the administrator check the route re-checks the caller’s authority under the current RBAC regime, as the other RBAC administration routes do.
   • MCP: the tool needs the supervised tier and is approval-gated (ticket-then-recall, see MCP); a call with no MCP tier is denied outright. There is no MFA on MCP: the tool’s protection is the supervised-tier approval ticket plus the administrator check, and it re-checks the caller’s authority under the current RBAC regime after the administrator check.

3. Read the response.

   Field	Meaning
   previous_generation	The stored generation immediately before the call.
   generation	The stored generation after the call.
   raised	true if the call moved the counter up.
   rules	Number of rules in the rule store, read just before the raise. It is not the number of guards agents will receive: agents converge only to enabled rules that are members of deployed Baselines.
   deployed_baselines	Number of deployed Baselines, read just before the raise.
   ceiling	The largest accepted target, 9007199254740991.

   The operation is idempotent. A target at or below the current value changes nothing and returns raised: false with generation equal to the current value, so repeating a call that already took effect, or a call that lost a race with an ordinary rule edit, is safe. A raised: false response with a generation at or above H + 1 means the counter had already moved past your target, which ends the refusals for any agent that holds no more than that value.

Audit and effect. Each call that reaches the store writes two guaranteed_state.policy_generation.raise audit rows: attempted before anything changes, then the outcome (success, denied or failure); if the first row cannot be written, nothing changes and the route answers 503 with Sec-Audit-Failed: true. A request turned away earlier (a gate, a cross-site cookie call, a rejected body) writes at most one denied row or none; the exact cases are in Audit Log. A raise that moved the counter drops this replica’s cached generation and rule bundle at once; other replicas read the value through a cache of about 2 s.

If the call returns 503, repeating it is the verification. Two 503 cases leave the outcome unconfirmed: the raise took effect but its outcome audit row could not be persisted (the attempted row is then the only record), and the store failed after the attempted row was written, so it is not known whether the value moved. In both, repeat the same request: it is idempotent. A repeat of a raise that took effect returns 200 with raised: false, with generation and previous_generation both equal to the target (or higher, if an ordinary rule edit moved the counter in between), so success on a repeat looks like a no-op; that is the confirmation, not a failure. If the first attempt never committed, the repeat performs the raise and returns raised: true; either result leaves the counter at the target or higher. The other 503 causes are refusals that changed nothing: the audit log was unavailable before the change (audit log unavailable; the change was not made), the server could not confirm administrator authority, the service was unavailable, and a 503 that says the policy generation was not changed (an unreadable rule or Baseline count). If the audit store is still down when you repeat after a lost outcome row, the repeat is refused with the change was not made; that does not undo the earlier raise, so check by repeating once the audit store is back. On MCP the same cases arrive as tool errors, and a repeat needs a fresh approval ticket, because the first call consumed its ticket (see MCP). Outcomes are counted in yuzu_server_guardian_policy_generation_raise_total{transport,result} (see Metrics). If a later look still shows refusals from an agent that held more than H, repeat the call with H set to the new maximum.

Last resort: raise the counter by hand. Use this only when the operation above cannot be used, for example when no administrator can reach it. It bypasses Yuzu’s audit trail, so record the edit in your change record. As a database role with write access to the guaranteed_state_store schema, run UPDATE guaranteed_state_store.guardian_meta SET value = GREATEST(value, <H+1>) WHERE key = 'policy_generation' RETURNING value;, replacing <H+1> with H plus one, and do the step 0 check first because the same convergence follows. Read the value back with the read route above once the server is reachable; it is the quickest check. Check the result before relying on it: the statement must return exactly one row (zero rows means the policy_generation row is missing and nothing was changed), and the returned value must be H + 1, or higher if the counter had already moved, and below the ceiling 9007199254740991. Do not enter a value near 9223372036854775807 (2^63-1): the counter is a BIGINT, so its next bump fails with bigint out of range (logged as GuaranteedStateStore: policy_generation bump failed), later edits can no longer raise it, and the heartbeat reconcile, which fires only when the server’s generation is above the agent’s, never fires again. Do not enter a negative value: the server treats a negative stored number as an unreadable policy generation, so the read route answers 503 and the heartbeat reconcile aborts as degraded until the value is repaired (see “A negative stored value” below). GREATEST keeps the statement from lowering a counter that has already moved. The statement is safe to run while the server is running, because the server’s own bump is one atomic UPDATE ... SET value = value + 1 on the same row, and no replica restart is needed: each replica reads the value through a cache of about 2 s and every push reads the store directly. The gauge yuzu_server_guardian_policy_generation moves only on the next push on each replica, so do not use it to confirm the edit.

A negative stored value. If the stored policy_generation is negative (only an out-of-band edit of guardian_meta can cause that), the server does not reinterpret it as a very large number: the read route (GET /api/v1/guaranteed-state/policy-generation) answers 503, and the heartbeat reconcile aborts as degraded (yuzu_server_guardian_reconciles_total{result="degraded"}), so no agent is reconciled. The raise operation repairs it: it treats the negative value as 0, so the response reports previous_generation 0, and sets the counter to your target. Choose the target from H as above.

Another way to move the counter: counted changes. Each change in the list above raises it by one, but every one of them is a real policy change that rolls out to the fleet: a rule create, update or delete, a Baseline deploy, or the delete of a deployed Baseline. Never toggle a Guard to bump the counter, because disabling one disarms its enforcement. Use only no-op re-saves of a rule or re-deploys of a Baseline, and only for a small gap: each bump starts a fleet reconcile. Pushes made before the counter reaches H are refused by the agents that hold more than the label, and nothing else changes on those agents. Step 0 applies here too, because each re-deploy re-pushes.

What you will see afterwards. Every agent behind the new value is reconciled: expect guaranteed_state.reconcile success audit rows reading generation <n> -> <H+1>, a spike in yuzu_server_guardian_reconciles_total{result="sent"}, and, on a large fleet, shed while the replica budget drains (see “Delay after a bump” in Server Administration). None of this is a new fault. A reconcile already in flight on the replica that took the raise is withheld rather than sent: it writes an ABORTED row reading rule bundle invalidated during load and increments yuzu_server_guardian_reconcile_invalidated_total (see Metrics), and the agent is reconciled from a fresh load at its next heartbeat after its 25 s slot, so a few such rows right after the raise are expected.

Confirm. Read the value back with GET /api/v1/guaranteed-state/policy-generation (or get_guardian_policy_generation): generation must be at least H + 1. The counter is sampled at most once a minute per replica, so wait at least 3 minutes, then check that changes(yuzu_server_guardian_agent_generation_ahead_total[3m]) == 0 on every replica and that the agents’ refusal log lines have stopped. An agent in the unresolved boot re-arm state (see the detection limits above) is not covered by this check. YuzuGuardianAgentGenerationAhead is built on a 15 minute increase of the counter, so it keeps firing for about 15 minutes after the cause is fixed; confirm with the changes() check above rather than by waiting for the alert to clear. The same holds for an alert built on a 15 minute increase of the raise counter’s result="failed" series (see Metrics): confirm with a repeated call rather than by waiting for it to clear.

DEX reads degrade to empty, not error (interim, #2659). The /dex analytic reads (crash/signal/app aggregations) currently return an empty result on a degraded PG connection rather than a 503 — a degraded dashboard can read as “0 crashes.” Each such degrade is counted at yuzu_server_guardian_read_degrade_total{reason,source="guardian_state"} — the other source on this shared counter is guardian_rules (the authoritative rule/status reads described above, which fail hard rather than empty). Alert YuzuGuardianReadDegraded (sum(rate(yuzu_server_guardian_read_degrade_total[5m])) by (reason, source) > 0, docs/prometheus/yuzu-alerts.yml, #4856) fires on either source; until #2659 widens the DEX reads, treat a zero DEX panel under a firing guardian_state window as “unknown,” not “healthy.”

Named exceptions (#4855, WS-A4 PR-1): three DEX reads fail CLOSED instead of degrading to empty. dex_device_signal_summary — the read behind the device-page DEX lens, GET /api/v1/dex/devices/{id}, and MCP get_dex_device_score — was the FIRST DEX/analytics read to depart from the “#2659 interim, degrade to empty” posture above: a degraded read surfaces as a degraded placeholder on the lens (“DEX store degraded.”), a 503 (retry_after_ms: 2000) on REST, and a retryable error on MCP — never a fabricated healthy score of 100 with no signals. GET /api/v1/dex/catalogue and MCP get_dex_catalogue are the SECOND named exception: a degraded read there also returns a 503/retryable error rather than an empty catalogue that would read as “every family healthy.” GET /api/v1/dex/apps and MCP list_dex_apps are the THIRD: a degraded read (or an invalid score policy) returns a 503 (retry_after_ms: 2000) / retryable error rather than an empty list; the dashboard Apps page still reads dex_top_apps and stays fail-soft. Every OTHER /dex analytic read (crash/signal/app aggregations, the fleet overview/health composites) still degrades to empty per the paragraph above; the /dex overview panel additionally now COUNTS connected devices whose per-device score could not be computed (“N device(s) could not be scored”) rather than silently thinning the great/fair/poor population — the health panel does not read per-device scores and carries no equivalent count.

Retention reap cadence + ceiling (#2660). Expired Guardian events/observations are reaped on a ~60-minute clock-guarded pass capped per pass; under a sustained high-write incident the backlog can drain slower than it accumulated. Alert on a stalled reaper with absent_over_time(yuzu_server_guardian_reap_passes_total[75m]); drain-rate tuning for large fleets is tracked in #2660.

5. Query events and status

# Recent events, filtered by severity
curl -H "Authorization: Bearer $YUZU_TOKEN" \
  "https://yuzu.example.com/api/v1/guaranteed-state/events?severity=high&limit=50"

# Fleet rollup (errored_rules is real, #2298 item 6d; compliant/drifted are still placeholder 0)
curl -H "Authorization: Bearer $YUZU_TOKEN" \
  https://yuzu.example.com/api/v1/guaranteed-state/status

Both require GuaranteedState:Read. The four count fields (total_rules, compliant_rules, drifted_rules, errored_rules) match the agent-side proto GuaranteedStateStatus; the REST response also carries note (human-readable caveats) and, on the per-agent route, agent_id — neither of those two has a proto counterpart. errored_rules is derived from the guardian_agent_rule_status census (the count of distinct rules with at least one agent currently reporting errored); compliant_rules/drifted_rules stay 0 until full status ingest lands in a later rung. The agent status is fail-closed: until the guard self-test lands, an armed guard reports errored / guard_healthy=false, never compliant. Treat the events stream (drift.remediated / remediation.failed), not /status’s still-placeholder compliant_rules/drifted_rules, as the source of truth for whether enforcement is working.

GET /api/v1/guaranteed-state/rules, .../events, and .../status each have an MCP twin (list_guardian_rules, list_guardian_events, get_guardian_status, #4037) calling the exact same store/builder as their REST sibling — an agentic worker sees the identical rule catalogue, event stream, and rollup a human operator sees over REST or the dashboard. Two more read surfaces are new with #4037 and have no dashboard-fragment sibling of their own shape: GET /api/v1/guaranteed-state/rules/{rule_id}/status / MCP get_guardian_rule_status (the per-guard fleet-wide agent-status drilldown — every agent’s state for ONE rule) and GET /api/v1/guaranteed-state/agents/{agent_id}/rules / MCP get_guardian_device_guards (the per-device all-guards view — every guard’s state for ONE device, unscoped to any Baseline; contrast with device-compliance, which is scoped to one named Baseline). #2146 Batch B1 adds MCP twins for the remaining write and per-device surfaces: rule create/get/update/delete (create_guardian_rule/get_guardian_rule/update_guardian_rule/delete_guardian_rule), push (push_guardian_rules), per-agent status (get_guardian_agent_status), and device-compliance (get_guardian_device_compliance) — every REST route on this page now has an MCP twin, including POST /api/v1/guaranteed-state/policy-generation/raise and its tool raise_guardian_policy_generation (administrator-only, see “Restoring the policy generation”). See docs/user-manual/rest-api.md and docs/user-manual/mcp.md for the full contract of each.

6. Enable/disable a Guard from the dashboard

The Guardian dashboard guard list (/guardian) exposes a per-Guard Enable / Disable toggle. It flips the Guard’s enabled flag and is state-only: it requires GuaranteedState:Write, audits under guaranteed_state.rule.update, and does not push. The change reaches agents on the next Baseline deploy or the periodic heartbeat reconcile, both of which re-push the union of deployed Baselines’ enabled members — a disabled Guard is then omitted and the agent disarms it. For scripted control use PUT /api/v1/guaranteed-state/rules/<id> with {"enabled": false}. This disable-or-remove path is also the incident lever for stopping Guardian rule delivery, because the __guard__.push_rules kill switch no longer stops it (see Rule delivery is protected from the kill switch).

There is no mode toggle. A Guard’s enforcement_mode (enforce vs audit/observe) is fixed at creation and immutable — a different posture is a different Guard. PUT /api/v1/guaranteed-state/rules/<id> with an enforcement_mode that differs from the stored value returns 400 (enforcement_mode is immutable — create a new Guard for a different posture (Watch vs Enforce)). To change mode, author a new Guard and add it to a Baseline.

Dashboard reads require GuaranteedState:Read — an authenticated user without it gets a 403 on each data panel (the page shell still loads). The compliance overview shows the live per-(agent, rule) census from the store (see Compliance overview), not a fabricated percentage.

A service-scoped API token is additionally denied outright (403) on every fleet-wide data panel — status, guards list, event timeline, the per-Guard drilldown, baselines list, and per-Baseline detail — regardless of the GuaranteedState:Read grant it carries. These panels have no single device to confine the token’s service-tag scope against, unlike the shared device page’s per-device Guardian lens (GET /fragments/device/guardian?id=<agent_id>), which a service-scoped token can still read for its own service’s agents. The denial itself is audited (guaranteed_state.fragment.access_denied); the per-Guard drilldown additionally audits every successful open (guaranteed_state.rule.view) since it discloses every reporting agent’s identity, hostname, and state for one rule fleet-wide.

The same deny applies on the write side. Creating a Guard, flipping its enable/disable toggle, and creating, editing, deploying, or deleting a Baseline are the same 403 for a service-scoped token, regardless of the GuaranteedState:Write/Push/Delete grant it carries — a service-scoped token cannot deploy a Baseline (the fleet-wide operation that determines what every agent enforces). Each mutation’s own denial reuses that action’s existing success-verb audit row with result=denied (guaranteed_state.rule.create, .rule.update, .baseline.create, .baseline.update, .baseline.deploy, .baseline.delete) rather than a separate generic verb — see Audit log.

Offline convergence: an agent that was offline or unreachable when the deployed set changed re-converges automatically on reconnect. It reports its applied policy generation (yuzu.guardian_generation, which also reads 0 intermittently while a boot re-arm is unresolved, see the back-off in limit 3 below) in each heartbeat; the server re-pushes when that generation is behind the current store generation (rate-limited per agent). No manual re-push is needed after a network partition. Once the Spark-backed detection path is live, the same convergence can take one or two additional heartbeat ticks under a slow-arming rule — an accepted arm is tracked by an internal acknowledgment ledger and the generation only advances once the arm is confirmed, instead of the agent blocking on it synchronously. This has no visible effect today: a generation advances exactly as it always has while the Spark path stays off by default.

Reconcile admission and fleet-scale convergence (#5513). This admission control removes the heartbeat reconcile’s O(fleet) cost and makes it shed-safe; it does not by itself make the server handle 500,000 agents. The heartbeat reconcile is rate-limited per agent (25 s) and, since #5513, admitted per server replica against a time budget (--guardian-reconcile-budget-ms-per-s, --guardian-reconcile-burst-ms, --guardian-reconcile-max-per-s; see Server Administration). A reconcile the replica declines to admit (its budget is empty, its per-second ceiling is reached, or about 256 pushes are already in flight, a soft cap) is shed: nothing is sent, no audit row is written, and the agent’s 25 s slot is not consumed, so the agent simply reports its stale generation again on its next heartbeat and is admitted when budget is available (yuzu_server_guardian_reconciles_total{result="shed"}). After a policy-generation bump on a large fleet, convergence time therefore scales with fleet size: roughly agents / (replicas x admitted pushes per second per replica). For example, assuming 1 ms per push (an assumption, not a measurement), 500,000 agents take about 5.5 minutes on 3 replicas that each admit 500 pushes per second. At an estimated 3 to 8 ms per push (not measured end to end) the default budget admits roughly 60 to 170 pushes per second per replica, so the same 5 minute convergence needs roughly 10 to 28 replicas. Each replica reads the policy generation through a short cache (2 s, 1 s after a failed read) that decides whether to admit, rate-limit or shed; an admitted push then re-reads the generation fresh (one Postgres read per admitted push, not per heartbeat) and labels the push with it, and a reconcile whose fresh generation is not ahead of the agent sends nothing. The cache means a per-agent delay of about 2 s plus one heartbeat interval after a bump: an agent whose heartbeat lands inside the TTL window is treated as converged and re-reports one heartbeat later (about 32 s at the 30 s default). The rule set is held per generation in memory for at most 60 s. A rule create, update or delete, a Baseline update (a deploy), delete or deployed create, a policy-generation raise that increased the stored value (a raise that changed nothing does not), and a Guardian push fanned out by THIS replica each drop the cached generation and the held rule set at once (the rule store and the Baseline store call a post-mutation hook after the change commits, and the rule store does so after its generation bump was attempted, even if the bump failed). A reconcile that was already running when that happens sends nothing (yuzu_server_guardian_reconcile_invalidated_total, an ABORTED audit row, its 25 s claim kept) and is retried from a fresh load, so on the replica that took the edit a failed bump does not leave a stale rule set served to a newly reporting agent, apart from the few instants around the hook. The guarantee is that narrow: an edit made through ANOTHER replica is seen after the cache TTL, and if that replica’s bump failed this replica can serve the pre-edit rule set for up to 60 s; an agent reconciled from it stays converged until the next bump, as does an agent that had already converged before the edit, and a reconcile that races the instants between a commit and its hook can still send the old rule set (the details, and the effect of a sustained stream of edits, are under “Delay after a bump” in Server Administration). Scope membership is never cached: it is evaluated for the one agent at push time. Behaviour change: if the tag or property store cannot be read while evaluating a rule’s scope, the reconcile now aborts (nothing is sent; yuzu_server_guardian_reconciles_total{result="degraded"} and the ABORTED audit row) and the agent is retried at its first heartbeat after the 25 s slot. Previously the unreadable store was treated as “agent not in scope” and the push carried a narrowed rule set that disarmed the rule on that agent. The default budget values are provisional starting points, not derived from fleet measurement. The abort-on-degrade applies to the REST and dashboard policy-bump fan-out (guardian_push_fn_) too (#5610), which is not budgeted (it runs synchronously). An agent with nothing armed yet receives no rules from an aborted reconcile, unscoped rules included, until the store recovers. The per-heartbeat Postgres writes and the gateway forwarding of each push remain separate follow-ups, no alert ships for shed, and the audit store receives at most one row per admitted push (none when the push ends without sending and without aborting: no_session, a fresh generation that is not ahead of the agent, or a failed queue write).

7. Create a Guard from the dashboard

The + New Guard button (top of /guardian) opens a structured create form for the Windows registry types: name, severity, enforcement mode, the registry-value-equals assertion (hive / key / value name / value type / expected), the remediation action (alert-only or enforce), and the resilience policy fieldset (mode + the mode-relevant tuning params; blanks use defaults). Submitting builds the same structured Guard the REST POST /rules endpoint accepts and runs it through the same validation (so an invalid resilience policy is rejected with an inline banner naming exactly what to fix). On success the Guard is created unscoped (draft) — device targeting is set at the Baseline, not per-Guard — and the guard list refreshes in place. Requires GuaranteedState:Write; audited under guaranteed_state.rule.create.

8. Edit or delete a Guard from the dashboard

The Guard page (/guardian/guard/<id>) shows Edit and Delete buttons to a user who holds GuaranteedState:Write and GuaranteedState:Delete respectively (checked once when the page loads; the mutators re-check).

• Edit changes name, severity and enabled state. Mode is shown as read-only text — it is fixed at creation, and a request carrying a different mode is a 400. Saving bumps the Guard’s version; the last write wins, there is no check for a concurrent edit. Audited under guaranteed_state.rule.update. A Guard’s trigger and assertion are not editable here; use PUT /api/v1/guaranteed-state/rules/<id> or the MCP update_guardian_rule for a re-authored spec.
• Delete asks you to type the Guard’s name. It removes the Guard and its live compliance state; agents already enforcing it keep enforcing until the next push (#4304). Audited under guaranteed_state.rule.delete.

Both actions need a fresh MFA step-up for a signed-in user with MFA enrolled (the dialog opens a nested “Confirm it is you” modal for the authenticator or recovery code and replays the request on a valid code; a 403 or 503 refusal, such as a SAML session or an unavailable authentication store, appears inside the dialog as the server’s message, with no prompt and no toast), are refused for a service-scoped API token, and are refused when the request is not same-site (audited as a denied row with detail csrf_denied). Their routes are POST /fragments/guardian/guard/<id>/update and /delete, twinned to the REST PUT / DELETE /api/v1/guaranteed-state/rules/<id> and the MCP update_guardian_rule / delete_guardian_rule.

The dashboard edit, delete and Enable/Disable write the audit row before the change and fail closed: if the row cannot be written the request is refused (503 with Sec-Audit-Failed: true for edit and delete; the toggle re-renders the list with a banner and the same header) and nothing changes. A change that is then rejected by the store (a name conflict, for example) adds a result=failure row for the same verb after the success row, so one request can leave a success and a failure row; read the pair together. The REST and MCP twins still audit after the change (tracked in #5797).

Baselines — how Guards reach agents

A Baseline is a named collection of Guards and is the only deployable unit. Authoring a Guard (REST or the dashboard form) creates it unscoped and inert — it enforces nothing until it is a member of a deployed Baseline. This mirrors a GPO / Intune configuration profile / Jamf profile: you assemble controls into a baseline and deploy the baseline.

The Baseline surface is on the dashboard (/guardian → Baselines). Its read side has a first agentic entry: GET /api/v1/guaranteed-state/device-compliance?baseline={name}&agent_id={id} looks a Baseline up by name and returns the Guards applicable to a device plus that device’s per-Guard verdict (see REST API → Guaranteed State) — built for embedding compliance into an external CMDB/ITSM CI record (the integration pins one stable Baseline name such as ServiceNow Compliance, so there is no per-environment baseline_id to track; a scripted baseline-list endpoint is a tracked follow-up). Create, edit, deploy, and delete remain dashboard-only (tracked as an agentic-first parity gap). Scripted/agentic clients can author individual Guards via REST or MCP (#2146 Batch B1 added the MCP twins for rule create/get/update/delete and push), but enforcement requires those Guards to be members of a Baseline an operator deploys — authoring a Guard (or calling POST /push/push_guardian_rules) never enforces a standalone Guard, and the deployable-collection authoring model itself is GUI-only for now.

All four actions above additionally deny a service-scoped API token outright (403), regardless of the securable grant it carries — see the note in §6.

The deployed snapshot is authoritative. What the fleet enforces is the member set captured at the last deploy, not the Baseline’s live (possibly-since-edited) membership. So editing a deployed Baseline’s members is a staged change: the dashboard shows a persistent “members changed since last deploy — Re-deploy to apply” warning, and the change reaches agents only when someone with Push re-deploys. This keeps which Guards are delivered gated behind deploy (Push) authority, separate from edit (Write) authority. A member Guard’s own properties are different: its enabled flag, scope and OS target are not part of the snapshot but are read live each time a push is built, so disabling or re-scoping a member Guard (Write authority, see §6) reaches agents with the next heartbeat reconcile or deploy, and the members-changed warning does not cover it (it compares membership only).

Targeting is fleet-wide for now. A Baseline carries an assignment (included − excluded management groups), but management-group targeting is not yet wired — every deploy currently converges the whole fleet, and the dashboard labels the assignment area as coming-soon. Do not rely on the assignment to contain a Baseline’s blast radius yet.

Rule delivery is protected from the kill switch. Every Guardian push to an agent — a Baseline deploy, the normal /api/v1/guaranteed-state/push call, and the periodic reconcile re-push — dispatches through the server’s one internal __guard__.push_rules capability. That capability is a protective action: it is exempt from the per-plugin/action kill switch, the per-OS switch and a degraded plugin-config store, because a contained device must keep receiving the rules that make containment meaningful. The __guard__/push_rules kill switch was previously an emergency stop for rule delivery; it no longer is. A row can still be written via PUT /api/v1/plugin-config/__guard__/kill-switch?action=push_rules or the MCP set_plugin_kill_switch tool, and GET .../kill-switch (or get_plugin_kill_switch) will display it, but it has no dispatch effect and never stops a push; the write is audited with protective_exempt=true. During an incident, stop delivery by disabling or removing the rules (§6). That lever is state-only and disarms the rule rather than freezing pushes: it takes effect on the agent’s next Baseline deploy or heartbeat reconcile, so the latency is the reconcile cadence. It needs GuaranteedState:Write (disable) or GuaranteedState:Delete (remove); PluginConfig:Write alone no longer stops anything, and a service-scoped API token is denied rule writes. Upgrading with a hold in place: see Upgrading. See ADR-3005 for the plugin-config/kill-switch design (the REST/MCP surface itself is not yet documented in the user manual — tracked as #3275, a pre-existing gap, not introduced here).

Assignment

A Baseline’s assignment is its device targeting: a set of included minus excluded management groups (exclude wins). It is deferred — management-group targeting is not yet wired (see the targeting note above), so every deploy currently converges the whole fleet and the dashboard labels the assignment area “coming soon”. Until assignment lands, a deployed Baseline applies to every device; do not rely on assignment to contain a Baseline’s blast radius.

The per-device REST read (GET /api/v1/guaranteed-state/device-compliance?baseline={name}&agent_id={id}, see REST API → Guaranteed State) is keyed by Baseline name and returns the Guards actually applicable to the device — the deployed Baseline’s members this device has reported. Because each Guard carries its own scope_expr, one Baseline can hold a superset of Guards and arm a different subset per machine, so two devices querying the same Baseline name legitimately see different Guards. The read is not a statement that the Baseline was targeted at the device via management-group assignment; broader assignment-awareness (returning not-applicable for an out-of-scope device, and an honest pending for an in-scope-but-unreported Guard) lands when assignment is wired.

ServiceNow CI sync — setup checklist

Stand up a ServiceNow (or any CMDB/ITSM) Guardian-compliance CI field against the device-compliance read:

1. Create + name the Baseline. Author the member Guards, create the Baseline, and give it a stable name the integration will pin (e.g. ServiceNow Compliance). The name is the external identifier — do not rename it later (renaming 404s the integration).
2. Deploy it (Push). A draft returns deployed: false; only a deployed Baseline yields compliance data.
3. Mint a token. Use a global GuaranteedState:Read API token for the service account (the documented setup). A management-group-scoped token works too but 403s for devices outside its groups — prefer global unless you intend per-group restriction.
4. Author the business rule to poll GET /api/v1/guaranteed-state/device-compliance?baseline=<name>&agent_id=<device> per CI on a cadence (15–60 min; stagger, don’t stampede), and branch on the response in this order:
   • deployed: false → render “No Baseline Deployed”.
   • assessable: false → render “Not assessable” (offline / newly-enrolled / all-out-of-scope) — never compute or show a compliance percentage.
   • assessable: true → show the compliant/drifted/errored/pending counts. Do not gate automated remediation on compliant/total_guards (it over-estimates on partial reports — see the REST caveats); treat it as advisory and use last_updated for freshness.
   • 503 handling — discriminate by header + retry_after_ms (two 503 classes are header-less): Sec-Audit-Failed: true → transient audit-subsystem fault, retry after retry_after_ms. No Sec-Audit-Failed but a retry_after_ms in the A4 body → transient store fault, retry after retry_after_ms. Neither header nor retry_after_ms → permanent misconfiguration (unwired stores / scoped-permission fn) — page an operator, do not auto-retry.

Compliance overview

The dashboard’s compliance overview (/guardian) reports live fleet compliance from a per-(agent, rule) census (table guardian_agent_rule_status, deliberately not subject to the event-retention reaper, so a long-quiet compliant Guard keeps its real state) that the server maintains off the event stream. Events map to it as follows: guard.compliant and drift.remediated set it to compliant, drift.detected to drifted, and remediation.failed, guard.unhealthy and guard.errored to errored; guard.armed, guard.disarmed and guard.healthy leave it unchanged. guard.errored is produced only by the spark detection backend’s subscription-death detach and stays dormant until prefer_spark ships. The Fleet Status panel has three views:

• Fleet — coverage stat cards, a compliant/drifted/error/unknown proportion bar, a “needs attention” summary, and a 7-day enforcement-effectiveness trend. The stat cards are clickable: Guards / Baselines jump to the matching view, Agents opens Fleet Viz, and Guards drifting / Enforcement failures / Unhealthy Guards jump to By Guard with the matching filter pre-applied.
• By Guard — one card per Guard (compliance bar, severity/mode/enabled pills, 7-day detected/remediated counts), above a filter bar (free-text search + state / severity / mode). Clicking a card opens the full Guard detail page (/guardian/guard/<id>).
• By Baseline — one card per Baseline (compliance bar, lifecycle, drift counts). Clicking a card opens the full Baseline detail page (/guardian/baseline/<id>).

The Guard and Baseline detail pages — which replaced the old detail modal — show a compliance hero, stat tiles, and a filterable/searchable per-device census (on the Guard page, device rows link to the host page /viz/host/<id>); the Baseline page also lists its member Guards and recent events, with Edit available from the Baselines section on /guardian. The Recent Events panel on /guardian has a free-text search box that filters rows client-side.

Compliance derives from one liveness-folded rollup: an agent that is currently offline folds to unknown (its last state is not reported as live truth). The Fleet census, the per-Guard/Baseline cards, and the detail pages all read that single rollup, so they cannot disagree.

The same rollup also folds in a synthetic not implemented count: an (agent, rule) pair whose platform can’t arm that Guard’s type — a per-guard-type matrix (service-status-change arms, observe-only, on Windows and Linux; registry-change/file-change are Windows-only; nothing arms on macOS today), never a blanket “non-Windows” rule — and which doesn’t already own a real status row. It counts toward the denominator, never toward compliant, so a fleet with unenforceable pairs never silently reads as 100%. The Fleet view’s honesty banner names the agents affected pair-level, not platform-level — an agent whose only deployed Guard type it does support is never flagged just for being on a less-covered platform. Two known, accepted edge cases of this fold (#4252):

• A Service guard that never actually arms on a genuinely-supported platform (no system D-Bus — every containerized/compose Linux agent — a disabled build flag, or an invalid unit name) reports nothing and so is indistinguishable from an unreported pair: it silently drops out of the denominator rather than showing “not implemented”. If a compliance number looks better than expected for a Linux fleet, check whether its Service guards are actually arming (D-Bus reachability on the endpoint), not just whether the dashboard flags them.
• A status row surviving a guard-type revision on the same rule id (e.g. re-authoring a Service rule as Registry) can render a now-unenforced pair as compliant with no “not implemented” marker on that rule — though the yuzu_server_guardian_platform_matrix_stale_total counter below still fires on it. Tracked as #4263, not yet fixed.

The REST /api/v1/guaranteed-state/status endpoint’s errored_rules is now real (#2298 item 6d), derived from the same guardian_agent_rule_status census as the dashboard and intersected against the live rule catalogue (a census row for a since-deleted rule is excluded — the deletion path’s own status cleanup is best-effort, so an unintersected count could otherwise include a deleted rule indefinitely). It is still not the dashboard’s liveness-folded rollup: it does not fold an offline agent’s last state to unknown, so a rule reported errored only by a currently-offline agent still counts. compliant_rules/drifted_rules still return placeholder 0 until full status ingest lands in a later rung, and the fleet endpoint denies a service-scoped API token outright (it aggregates across every agent’s census; a token scoped to one service must not read the fleet-wide count). The fleet route is also management-group-confined (ADR-0017): a global grant sees the fleet-wide count, a group-confined grant sees errored_rules scoped to that operator’s visible agents, and total_rules is never confined (it is the global rule-catalogue size, with no agent dimension). A SIEM/automation integration reading errored_rules should treat it as a raw census-derived count, not the dashboard’s folded compliance posture; corroborate against the /events stream for compliant/drifted state until the status endpoint’s remaining fields are wired.

Resilience policy (enforce-retry behaviour)

When a guard is enforcing and a competing writer keeps reverting the value, a resilience policy governs how the agent re-enforces. The policy travels as string entries in the structured Guard’s remediation.params (no proto change) and is validated + canonicalised by the server on create/update.

curl -X POST https://yuzu.example.com/api/v1/guaranteed-state/rules \
  -H "Authorization: Bearer $YUZU_TOKEN" -H "Content-Type: application/json" \
  -d '{
    "rule_id": "block-rdp-outbound", "name": "Block outbound RDP",
    "enforcement_mode": "enforce",
    "spark":      {"type": "registry-change",       "params": {}},
    "assertion":  {"type": "registry-value-equals", "params": {"hive": "HKLM", "key": "SYSTEM\\CurrentControlSet\\Services\\TermService", "value_name": "Start", "value_type": "REG_DWORD", "expected": "4"}},
    "remediation":{"type": "enforce", "params": {"mode": "bounded", "max_attempts": "3", "resume_after_s": "3600"}}
  }'

Validation is lenient-in / canonical-out. Only the chosen mode’s load-bearing params are range-checked, so a persist rule that carries stray backoff_* values is accepted, not rejected. Values are stored canonical (mode lowercased, numerics as decimal strings), and GET echoes the form you would re-POST. An invalid value (e.g. bounded with max_attempts: 0, or backoff_initial_ms > backoff_max_ms) returns 400 with the structured A4 error envelope (error.code / error.message / error.correlation_id / error.remediation) so an agentic author can self-correct. The same validation runs on PUT — a structured update re-authors the Guard rather than silently dropping the blocks.

Resilience modes apply in enforce mode only (an audit guard always just detects and alerts). event_debounce_ms applies in both modes.

Schema discovery

GET /api/v1/guaranteed-state/schemas returns the static catalog of Guard spark / assertion / remediation types with per-type JSON Schemas — including the resilience subschema above and the discriminated registry-value-equals encoding (the expected format keyed on value_type). It is the machine-discoverable source dynamic authoring forms and agentic clients build against (the same schema the server validates with). Requires GuaranteedState:Read. The response is cacheable: it carries a content-derived ETag, and a conditional request (If-None-Match) returns 304 Not Modified.

curl -H "Authorization: Bearer $YUZU_TOKEN" \
  https://yuzu.example.com/api/v1/guaranteed-state/schemas

Upgrading from a detect-only build

Breaking behaviour change. enforcement_mode defaults to enforce, and enforce mode now writes to the endpoint registry on drift. A rule authored on a pre-enforcement build (when Guardian was detect-only) will begin remediating on the next push after the agents are upgraded — with no further opt-in. Before upgrading agents: audit existing rules (GET /api/v1/guaranteed-state/rules), set enforcement_mode: audit on any rule you want to keep detect-only, and re-push. HKLM-scoped enforce rules require the agent service account to have write access to the key.

Breaking: Guards now reach agents only via a deployed Baseline. If you ran a pre-Baseline Guardian build, a Guard alone no longer enforces anywhere — the push fan-out and heartbeat reconcile source their rules from the union of member Guards of deployed Baselines. After upgrading, any Guard not in a deployed Baseline is silently omitted from every agent push and stops enforcing. To preserve enforcement: create a Baseline containing your active Guards and Deploy it (/guardian → Baselines). Until you do, the fleet converges to zero enforced Guards.

DEX signal observations

The agent runs a fleet-wide DEX signal observer for Digital Employee Experience. Unlike a Guard, it is ruleless: it records reliability/experience signals on a managed endpoint with no rule, scope, or Baseline involved. On Windows it subscribes to 103 catalogued signals across 22 event-log channels (app crashes & hangs, service failures, device stability, boot/resume durations, disk/filesystem, network, identity/sign-in, security, updates, policy & management, printing — see docs/dex-signal-catalog.md for the full catalogue). Each subscription is kernel-filtered and idle-until-signal — no polling, no streaming — in keeping with Guardian’s resource-light, network-kind posture.

Every observation is emitted through the same event pipeline as drift, distinguished by two markers — not a category field:

• rule_id = the reserved sentinel __observation__ (a ruleless observation belongs to no rule).
• event_type = the obs_type (e.g. process.crashed, process.hung, service.crashed, os.boot, network.wifi_drop, logon.aad_token_error).

The dedicated DEX dashboard (/dex) is the operator-facing view over these observations — a hub of measured reliability rates that links into a signal Catalogue, a derived Health score, and cross-OS Trends, with per-app and per-device drill-downs. You can also query the raw events with the events endpoint, filtering on the sentinel:

curl -H "Authorization: Bearer $YUZU_TOKEN" \
  "https://yuzu.example.com/api/v1/guaranteed-state/events?rule_id=__observation__"

Each row carries a human detected_value sentence plus a structured detail_json with uniform keys (subject, reason, symbolic, component, metric, platform); severity is a fixed info (DEX applies its own experience framing and ignores Guardian severity); expected_value is empty (an observation has no desired state); guard_type is the obs_type’s family (process, service, os, network, …). Observations inherit the same retention window as drift events.

Scope and limitations:

• Windows, macOS, and Linux. The Windows observer arms the full event-channel catalogue; the macOS observer (dex_macos_collector.cpp) covers ~10 of 11 headings via DiagnosticReports / unified log / IOKit; the Linux observer (dex_linux_collector.cpp) targets headless servers — Performance (/proc/stat, /proc/meminfo) and Hardware & storage (statvfs) — with workstation-only headings (battery, Wi-Fi, display, GUI app dialogs) staying at zero, not gapped.
• Windows 11 / Server 2022+ field layout. Extractors target the field layouts those builds emit; an unexpected layout degrades to a counted occurrence with empty subject rather than a dropped or crashing signal. Channels absent on a given SKU fail to arm individually and are skipped (the rest still collect).
• Forward-only. Signals are captured from subscription time onward; events written before the agent armed (e.g. a bugcheck during boot) are not back-filled.

Privacy — what is collected. Observations carry the failing entity (app/service/printer name, etc.), failure codes, and numeric metrics. User content is dropped at the edge, before anything leaves the device: usernames (profile/VPN/RDP/service-logon/Defender), file and document paths, DNS query names, URLs, AAD message text, and .NET stack frames are never extracted. To opt a fleet — or an individual endpoint — out entirely, start the agent with --dex-disable (or set YUZU_AGENT_DEX_DISABLE=1): the observer never arms and no DEX signal telemetry of any kind is collected (all 103 signals, all channels). This is a deploy-time agent setting, not a server-side runtime toggle. Whether observers armed across the fleet is surfaced by yuzu_fleet_agents_dex_observer_disarmed (see Observability). See the DEX dashboard guide and the behavioral-telemetry / works-council posture in docs/enterprise-readiness-soc2-first-customer.md.

Retention

Guardian events are pruned on a rolling window. The default is 30 days. Override with --guardian-event-retention-days at server start (or set YUZU_GUARDIAN_EVENT_RETENTION_DAYS). See the Retention Settings table in the server administration guide. A runtime PUT /api/config/guardian_event_retention_days does not change the enforced window: the running store captures its retention window at construction and never re-reads it, so the PUT returns "applied": true while the reaper keeps using the startup value (#483). Set the startup option or environment variable and restart. See REST API -> When a change takes effect.

Missing-anchor decline (#2579). The reap pass keeps a durable clock-guard reading (gc_meta) so a skewed host clock can’t mass-expire live rows. On a fresh deploy, before any pass has reached a verdict, a partial expiry with no comparison point yet declines once rather than deleting — the same missing-anchor trigger AuditStore’s retention guard adopted, and the same answer (decline): a mis-timed pass here would destroy non-regenerable Guardian/DEX compliance evidence. Nothing is deleted on the declining pass; the next pass has an anchor and proceeds, paced by the cap as always. Counted at yuzu_server_guardian_reap_passes_total{result="declined_no_anchor"}; the YuzuGuardianReapAnchorNotSurviving alert (docs/prometheus/yuzu-alerts.yml) fires on increase(...[24h]) > 1 — a single decline is the ordinary fresh-deploy case, a climbing value means the anchor is not surviving.

Long-retention horizon (#2663). The reap pass also probes for rows implausibly far in the future (a forward-skewed clock’s own writes), and that bound scales with the store’s own configured retention — never a fixed guess — so a long-retention deployment (e.g. a multi-year compliance configuration) does not have its own honest, live rows misclassified as implausible under a stable or growing retention window. Shrinking --guardian-event-retention-days across a restart can still leave older rows stamped under the previous, larger window past the new horizon — a self-correcting decline, not data loss (see Retention above on when a change takes effect).

Reconnect replay traffic (durable lifecycle journal)

This is prefer_spark-gated and currently inert in production (prefer_spark=false). It is documented here ahead of that flag flipping so a pilot’s network monitoring is not surprised by it later; see Behaviour changes for how you’ll be told when that flag actually flips.

Once active: on every agent reconnect (a new Subscribe stream), the agent re-sends its full backlog of not-yet-expired lifecycle-journal entries to the server. There is no delivery cursor, so an entry keeps being re-sent - on every reconnect, and periodically while connected - until it ages out of the retention window below. If your SOC or NDR tooling watches for reconnect-correlated outbound bursts, this is genuine, deliberate traffic, not beaconing: it rides the existing agent-server gRPC/mTLS Subscribe stream (the same stream carrying commands and other Guardian events); there is no new port, protocol, or egress destination.

There is no dedicated on/off switch for this traffic specifically. Once prefer_spark is active, its only lever is --spark-disable (see SparkEngine below) - a boot-time agent setting that also turns off spark-based detection, not just journal replay.

Registry detection under spark (also prefer_spark-gated, inert today). Documented ahead of the flip for the same reason. Once active, a change to a watched registry key produces the usual drift.detected evaluation immediately and one further re-evaluation a few tens of milliseconds later, when the mechanism has re-armed its notification (two invalidation hints for one change; the second re-reads the key and, with the default 1000 ms event_debounce_ms, folds into the next drift row’s collapsed_count rather than producing a second row - a rule debounced below roughly 200 ms shows a second, uncollapsed drift row instead). A key that takes longer than 50 ms to arm or re-arm (a slow or contended hive) is reported guard.unhealthy and then guard.healthy when it establishes, so Registry rules on such hosts can show short health flaps that the legacy backend never reported. Neither behaviour needs an operator action; both are recorded so dashboards and alert thresholds can be read correctly when the flag flips.

Windows Service detection under spark (also prefer_spark-gated, inert today). Documented ahead of the flip for the same reason. watch() itself is instantaneous (a queue push); establishment - opening the service against the Service Control Manager - happens asynchronously on a bounded worker. Registering for change notification and delivering the first observed state then happen on the mechanism’s own thread, as before - that registration step is not isolated by this change (a Win32 requirement ties it to the mechanism thread), so a service whose registration itself hangs can still delay sibling services on the host, a narrower, still-open residual of the original hazard this restructure otherwise closes. Expect that delay to be dominated not by the underlying Windows API cost (well under a millisecond per call) but by the mechanism’s own poll cadence: end-to-end, arm to first observed state runs roughly 50-65 ms on measured hardware (idle or under SCM load - the two are within noise of each other), because the mechanism only checks for a completed lookup once per its fixed 50 ms tick rather than being woken the instant one finishes. This is a structural property of the current design, not a fault condition, and not something an operator needs to act on - but a pilot’s own tooling expecting sub-millisecond Service Spark delivery (matching the raw API cost) should be calibrated to this instead. A genuinely absent service resolves to an ordinary Stopped state, never a fault; a service the agent cannot query (permissions, a backend SCM error) is reported guard.unhealthy until it resolves, mirroring Registry’s health-flap behaviour above but driven by SCM query failures rather than hive contention. Retried lookups re-poll on a fixed cadence and do not extend or reset an already-in-flight health window once the mechanism has a definitive answer for that attempt, so a persistently-absent or persistently-unreachable service settles into a stable, non-flapping state rather than repeatedly re-arming its own grace period.

The traffic is bounded and small by construction - and every number below is something your own network capture can check directly, independent of any Yuzu-side tooling:

• Rate: a per-agent token bucket refills at 0.1 batches/sec (1 batch every ~10 seconds), so one reconnected agent adds at most 1 batch/10s of steady-state traffic. A fleet of N agents therefore adds at most N * 0.1 batches/sec server-wide in steady state. At an average retained batch size of roughly 32 KiB (derived from the 32 MiB / 1000-batch retention cap below - most batches in practice run smaller), that steady-state rate works out to on the order of 3 KB/s per agent.
• Startup burst: each agent process gets a one-time allowance of 5 batches it can send immediately on its first reconnect, before the steady-state rate kicks in. This allowance is a single process-lifetime budget, not reset on every reconnect - a flapping agent reconnecting every few seconds does not get a fresh 5-batch burst each time; it draws down the same slowly-refilling budget.
• Fleet-wide reconnects: because each agent’s burst is independent and tied to its own process lifetime, a mass-reconnect event - a scheduled fleet restart, or many agents recovering together from a brief network blip - can see every freshly-started agent contribute its own up-to-5-batch burst at roughly the same time: up to 5 * N batches for N freshly-started agents, before the fleet settles to the steady N * 0.1/sec rate. This is still bounded (5 batches is small per agent) and one-time per agent process, not a sustained multiplier.
• Per-pass cap: a single reconnect (or periodic tick) scans at most 128 batches of backlog per pass, regardless of how much unexpired backlog exists - this bounds the CPU cost of each pass to a fixed ceiling. The token-bucket rate above separately bounds how many of those scanned batches actually go out on the wire; the per-pass cap limits work done, not just bytes sent.
• Retention window: the journal itself keeps at most 7 days of not-yet-expired entries, evicted oldest-first (also capped at 1000 batches / 32 MiB per agent). This is a separate, fixed, non-configurable window from the Guardian event retention described in Retention just above - that one is the 30-day, operator-configurable server-side event store; this is the agent-side journal that feeds it, and the two numbers are unrelated. Replay can therefore never cover more than the last 7 days since an agent’s last arm/disarm activity, never unbounded history. Worst case - an agent reconnecting with a full 1000-batch backlog - drains at the steady 1-batch-per-10s rate in a little under 3 hours, not all at once.
  • This 7-day clock runs on the agent’s own local schedule, independent of whether it can reach the server. An agent that is network-partitioned (but still running) for most of a week can age out and lose entries on schedule, before it ever gets a chance to reconnect and replay them - “reconnects within 7 days” is not quite the right mental model; “has had a chance to attempt a send within 7 days of each entry” is closer. A loss like this is still a rare, counted event, not silent - see the eviction-loss note in Guardian metrics - but the count itself has the same visibility limits described immediately below.

Observability status: a fleet-wide Prometheus rollup for this traffic EXISTS today - the yuzu_fleet_guardian_journal_* gauge family on the server’s /metrics endpoint (see metrics.md): fleet sums of every yuzu.guardian_journal_* heartbeat counter (pages, records paged, the eviction-loss counters, and the rest), plus three *_seconds_max age gauges (worker staleness and the headroom-blocked episode age). Like everything in this section it is live only once the Spark detection path is the authoritative backend (prefer_spark); pre-cutover the family is deliberately absent. What is genuinely still missing: a dashboard panel, a per-agent REST/MCP view of these counters (fleet aggregates only today - no per-endpoint attribution axis), and durability (the counters are in-memory per agent process and reset on restart, so they are telemetry, not an audit trail). Those remain tracked under #2298 and the per-agent-axis work. Until the dashboard lands, read the rollup directly from /metrics, and treat the bounds above as falsifiable predictions to check against your own capture.

Observability

Guardian surfaces readiness and event counts via the standard observability endpoints:

• /healthz (authenticated) reports stores.guaranteed_state and stores.baselines alongside the other store-health entries. Both stores are on the shared Postgres substrate (schemas guaranteed_state_store / baseline_store, ADR-0038 / ADR-0055). These entries report whether each store opened at startup (a store’s open state does not change afterwards); construction/backfill failing at boot refuses to start (non-zero exit) rather than serving /healthz at all. A Postgres outage while the server runs shows as stores.pg_reachable = error and status = "degraded" — the one runtime check (HA WS-8).
• /readyz includes both guaranteed_state_store and baseline_store in the per-store readiness conjunction (startup state), and pg_reachable for Postgres reachability at runtime. Either failing takes the replica out of rotation.
• yuzu_server_guardian_baselines_total reports the number of persisted Baselines.
• yuzu_guardian_push_rule_excluded_total{reason} — two reasons today, depth_exceeded and invalid_rule_id (#4665, described in the next bullet); both share the same per-rule rate-limited log line described below. reason="depth_exceeded" (json-dump-depth-guard fix) counts a rule whose stored spec was excluded from a push (both the periodic heartbeat reconcile and the on-demand push fan-out triggered by a rule create/update/dashboard toggle) because it could not be safely parsed. A rule this happens to keeps looking like an ordinary rule everywhere except its own detail page (which shows an “invalid data” state) - if a Guard appears not to be enforcing on any agent, check this counter and that page before assuming a scope/targeting problem. This can only happen to a row poisoned before the #4478 depth-guard fix shipped, or by direct database manipulation; the create/update routes reject an over-deep body outright. The counter increments on every exclusion, unconditionally, but its paired log line does not (#4497/#4499): a rule’s first exclusion logs immediately, then that same rule logs again at most once per 60 seconds no matter how many further exclusions it has in between - so a counter that keeps climbing with no matching log line is expected behavior, not evidence of a separate bug. Rate-limiting is now tracked per rule (a bounded 256-rule LRU cache), so a second, distinct poisoned rule always gets its own immediate first log line rather than being masked by an unrelated rule’s ongoing suppression window - but the cache is a per-rule pacing bound, not a fleet-wide log-rate ceiling: above 256 simultaneously-poisoned rules cycling through the cache in a repeating reconcile order, pacing for that whole set collapses to a hard cliff - every rule logs on every reconcile pass (verified: 257 rules cycling produced 257/257 logged on every pass, a 0% suppression rate), not merely “somewhat more often than once per 60 seconds” - and a burst of many distinct rules newly poisoned at once each still logs immediately (by design - a genuinely new rule is never suppressed). In that regime the log line fires once per exclusion, so its rate equals the reconcile rate (heartbeat rate times connected agents): bounded, not runaway, but with no pacing at all - worse than the pre-#4497 shared sampler, which floored the same regime at roughly one line per hundred exclusions. The underlying exclusion EVENT rate (how often build_agent_push encounters the row at all) is unchanged by either fix; only the LOG LINE’s rate in this specific regime is worse than before. Whichever rule(s) are affected, the reliable way to identify them is each rule’s own detail page (the “invalid data” state above), not the log stream.
• reason="invalid_rule_id" counts a rule excluded because its rule_id fails the create-time charset/length contract ([A-Za-z0-9._-]+, ≤256 bytes, #4665) — only reachable by a row created before that contract was enforced (creation itself has rejected a non-conforming rule_id since #4665 shipped). This is a hard cutover, not a preservation mechanism: the excluded row is never included in a push again, and the agent’s full_sync unconditionally rebuilds its whole active rule set from what the push contains, so the first full_sync after upgrade disarms the excluded row on every agent that was enforcing it (logged at warn on the agent side too). Every other rule in the same push is unaffected. Find and remove non-conforming rows before upgrading — see docs/user-manual/upgrading.md’s #4665 Behaviour change entry for the detection query and for which delete surface (MCP delete_guardian_rule, REST DELETE /guaranteed-state/rules/{rule_id}, or a direct database delete) reaches which shape of violation.
• yuzu_server_guardian_platform_matrix_stale_total{spark_type} (server) fires when the dashboard’s not-implemented fold above (Compliance overview) finds a pair the server’s guard-type/platform matrix calls unsupported, but the agent already reports real status for it — the matrix disagreeing with what an agent has actually observed. Render-time only, no dedicated alert (a diagnostic signal for future matrix drift, checked on incident, not paged on) — see metrics.md.
• yuzu_fleet_agents_dex_observer_disarmed (server) counts agents — DEX enabled, on any platform with an observer (Windows/macOS/Linux) — whose signal observer is not currently healthy: it failed to arm at startup, or its health flipped at runtime (Windows: a channel subscription died — an EventLog-service restart or channel ACL change; Linux: persistent /proc/stat read failure). > 0 means reliability telemetry is off or degraded on that many endpoints; investigate. Agents started with --dex-disable, or on a platform without an observer, are excluded, so this is a genuine fault count, not an opt-out count. Caveat: it does not detect a host where the underlying reporter is disabled (e.g. Windows Error Reporting off → no Event 1000, observer still armed) — verify coverage with a synthetic event on a representative host. (Agents report arm-state in each heartbeat; the server rolls it up — the agent has no /metrics endpoint.)
• yuzu_fleet_dex_observed_total (server) is the fleet-wide sum of DEX signals observed since each agent started. It is a resetting gauge, not a monotonic counter — it drops when an agent restarts, so do not apply rate()/increase(). The per-signal records live in the events store and the /dex dashboard.
• yuzu_server_guardian_proj_failures_total (server) counts DEX observation projection failures. The source event is always preserved (the read model degrades, never destroys); > 0 means /dex is under-counting — investigate (commonly a stale-schema dev DB). yuzu_server_guardian_observations_reaped_total is the disposal-evidence counter for reaped DEX projection rows.
• Guardian T_server (server), Guardian T_wire and Guardian T_detect (agent) are info-level diagnostic log lines (#4606): per-event latency waypoints for the Spark cutover benchmark, correlated by event_id. They are plain log lines, not metrics or an audit trail - no alert or dashboard consumes them. Volume tracks Guardian event volume; --log-level warn suppresses them (and every other info line). If you consume them, join on the agent and event_id (the agent-side lines carry no agent field) and never on log-file line order (a T_wire line can precede its own T_detect line), and read sent=0 per path: on the Spark path it means the local write failed and the entry is normally retried under the same event_id (unless a newer observation replaced it or the rule was dropped meanwhile; a down stream logs no line at all), on domain=legacy it means the link was down or the write failed and the event was dropped. The fields are described under “Reading the Guardian T_* diagnostic lines” below; how to read an orphaned line (one with no partner) is in the header comment of agents/core/src/guardian_spark_timing.hpp; the operator-facing summary is the Upgrade note in server-admin.md.
• Broader Prometheus metrics — rule push counts, agent apply latency, parse errors, and a fleet compliance-state distribution (compliant/drifted/error/unknown) — are on the roadmap alongside agent-side enforcement metrics.

Reading the Guardian T_* diagnostic lines

All *_ns fields are nanoseconds. *_wall_ns, recv_ns, committed_ns and agent_ns are wall-clock instants (nanoseconds since the Unix epoch); *_mono_ns come from a steady clock with an arbitrary origin and are only for intervals within one process. A difference between two instants taken on different hosts includes their clock skew. Every line is best-effort: one can be missing because logging failed, the log level was raised, a file rotated or the process stopped.

The ids in all three lines are neutralised and length-limited the same way (the Upgrade note in server-admin.md gives the rule, and lists the other log lines that still print a rule id as authored).

The trigger fields describe the event that started the pass and are repeated on every entry that pass stages, including an entry for a rule that event did not change (a refresh or a first verdict), so do not treat every trigger_present=1 line as a detection of its own rule. On the agent the order mechanism, handler, detect, fire holds in program order; fire before wire, and wire before recv_ns, are not guaranteed (they are stamped on different threads and hosts), so a small negative difference between them is possible.

SparkEngine — the next-generation detection engine (observe-only)

Guardian is migrating its event-driven detection (file / service / registry watches) onto a new SparkEngine in incremental rungs (ADR-0021 Stage-2). Today the agent stands SparkEngine up observe-only: it runs and reports its own health, but it is not wired to drive Guardian — the existing detection path still does all detecting and all enforcing. Nothing about how your Guards detect or remediate changes in this build. The migration is deliberately staged so each rung can be measured and rolled back before the next.

--spark-disable / YUZU_AGENT_SPARK_DISABLE is a boot-time agent opt-out: when set to 1, SparkEngine is never instantiated and watches nothing — but the heartbeat still carries the posture itself (spark_running=0 + spark_disabled=1), so the fleet can tell your deliberate opt-out apart from an engine that failed to start. (Emitting nothing at all is what used to make a fleet-wide boot failure invisible.) Like --dex-disable it takes effect only at agent start — a live change is ignored until the agent restarts. It is a deploy-time agent setting, not a server-side runtime toggle, and it does not affect the enforcing legacy path. At rung 1, leaving it unset (the default) is recommended — the engine is observe-only and safe; there is no rung-1 reason to disable it (if a boot exception is hit, e.g. thread exhaustion, the agent already degrades to no-spark on its own and continues). On Linux/systemd installs, persist it across restarts and package upgrades by adding YUZU_AGENT_SPARK_DISABLE=1 to /etc/yuzu-agent/yuzu-agent.env (root-owned 0600, not shipped - deliberately NOT the shared /etc/yuzu/, which a co-installed yuzu-server package owns and re-asserts on every upgrade; the unit loads this file via EnvironmentFile=-, so absence is a no-op) - write one bare NAME=value line; systemd does not strip a trailing # comment, and a malformed value stops the agent starting. Restart the agent to apply; re-enable by removing the line, not by setting it to 0.

Per-rung default posture (recorded here so an upgrade never silently changes what enforces — this is safety-critical, keep it accurate against docs/spark-stage2-guardian-consumer-design.md):

The rung-2 row is the load-bearing one: by default rung 2 detects but does not enforce for spark-managed rules (an accepted, greenfield burn-in window), and an operator who needs continued enforcement during that window sets --spark-disable to keep the enforcing legacy path. At rung 1 the agent logs the active detection path at startup (SparkEngine: instantiated OBSERVE-ONLY … Guardian detection path = legacy IGuard (enforcing)).

macOS note: a macOS agent reports spark_running=1 with an empty mechanism capability — all three mechanisms are Windows/Linux facilities, so SparkEngine on macOS runs with zero mechanisms by design. This is expected, not a fault: the absence of yuzu_fleet_spark_mechanisms{os="darwin",...} series is the correct signal. Do not file it as “spark broken on my Mac”.

Fleet observability (all os-labelled — File/Registry are Windows-only, Service is Windows + Linux, macOS has none — so never aggregate across os; absent means no reporting agent of that OS/mechanism, never “0 = healthy”). An agent shows no spark series at all (ABSENT) in three cases: it pre-dates rung 1, it is dead/not heartbeating, or it is mid-graceful-shutdown (the final beats of a stopping agent deliberately ship no spark keys, so a clean restart never counts as a failure):

• yuzu_fleet_spark_reporting{os} — agents actually running the engine (the denominator).
• yuzu_fleet_spark_disabled{os} — agents deliberately running with --spark-disable. An operator decision, expected to be non-zero in some fleets — do not alert on it.
• yuzu_fleet_spark_failed{os} — agents where the engine was enabled but boot-time instantiation threw, so the agent degraded to no-spark. A fault, not a choice — alert on this one. It is deliberately separate from _disabled: without it, a fleet-wide spark boot failure would be indistinguishable from a deliberate opt-out (a failed agent used to emit nothing at all), which would defeat the entire point of an observe-only rung.
• yuzu_fleet_spark_mechanisms{os,mechanism} — agents whose spark capability includes that mechanism. Counts only mechanisms that are registered and functional: one that started but could not bind its OS facility — most commonly a containerised Linux host, which has no systemd system bus — is reported inert and excluded, because every watch on it would be refused. A Registry sweeper or File worker that fails three consecutive passes is likewise reported inert and excluded, because it cannot serve watches until a pass succeeds (its watches are still accepted), and is counted again once a pass succeeds. An OS reporting but with no {mechanism} series either does not support it, or cannot use it there.
• yuzu_fleet_spark_armed_faulted{os} — armed watches a mechanism reported deaf (a live gauge; > 0 means detection is silently down for that many watches).
• yuzu_fleet_spark_watch_rejected{os,mechanism}, …_quarantined{os,mechanism}, …_slow_op{os,mechanism} — per-type health counters (watch-cap rejection / mechanism quarantine / slow ops). quarantined should stay 0 (page-worthy).
• yuzu_fleet_spark_unsupported{os,mechanism} (F7, #2298 rung 2) - rules currently classified unsupported: a known spark type with NO mechanism registered on that host, OR one that IS registered but BOOT-INERT (start() refused to bind its OS facility - #4685), enforced by neither backend either way. A live gauge, not a counter - it can legally decrease. Routine, not page-worthy (every rule on macOS reads this once spark is preferred, since macOS has no mechanisms). A TRANSIENT runtime-degraded episode (a Registry sweeper or File worker failing consecutive passes) does not land a rule here (#4685 fixed this): the rule stays logically armed/committed via spark for the whole episode - not actually detecting, since the mechanism cannot serve a watch until a pass succeeds - reporting None coverage until the mechanism’s next successful pass, with no push or restart needed to recover - watch yuzu_fleet_spark_mechanisms drop that mechanism from its capability CSV instead, never this gauge. Telling routine from anomalous: a nonzero reading on an OS/mechanism combination that SHOULD have a mechanism (file/registry on Windows, service on Windows or Linux-with-systemd) means a registration failure or a boot-inert registered mechanism, not a routine cross-platform gap - cross-check yuzu_fleet_spark_mechanisms{os,mechanism} and the agent log (see Diagnosing an inert File worker or Registry sweeper): that mechanism’s start() boot lines (a mechanism that could not bind its OS facility - spark_file: CreateIoCompletionPort failed, spark_registry: CreateThreadpool failed, spark_service: system bus unavailable, and similar). The agent’s own classification log line says which case it is: ... is registered but initialisation refused at start() (boot-inert) ... at warn level for a boot-inert registered mechanism, versus ... has no mechanism on this host ... a routine cross-platform gap, not an error at info level for one never registered at all (reconcile_rule_locked’s Unsupported branch, guardian_engine.cpp). yuzu_fleet_spark_mechanisms counts registered-AND-functional mechanisms - it still excludes a runtime-degraded one too (a SEPARATE, wider reading than this gauge’s boot-inert-or-absent one: #4685 keeps the capability CSV reading the union while narrowing only Guardian’s own classifier) - so a 0 there does NOT by itself distinguish “never registered” from “registered but boot-inert” - it is consistent with either. The agent log is what actually disambiguates the two.
• yuzu_fleet_spark_watch_faults{os}, …_queued_dropped{os}, …_consumer_errors{os} — engine-level fault / drop / handler-error sums.

A spark-capable agent’s wire state maps to exactly one of four postures, never more than one at once: running (_reporting), failed (_failed), disabled (_disabled), or absent — no yuzu.spark_* tag at all, meaning a pre-rung-1 agent, one that has stopped heartbeating, or one mid-graceful-shutdown.

At rung 1 the mechanism- and consumer-health counters (armed_faulted, watch_rejected, quarantined, slow_op, watch_faults, queued_dropped, consumer_errors, unsupported) are all 0, because nothing is armed; they go live as later rungs arm rules. reporting, disabled, failed and mechanisms carry live signal today - none of them depends on a consumer being armed. The agent has no /metrics endpoint - these are rolled up server-side from heartbeat tags.

yuzu.guardian_backend (F7, #2298 rung 2, agent heartbeat tag, not a fleet gauge): reports which detection backend a device is actually enforcing with - spark, legacy, unwired, or spark_failed. Unlike the counters above, this one IS live today: it reads legacy fleet-wide, since prefer_spark is not yet enabled anywhere in production. It exists to let the server distinguish spark_failed (agent-wide - nothing enforced) from a routine per-rule unsupported classification, which the fleet gauges above cannot do on their own.

Alerts. One rule ships ACTIVE today: YuzuSparkBootFailed (group yuzu-fleet-spark-rung1 in docs/prometheus/yuzu-alerts.yml) — yuzu_fleet_spark_failed > 0 for 30 minutes, severity warning (a ticket, not a page: the posture is agent-self-reported, so one forged or buggy agent can trip it). _failed is a per-sweep state gauge, not a latching counter sum, and boot failure is genuine rung-1 signal. The reviewed yuzu-fleet-spark counter alert group ships commented-out until rung 2: those counters are all 0 at rung 1 (an enabled rule could only fire on a forged heartbeat), and they are fleet sums of monotonic per-agent counters — a > 0 alert on them latches: once any agent reports a non-zero value it stays firing until that agent’s process restarts (only armed_faulted, a live self-healing gauge, is latch-free). The rung-2 templates use increase(...[15m]) > 0 over counter-typed metrics to avoid this; enable them when rung 2 arms real rules.

Wedged-rule hold (rung 9c PR-5e, #4221, replaced by retry suppression in #5459; dormant until Guardian enforces through SparkEngine, same as the rest of this section). Once enforcement is live, a rule whose arm genuinely hangs past its backend deadline (“wedged”) HOLDS its policy generation: the agent does not acknowledge the generation while the wedge is outstanding, however many times the server re-sends it. Earlier builds acknowledged the generation after three identical re-applies of a wedged rule; that waiver is removed, because a late failure of the hung arm then left the rule unarmed under an acknowledged generation with nothing to retry it (#5459). This is retry suppression with a safety valve, not a guarantee of enforcement; the known limits are listed below. Three terms are used below. An application is the agent’s record of one run of applying a pushed policy; it tracks which of the pushed rules have armed. A claim is the agent’s record of one arm or disarm call that is in flight or hung for a watch target. A congestion expiry is a rule that waited for a free start slot and was not started by its deadline.

• Symptom. yuzu.guardian_arm_failed/yuzu_fleet_guardian_arm_failed (see docs/user-manual/metrics.md) stays the signal that this specific rule is failing, for as long as the current application stands, and yuzu.guardian_generation (a per-agent heartbeat tag) lagging the server’s target is NECESSARY for a held wedge but NOT specific to one: an ordinary arm refusal, a congestion expiry or a latched apply failure also holds the generation, so corroborate with arm_failed/arm_pending, the compensation tags below and the agent log before concluding wedge. A call that never returns shows only a steady arm_failed, the generation lag and one agent-log warning per affected rule per application (a suppressed re-push starts no application and adds none); the compensation age and deadline tags exist only after the hung call has returned late and a compensating teardown is owed.
• What the agent does while the hold lasts. Each identical re-push from the server (same generation, same content, same full_sync setting; full_sync=true, a “full push”, makes the agent apply exactly the pushed set: it removes the rules the push omits and rebuilds any rule that changed or is not healthy, and leaves a healthy, unchanged rule alone (see “What a full push does on the agent” under Push to agents), while full_sync=false only adds or updates the rules the push carries and leaves every other rule as it is) is answered without tearing anything down: the open application survives and the healthy rules in the same push are not re-armed. This also holds for a wedge whose late result has already returned and is being compensated (#4472: the arm call finally returned but nobody adopted its result, because the rule was withdrawn or superseded while it was in flight or the commit threw; the agent then owes a compensating teardown of that late subscription).
• How it ends. Three ways, or an agent restart (see the next point).

  • (1) The hung call returns. If its late result succeeds and is adopted, the agent acknowledges the generation by itself. If the late result fails, or is not adopted, the wedged claim ends (for a result that is not adopted, once the compensating teardown has finished) and the next identical re-push from the server re-arms the rule through the normal path. The rule can then end armed without operator action, provided the server is still re-pushing, the agent is connected and none of the known limits below applies.

  • (2) A full push omits the wedged rule. The agent then removes the rules the push omits, rebuilds any rule that changed or is not healthy and leaves the rest alone, under a fresh application. Get the rule out of the push by disabling it, or by narrowing its scope or OS target: both are live properties of the rule, so the server’s next automatic re-push carries them, normally at the next heartbeat (that automatic re-push, while an agent’s generation lags, is itself a full push). Removing the rule from the deployed Baseline is slower: the push is built from the member set captured at the last deploy, so that edit stays staged until someone with Push authority re-deploys the Baseline (the dashboard Re-deploy sends a full push to every agent). REST POST /api/v1/guaranteed-state/push and MCP push_guardian_rules send a full push only when full_sync is true (they default to false). They need GuaranteedState:Push, refuse a service-scoped API token, may ask an interactive REST session for MFA step-up, and take an optional scope (a scope expression, for example hostname == "web-prod-01") that limits the push to the matching agents. A full push that changes nothing for that agent is identical to the automatic re-push and is suppressed like it, so change the rule first.

    Warning: disabling the rule stops enforcing it everywhere it is deployed, narrowing its scope or OS target stops it on the agents that drop out, and a Re-deploy re-applies every agent (see Costs) and also publishes every other staged member edit on that Baseline. On the held agent, any other armed rule of the same mechanism type that the full push changes, or that is not healthy, is torn down by it and queues behind the hung call, then expires (the Same-type condition below): it stays unenforced until the call returns or the agent restarts, and its expiry is never suppressed, so the agent gets a full re-apply on every re-push (about one per heartbeat), each carrying the baseline exposure described under the Safety valve below (a Spark capture that was lost, never written or overwritten; see “What can still go wrong” under the file-hash-equals baseline note).

  • (3) The wedged rule is edited so that what it watches changes (its file path, service name, or registry hive and key). It is then a different watch, is not matched to the hung call, and is armed afresh; a rule edit is live, so the next automatic re-push carries it. A registry rule’s value name and expected value are not part of what it watches, so editing them does not do this (see the known limits).

  • Same-type condition for (2) and (3). While the hung call is still inside a mechanism call, it holds that mechanism type’s call lock (the agent runs one watch or unwatch call at a time per type, for example File, Registry or Service). The call holding that lock can be ANY hung watch, unwatch or compensating teardown of that type, including one that the wedged rule being exited was itself only queued behind, not necessarily the wedged rule’s own call. Any OTHER rule of the same type in the new push, and the edited rule itself under (3) when the edit leaves its type unchanged, then queues behind the hung call and expires, so it holds the generation on a failure of its own (a congestion expiry is re-applied in full on every re-push, not suppressed). The new application is therefore acknowledged only if no other arm of that mechanism type is in it, or the hung call has already left the mechanism. Otherwise the generation stays held until the hung call returns or the agent restarts. Rules of other types are not affected.

  • What does not end a hold. A new policy generation, a re-deploy of an unchanged Baseline, a different full_sync setting, or an edit that leaves what the rule watches the same (its expected value, for example). The push is applied in full; an unedited wedged rule is recognised as the same hung call and counted as failed again, and an edited one (same watch target, changed content) queues behind the hung call (#5512), so the new generation is held again with a fresh budget of suppressed re-pushes. A push with full_sync=false (the REST and MCP default) is the wrong tool for ending a hold: it leaves a rule that was removed from the Baseline armed, or wedged, on the agent. It still carries the whole deployed set for that agent, so the flag value is not itself dangerous. A delta push that omits a rule that is still unresolved no longer drops that rule’s retry obligation (#5547): the agent carries the omitted rule’s unresolved outcome into the new application, so the generation stays held until a full push, or a push that names the rule, replaces it. A carried wedge or arm failure also stays counted in yuzu.guardian_arm_failed, whereas a carried synchronous failure holds the generation without being counted there (see the known limits for what that does and does not guarantee).

  • After an exit. Neither (2) nor (3) releases the hung call itself: it keeps its watch key (and, while it is inside a mechanism call, that mechanism type’s call lock) until it returns or the agent restarts. While the rule stays omitted, a late success is torn down rather than adopted. Adding the same rule back with the same target attaches to the same hung call and is held again at once, and if the call then succeeds, that late result is adopted. A different rule that watches the same target is refused with the message “spark key wedged” and counts as an ordinary arm failure. After a successful exit (the new application acknowledged, see the same-type condition) the agent’s generation catches up and yuzu.guardian_arm_failed returns to 0.

• Restarting the agent ends the hold only if the target is no longer hung, and is not always the right remediation: it re-arms the rule, which resolves a transient wedge (a temporarily stuck watch or backend call) but does nothing for a permanently wedged target (a dead host or service) - restart simply re-arms against the same dead target and re-wedges. Check whether the underlying target is transient or permanently unreachable before restarting on a sustained arm_failed alert (an alert you define; none ships). A restart does not re-open a generation that an earlier build had already acknowledged by the old waiver. A rule whose re-arm fails during the restart itself is caught up by the server (known limit 3 below, #5513). A teardown that never returns is cleared by restarting the agent, but a restart clears the stuck call, not necessarily its cause; if the stuck mechanism is the cause, restart with --spark-disable, which takes effect at boot only.
• Safety valve. After 10 suppressed re-pushes the agent applies the next one in full anyway: roughly every 330 s at the default 30 s heartbeat (the count is of pushes, not a clock: the push interval is the smallest multiple of the heartbeat interval that is at least 25 s, so other heartbeat intervals scale, for example about 275 s at a 25 s heartbeat, later if a heartbeat arriving just under 25 s after the last push delays that push to the next heartbeat). That full re-apply does not release a hung call: the wedged rule is simply observed again. It has two side effects. It re-arms every rule in the push that is not healthy and unchanged (a healthy, unchanged rule is skipped since #4415), so it can recapture a Spark-first FileHashEquals baseline that was lost, never written or overwritten (a file-hash Guard with no expected hash takes the first hash it sees as its known-good baseline; one the agent lost to a crash, to an allocation failure while noting it (when the retries at the re-arm also fail), or to a re-authored path is captured again by a forced re-arm, so a file that drifted in between can be accepted as the new baseline). A capture whose write is merely failing is not in that group: it stays waiting in memory and the new arm is handed it directly. Every push writes the waiting captured-but-unsaved baselines within its pass budget (the remainder is handed to the new arm from memory, and the background worker writes it next) and every arm re-seeds from the saved record, so a healthy store is not exposed apart from the cases listed under “What can still go wrong” in the file-hash-equals baseline note above (#4045). And while a call on the same mechanism type is hung, it can withdraw a rule of that type that it rebuilds, because its disarm queues behind the hung call (the engine serialises calls per mechanism type); its re-arm may then expire as a congestion expiry. Since #4415 a healthy, unchanged rule is not rebuilt, so a healthy sibling of the hung call’s type is no longer withdrawn by the valve’s re-apply; one the push changes, or that is not healthy, still is. Before #4415 the first valve re-apply withdrew every healthy rule of that type, and from then on the push was a mixed failure (below), applied in full on every re-push (about one per heartbeat) for as long as the hang lasted. A re-apply for another reason, such as a push with full_sync=false, adds to either case. A push that mixes a wedged rule with any other failure (a genuine backend refusal, an admission rejection, a congestion-only expiry) is applied in full every time, and that other failure holds the whole generation’s acknowledgment indefinitely regardless of this rule’s own state.
• Costs. The hold is not bounded: a call that never returns holds the generation forever, and the hold ends only as described under “How it ends”. The server keeps re-pushing during the hold, at most once per 25 s per agent and in practice about once per heartbeat (30 s by default) unless the replica sheds it, and there is no server-side back-off. The limit holds even for an agent that alternates between reporting an old generation and the current one: a claim younger than 25 s is kept when the agent reports converged, so alternating reports cannot cause more than one push per 25 s per agent. A replica under load sheds reconciles, which lengthens the cadence for the shed agents. Each re-push writes a guaranteed_state.reconcile audit row for that agent (a synchronous insert on the heartbeat-ingestion path) and one Guardian: reconciled agent ... line in the server log. At the default heartbeat that is about 2,880 audit rows per held agent per day (about 1.05 million per held agent per year, 2,880 x 365), or about 333 rows per second if 10,000 agents are held at once, kept for the audit retention (365 days by default); a wedge with a shared cause, such as a dead shared target, holds many agents together. Audit expiry also feels a large held cohort: once the oldest rows reach the retention age they expire at the rate they were written, and more than about 208 held agents (N / 30 rows per second against the 25,000 rows an hourly retention pass may delete, about 6.9 per second) means one pass per hour no longer keeps up, so the retention job re-runs every 5 seconds while its per-pass cap binds, a ceiling of about 5,000 rows per second, above the 333 per second of 10,000 held agents. yuzu_server_audit_retention_cap_reached_total rises during those catch-up runs; the shipped YuzuAuditRetentionCapBinding alert needs the cap to bind in every 10-minute slice for 30 minutes, so it is not expected to fire from a held cohort whose rows the job can drain (see the Capacity section of Audit Log). Each forced full re-apply (the safety valve, and any re-push that rebuilds a rule; before #4415 a healthy rule of the hung call’s type in the push made every re-push after the first forced one rebuild it) could also push older guard.armed and guard.disarmed records out of the endpoint’s reconnect-replay journal before they are delivered; how much is not measured yet (it is a condition for enabling Spark enforcement, tracked as #5504), see Reconnect replay traffic. Evictions are counted by the agent’s yuzu.guardian_journal_evicted_sent_unacked, yuzu.guardian_journal_evicted_no_send_evidence and yuzu.guardian_journal_evicted_unclassified heartbeat tags. Ending a hold has costs of its own. Any full push re-applies (rebuilds every rule that changed or is not healthy and leaves the others alone, with the effects described under the safety valve) every in-scope agent that has no hold it can suppress against, so a full push sent to end one agent’s hold, including a Re-deploy to the whole fleet, disturbs the others too. And taking the rule out of the push stops enforcing it: disabling it, or removing it from the Baseline, stops it on every agent the Baseline delivers it to, and narrowing its scope or OS target stops it on the agents that drop out. A push with full_sync=false during a hold counts as different content, so it is applied in full, as is the next full_sync=true push after it, and the 10-push budget restarts.
• Known limits. (1) A push with full_sync=false that omits a rule that is still unresolved carries that rule’s unresolved outcome into the new application (#5547; before that change it replaced the open application and dropped the rule’s retry obligation). The carry holds the generation while the omitted rule is unresolved, and, on the Spark path, the agent keeps counting a carried arm failure or wedge in yuzu.guardian_arm_failed (a wedge that was adopted or settled before the agent had retained it, see the flip gate’s AC-15, is carried as an ordinary failure, so it can read as failed after it has armed). A carried synchronous per-rule failure and the application-wide latch (below) hold the generation but are NOT counted in yuzu.guardian_arm_failed, and at the default prefer_spark=false the agent emits no arm gauge at all, so the only signals there are the reported yuzu.guardian_generation not advancing (visible only while a pushed generation is newer than the held one) and the agent log line of the original failure, where one is logged (an exception that unwinds out of a push logs no Guardian-specific line); a delta push’s own apply_rules ok log line counts only that push’s failures (failed=), not the carried hold. What the carry does NOT do is re-attempt the omitted rule: a repeat of the delta processes only the rules it names. Only a full push, or a push that names the rule, retries it. The server’s heartbeat reconcile re-pushes a full push only while the agent reports a generation behind the server’s; the REST and MCP push endpoints do not bump the generation, so a delta applied at the generation the agent already holds is not followed by an automatic full push, and the hold lasts until an operator sends a full push or the generation moves (a rule create, update or delete bumps it and the heartbeat reconcile then re-pushes the trailing agent; a Baseline deploy bumps it and sends a full push to the agents connected to the server that handles it, which retries the hold, except that a wedged rule stays held until its hung call returns or another exit in flip gate AC-1 applies (a full push that drops the rule, an edit that changes its watch target, or an agent restart, each with AC-1’s conditions); deleting a deployed Baseline bumps it and sends a full push too, to the agents connected to the server that handles it, but withdraws every Guard that no other deployed Baseline delivers, so it is not a repair). The carry applies to the legacy path too: an apply failure that has no rule id (a full push’s rule-key sweep, key delete or teardown failure, a failure to persist a rule, or an exception that unwinds out of a push) is now inherited by later deltas and discharged only by a full push whose steps all succeed, where before the next delta forgot it. A clean full push shows that every withdrawal was issued and queued; for a Spark-attached rule that is not the same as its backend disarm having completed, which the acknowledgment ledger does not track. The suppression budget of a held wedge restarts on every replacement, so alternating deltas and full pushes give no elapsed-time bound on a hold (the budget has restarted on every replacement since before this change). A push that names the same rule id twice is rejected before anything is changed, with the error rejecting push: rule '<id>' (name='<name>') appears more than once; the server builds pushes from its rule table, whose rule ids are unique, so only a hand-built push can contain one. The pushes the server builds today carry the deployed inventory, so this does not claim that an omitting delta cannot occur. (2) A wedged rule is matched to its hung call by rule id, watch target and content (#5512, closed by the #4415 change). A rule edited without changing what it watches (its expected value, for example) is no longer discarded: it queues behind the hung call under the same watch target, so when the hung call finally succeeds the rule is armed with the edited content, not the content it had when the call started; a later push of the old content withdraws the queued edit. What counts as the watch target is the file path, the service name, or the registry hive and key; a registry rule’s value name and expected value are not part of it, so editing them does not re-key the rule, which is why such an edit reaches this path. (3) A re-arm that fails at boot (#5513). When the agent’s start-up pass over its cached rules cannot re-arm a rule (a guard that throws while arming, for example when it cannot create its watcher thread, or, on the Spark path, an arm that ends in failure), the agent reports policy generation 0 on yuzu.guardian_generation together with the sparse companion tag yuzu.guardian_boot_rearm_unresolved = 1, instead of the persisted generation it would otherwise report (except while backing off, below). The server sees a generation behind its own and pushes the full deployed set (the ordinary offline-convergence path above, with one guaranteed_state.reconcile audit row per push). The flag clears on the first full push that applies cleanly, including a full push that omits or disables the failed rule. A delta push that omits the failed rule does not clear it (#5547): the failure is carried, so the flag, the reported generation 0 and the back-off state stay as they were, whereas a delta that names the failed rule clears it only when it arms the rule cleanly and nothing else is still unresolved or inherited (limit 1 above; AC-11 and FU-12 in the Spark flip gate); on the Spark path that clearing happens at the heartbeat maintenance pass, once every accepted arm has resolved, so it can follow the push by a tick or two. One residual stays open: if the agent could not open its boot application (an allocation failure at start-up) and a boot re-arm also failed, the failure has no identity to carry, so the first later clean push, even an unrelated delta, clears the flag while that rule stays unarmed. This is a known, unfixed residual that predates #5547 (#6387; the Spark flip gate, AC-13), and the synchronous variant is live at the default prefer_spark=false. No in-product signal detects it once the flag has cleared: the only trace is the agent log from that boot, and a restart re-runs the boot re-arm pass over the cached rules, which arms the rule only if the cause has cleared. The agent’s persisted generation is never rewritten, so after a restart the agent reports that persisted generation and re-derives the flag from that boot’s own re-arm pass; the flag itself is not carried across the restart. While the rule keeps failing, the agent keeps reporting 0 between back-offs and the server re-pushes, the same held-generation behaviour as any other failed apply. Each time a full catch-up push finishes with the flag still set (a failed catch-up; a delta push never counts, since the server’s catch-up is always a full push), the agent backs off: it reports its real persisted generation on the next N heartbeats, with N = 1, 2, 4, 8 and then 10 for the fifth and later failed catch-ups, and then reports 0 again so the server re-pushes. The first catch-up after boot is not delayed (there is no failed catch-up yet, so the agent reports 0 at once), and a clean application clears the flag and resets the back-off. The back-off applies when the agent’s persisted generation equals the server’s current generation and the failed catch-up reaches the end of apply_rules. The server is unchanged: on the held-back heartbeats it sees a generation equal to its own and does not push. At the default 30 s heartbeat the k-th failed catch-up is followed by N+1 heartbeats before the next push, so the pushes (and their guaranteed_state.reconcile audit rows) come about 60 s, 90 s, 150 s and 270 s apart and then settle at about 330 s (5.5 minutes); at another heartbeat interval multiply by the interval (this holds for heartbeat intervals of about 12.5 s or more, because the server’s own 25 s per-agent limit dominates shorter ones: a claim younger than 25 s is kept even while the agent reports its real generation, so a report of 0 inside it is rate limited; a replica that sheds the reconcile can delay any of these pushes by further heartbeats). The companion tag stays 1 for the whole back-off, so the state stays observable. The back-off does not cover three cases, all the existing held-generation retry and unchanged by this fix: an agent whose persisted generation is behind the server’s (or is 0) is still re-pushed at the server’s 25 s per-agent limit (every heartbeat at the default 30 s, unless shed), because the held-back report is still below the server’s generation; a push that returns before the end of apply_rules (a persistent KV write failure while persisting a rule, a begin-application throw, the invalid-rule-id reject, a duplicate rule id) earns no holdoff; and repeats suppressed under prefer_spark=true (an accepted Spark arm still pending, or a wedge outstanding) do not extend it either. A generic server-side per-agent back-off for the held-generation retry is a follow-up. Two residuals remain: a legacy guard that returns false at arm (ReconcileOutcome::Inert, for example no reachable system bus) is not covered; it is the boot-path analogue of #2797, whose defect on apply_rules is that a returned arm failure is discarded, so such a rule stays persisted but unarmed. Because such a push still counts as applied without an arm failure, the flag clearing is therefore not evidence that every rule is armed. Also, a server whose current generation is 0 does not push, so on such a server the flag holds until a push arrives or the agent restarts with the cause gone. The Spark-path parts of this (a boot arm that fails late or is drained as a failure) are dormant while prefer_spark is off; the legacy path, where the arm throws at boot, is the live one only where a legacy guard actually arms: Windows (File, Registry and Service rules), and Linux built with libsystemd and with a reachable system bus (Service rules only). On macOS the legacy guards are stubs that never arm, so this catch-up does not apply there. On the Spark path a latched boot application can also read yuzu.guardian_arm_failed = 0 beside generation 0 and the companion tag, because that gauge sees neither the latch nor a synchronous failure record; treat the companion tag, not arm_failed, as the boot signal. The companion tag is a per-heartbeat wire diagnostic: no dashboard or server reader consumes it today (#5558). If an agent reports generation 0 for a long time: a persistent cause is the likely reason, because the agent backs off between failed catch-ups (see above), so, when its persisted generation equals the server’s, the server re-pushes about every 5.5 minutes at the default heartbeat once the back-off has reached its cap (one guaranteed_state.reconcile audit row per push; an agent in the exceptions above is re-pushed more often) and a clean push clears the flag. During the back-off the agent reports its real generation, so an agent that reports a yuzu.guardian_generation equal to the server’s generation with the companion tag still 1 (no operator page lists either per agent today) means “unresolved, backing off”, not converged; the tag, not the generation, is the signal. The agent log line Guardian: boot re-arm still unresolved after catch-up push #N (one per failed catch-up, giving the number of held-back heartbeats) is another sign that the back-off is active. Find the agent log lines N cached rule(s) failed to re-arm at boot and Guardian: rule '<id>' failed to re-arm, and fix the host condition they name (for example the agent being unable to create a thread). Removing the failing rule server-side, or disabling it, clears the flag on the next clean full push, but it removes enforcement of that rule for every agent in its scope, so it is a last resort, not a fix. Restarting the agent only re-derives the flag from that boot’s own pass, so it does not help while the cause persists. The server-side tell is repeated guaranteed_state.reconcile rows reading generation 0 -> N for one agent id. On the Spark path (dormant today) the line to look for is Guardian: spark arm failed for rule '<id>' (a synchronous refusal is logged by guardian_engine.cpp, a failure that resolves later by the acknowledgment ledger), and a boot arm that fails late produces no N cached rule(s) failed to re-arm at boot WARN. (4) A wedged rule whose late result lands before the agent has finished recording the wedge costs one avoidable full re-apply on the next identical push, never an acknowledgment: the agent records wedges at the heartbeat maintenance pass, which handles a bounded number of rules per pass, and a rule whose deadline passed just as its arm was being handed to its worker is recorded as a failure but not tracked as a hung call. If a delta push that omits the rule arrives in that window, the recorded failure is carried as an ordinary failure (limit 1 above), so the generation stays held and arm_failed keeps counting until a full push or a push that names the rule ends it, even though the rule may have armed in the meantime.
• Watching a hold. The compensation deadline that the deadline counter uses is 5 s (the Spark runtime’s backend-operation deadline: fixed in production, only tests change it), measured from the instant the compensation became owed; the pending-Disarm threshold, by contrast, is a fixed 30 s from the claim’s creation. The count is observed at the next heartbeat maintenance pass (about 30 s, and only while prefer_spark is on), so it can lag the 5 s deadline by up to a heartbeat, and the age gauge can read well above 5 s before the count first fires. Watch yuzu.guardian_compensation_pending_age_seconds (fleet gauge yuzu_fleet_guardian_compensation_pending_age_seconds_max) and yuzu.guardian_compensation_deadline_elapsed; see Metrics. These two are scoped to the claim and do not dip when the safety valve forces a full re-apply (only arm_pending and arm_failed are scoped to the application). The fleet gauges have no per-agent axis; to find the affected endpoint, look for repeated guaranteed_state.reconcile audit rows for the same agent_id. Any agent being re-pushed also produces repeated rows (for example one whose non-wedge arm failure holds its generation), and the store-degraded abort path writes rows with the same action (result degraded, detail starting heartbeat reconcile ABORTED), so this finds re-pushed agents, not specifically a wedge hold; confirm with that agent’s yuzu.guardian_compensation_pending_age_seconds or yuzu.guardian_compensation_deadline_elapsed tag. (The server counters yuzu_server_guardian_reconciles_total{result="sent"} and yuzu_server_guardian_pushes_dispatched_total{reason="reconcile"} rise fleet-wide while agents are held.) Guardian keeps no counter and writes no log line of its own for an individual suppressed re-push, and nothing marks the valve’s forced re-apply. Every push is still an ordinary command on the agent: it logs the info line Received command: plugin=__guard__, action=push_rules, id=... and increments yuzu_agent_commands_executed_total{plugin="__guard__"}. What tells the two apart is the line that follows. A full application, the valve’s forced re-apply included, logs Guardian: apply_rules ok (...); a suppressed push does not, so its signature in the agent log is a Received command ... push_rules line with no apply_rules ok line after it (a push that aborts early also lacks that line, so use the signature only on an agent already confirmed held). Full mechanism: docs/spark-stage2-guardian-consumer-design.md §R5.3.

Congestion parking (#5168 - dormant until Guardian enforces through SparkEngine, same as the rest of this section). Once enforcement is live, a policy that arms more rules of one type than the agent can start at once (the per-type start limits are Service 3, File 4, Registry 3) no longer has the surplus refused outright. The surplus rules wait for a free slot and are started as earlier ones finish, so yuzu.guardian_arm_pending can stay above zero for up to the claim deadline plus one heartbeat interval (5 s and 30 s by default) while they wait. A rule that still has not started when that deadline is observed (at the heartbeat maintenance pass) ends as a congestion expiry: yuzu.guardian_arm_failed counts it, the agent log carries status=CongestionExpired, and, like any congestion-only expiry (see the wedged-rule hold note above), it holds the policy generation’s acknowledgment for as long as the congestion lasts; the server’s automatic full-sync retry keeps re-applying the policy until the rule arms. A rule that was dispatched just as its deadline passed may instead be logged status=Wedged; it clears if its late result succeeds, and is otherwise covered by the wedged-rule hold note above. A terminal refusal (for example a worker launch failure) is not a congestion expiry and fails at once: a rule refused at first dispatch logs spark arm failed for rule ...: arm worker launch failed, and one refused after it had parked logs status=Failed. A very large policy of slow-starting rules can therefore still leave some rules unenforced until that re-apply; the agent’s real start latency for service rules has not been measured.

Diagnosing an inert File worker or Registry sweeper (Windows)

Runtime inert is a state the agent enters and leaves by itself, and the failed-pass counters are test-only (nothing on the box or the server exposes them), so the agent log is the on-box evidence and yuzu.spark_mechs (with its fleet gauge) is the only signal the server sees. The agent log is not flushed per line, so read the “The log is not flushed per line” bullet before trusting a quiet log.

• Where the log is. The shipped Windows installer registers the service with --log-file {app}\logs\yuzu-agent.log ({app} is the install directory); sc qc YuzuAgent shows the command line, and so the log path, actually in effect. A service registered by hand without --log-file writes <data_dir>\yuzu-agent.log, and an interactive (non-service) agent logs to the console (stderr), and also to that file when --log-file is set. Rotation and the other log flags are in the agent flag table in Device Management (--log-file, --log-max-size, --log-max-files).
• The log is not flushed per line. The agent sets no flush level or flush interval on its logger, so lines pass through the sink’s write buffer and can lag behind the event; they are guaranteed to be written at rotation and when the service stops (service_win.cpp flushes once run() returns), and a crash or a hard exit can lose the buffered tail. An empty search on a quiet agent therefore proves nothing: the absence of worker pass failed lines does not show a boot-time inert. The fleet series is the reliable signal: a service stop flushes the log, but it also ends the episode you are diagnosing.
• Boot-time versus runtime. A mechanism that could not bind its OS facility logs an error from start() (for example spark_file: CreateIoCompletionPort failed, spark_registry: CreateThreadpool failed) and has no worker pass failed / sweeper pass failed lines. A runtime-inert File worker or Registry sweeper logs spark_file: worker pass failed (consecutive # or spark_registry: sweeper pass failed (consecutive # lines, then worker failing persistently / sweeper failing persistently, and worker pass recovered / sweeper pass recovered when it clears. The failed-pass lines are logged at consecutive failures 1, 2, 4, 8 and so on, so the third failure logs only the failing persistently line. The failed-pass line carries no exception text (#4686), so a caught allocation failure looks like any other throw. The failed-pass and failing persistently lines are logged at error level and the recovered line at info level, so --log-level warn hides the recovery.
• Timing. Retries start at 50 ms and double up to a 30 s cap. The mechanism is reported inert at the third consecutive failed pass (about 150 ms of backoff after the first failure at the default cadence) and clears at the first successful pass. For File, a watched-directory event also runs a pass at once, which ends the episode if it succeeds, and a watch added during an episode has its directory read armed inline when the directory resolves within the 50 ms caller budget (a slower or missing directory also waits for a pass); its establishment report and faults are not delivered before the next pass, up to 30 s away unless such an event runs one sooner.
• Alternating failures. A retry pass that finds nothing due counts as a success, so a worker that alternates failing and succeeding logs worker pass failed (consecutive #1 and worker pass recovered over and over and never logs failing persistently; file then stays in yuzu.spark_mechs. Repeated #1 / recovered pairs mean an unhealthy worker even though the fleet series looks healthy.
• A blocked log sink stalls the worker. The failure and recovery lines are written on the File worker’s (or Registry sweeper’s) own thread, so a blocked or slow log sink (a stalled pipe, a hung log volume) stalls that worker for as long as the sink blocks. On both mechanisms those lines are written with the mechanism’s internal lock released (Registry since #4704; before that fix its lines ran while the lock was held, so a stalled sink also blocked adding or removing a rule on that mechanism), so a stalled sink stalls only that worker’s own work: File’s directory-event draining, Registry’s next sweeper pass and any re-arm waiting on it. Adding or removing a rule on that mechanism still works for a stall on one of the three pass-outcome lines that #4704 moved off the lock; a stall on a few other per-key Registry warning lines that still log while holding the lock still blocks it (tracked as #4999 - File has its own analogous per-key lines still under its own lock, tracked separately as #5002). The fleet gauge could already report a mechanism inert before the explaining log line landed locally before this fix (that atomic was always read lock-free); what’s new is the mirror case on recovery, where the gauge can now flip healthy before the “recovered” line lands, since that line is off-lock and can itself lag behind the same slow sink. And because adding or removing a rule no longer hangs alongside a stall on one of those three lines, there is no on-box corroboration for that case in the meantime - subject to the lag described next, the fleet gauge is the first place the trouble shows. Only agent shutdown is bounded: a blocked sink there ends in a forced exit after about 20 s. This is a known limit, recorded in R5.7 (g)(4) of the Guardian and Spark design.
• Fleet lag. The fleet gauge trails the agent by up to about 45 s (the 30 s default heartbeat plus the 15 s server sweep), so an episode shorter than that can pass the gauge unseen and shows only in the log. The fleet-side query is in Metrics.

Related documentation

• Design v1.1 — the authoritative architecture document (engineering reference).
• Windows implementation plan — PR-by-PR delivery plan.
• Spark/legacy delta registry — every deliberate behavioral difference between the legacy IGuard detection path and the SparkEngine path (engineering reference).
• Policy Engine — server-side desired-state rules and compliance checks (complementary to Guardian, not replaced by it).
• Scope Engine — expression language used by scope_expr.
• RBAC — role seeds and the Push operation.
• Audit Log — complete list of Guardian audit actions.