Observability Conventions

The sre and architect agents load this document on any change that adds, removes, or modifies metrics, audit events, or lifecycle events. CLAUDE.md keeps a one-line pointer.

Prometheus metrics

• All metrics use the yuzu_ prefix.
  • Server metrics: yuzu_server_*
  • Agent metrics: yuzu_agent_*
• Consistent label set: agent_id, plugin, method (bounded on yuzu_http_requests_total: the accepted HTTP methods plus UNKNOWN), status, os, arch, principal_class. Avoid one-off labels — they prevent cross-cutting Grafana queries. principal_class (ADR-1005) is a closed set — human | agent | none | engine — on yuzu_http_requests_total; never free-form, never an authorization signal. Hybrid basis (execution-plan PR 4.5): human/agent/none classify by CREDENTIAL PRESENTATION (session cookie / bearer-or-API token / neither) — a traffic-shape signal, not a validated session. engine is the one exception — it requires the RESOLVED session (an engine principal authenticates with a Bearer token indistinguishable from a human API token at the header level, so presentation alone can’t tell them apart), piggybacked off the session already resolved at the pre-routing chokepoint for the PR 4.4 quota gate (zero extra per-request auth cost). A request bearing an engine token whose session resolution FAILS stays presentation-shaped agent, never engine — see server/core/src/principal_class.hpp for the classifier pair (principal_class_of / principal_class_resolved).
• HELP text is 7-bit ASCII, on the server, the agent and the gateway. On the C++ side a non-ASCII character only renders mangled, and scripts/ci/check-metrics-help-ascii.sh (#2128) rejects it. On the gateway it is fatal: an Erlang string literal holding a character above 255 makes prometheus_text_format raise badarg, so every scrape of :9568/metrics returns a bare HTTP 500 and every gateway metric and alert goes dark (#5177). The same lint also scans the gateway’s {help, ...} literals (#4845), so the no-build PR fast path catches a regression without Erlang. yuzu_gw_telemetry_tests:metrics_scrape_renders_test_, which renders the whole registry as a scrape does, stays the backstop for what the lint cannot see: an escape inside a string (\x{...}, octal), a non-ASCII value outside a string literal (a $ char literal, an integer code point, a byte binary), and a HELP that is not a literal at the {help, ...} site (a ?MACRO, a variable, a helper’s argument, the map form, a {'help', ...} key spelling).
• Histogram buckets (default for latency-style histograms): 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0.
• Bounded-label counters are pre-seeded to 0 at startup — every known label combination of a closed-set label is initialised when the metric is described, so the family + HELP/TYPE are present on a healthy server and absent() alerts stay meaningful (examples: the NVD reason series, yuzu_onbehalf_rejected_total{surface=http|grpc}, the gateway’s yuzu_gw_mgmt_auth_rejected_total{reason}). It also matters for increase(): a series that first appears already at 1 contributes no increase, so the first event per label value would never alert (#5177). A pre-seeded series still needs one scrape before the event to count it. On the gateway this is enforced: yuzu_gw_metrics_scrape_tests scrapes a started gateway and fails on a declared family with no series unless yuzu_gw_telemetry:unseeded_metrics/0 gives the reason (a histogram, or label values not known at start), and on a declared metric missing from docs/user-manual/metrics.md.
• An alert names the action, and pages only when someone has to act (owner policy, 2026-10-17). Every shipped rule’s description says what the responder does. An expected transient is suppressed by a metric of the event that explains it, not by a blanket for: that would also delay the real case: the gateway’s pin_mismatch rejections during a routine server-certificate rotation are suppressed for four minutes after yuzu_gw_mgmt_pin_rotated_timestamp_seconds moves (YuzuGatewayMgmtPinMismatch), and the counter is read offset 1m because the rejections precede the timestamp by up to one refresh interval and a scrape can fall between them. That suppression is the gateway rule’s only: the server’s YuzuGatewayForwardUnauthenticated still counts the commands that failed in the rotation window and can fire for up to an hour after it; #5988 (a server retry after UNAUTHENTICATED) removes that cause rather than suppressing it. Seed such an “explaining event” gauge at 0 and write the rule so 0 (never happened) does not suppress. Escalating conditions use disjoint bands, one rule at a time, with rising severity: certificate expiry is 30 to 7 days warning (schedule), 7 days to 24 hours critical (renew today), under 24 hours critical (renew now), for the server (YuzuCertificateExpir*) and the gateway (YuzuGatewayCertificateExpir*) alike.
• Fleet rollups over dynamically-reporting agents are absent-not-zero, NOT pre-seeded — the complementary rule. A gauge family aggregated per-cycle from the reporting agent population (yuzu_fleet_net_*, yuzu_fleet_spark_*, yuzu_fleet_perf_*, yuzu_fleet_guardian_journal_*) is clear_gauge_family()’d and re-emitted each sweep, so a (metric, os[, …]) nobody reported this cycle goes absent, never a fabricated 0. Pre-seeding {os=windows} before any Windows agent enrolls would fabricate a false “0 problems” reading — the exact trap this avoids. > 0 alerts against a genuinely-absent series correctly evaluate to no-match (not a false negative), so the alert still fires the instant real data appears. The pre-seed-to-0 rule above is for process-owned (server or gateway) closed-set series known at startup; this absent-not-zero rule is for agent-population rollups whose label values are only known once agents report. Note the > 0-still-fires remark above is about ABSENCE semantics only - it is not an endorsement of > 0 as an alert form for any given family. The yuzu_fleet_guardian_journal_* counter table in particular is monitor-only: an unlabelled fleet SUM of per-agent cumulative counters, where neither > 0 nor increase() is sound (see its type-honesty note). That claim is scoped to the counter table, not the whole name prefix - the family also carries three *_seconds_max age gauges with different semantics (next bullet).
• Rollup op is chosen per table and lives in the CONSUMER, never in the row. The {tag, gauge, help} row shape is deliberately op-agnostic; SUM is implicit in the counter table’s += and MAX in the age table’s std::max (agent_registry.cpp), and each (agent struct, table) pair carries its own sizeof pin. A MAX-rolled family answers “how bad is the WORST endpoint” (a fleet SUM of ages is meaningless), carries a mechanical _max suffix on every gauge name so the semantics are visible on a graph, and - unlike the counter families - supports sound absolute-threshold alerting (> X for: Y; see the commented staleness examples in yuzu-alerts.yml). Caveat inherent to MAX: one agent’s plausible-but-forged value below the plausibility ceiling OWNS the fleet value (a sum dilutes, a max is captured), so attribution needs a per-agent axis (#2083-class).
• Emission-posture decision rule for agent-reported fleet signals: sparse-omit-zero is correct ONLY when a 0 is uninformative (a counter that never fired). When 0 is a REAL reading (an age: “fresh just now”), the writer must emit it and dormancy must be expressed as structural absence (the emitter receives nullopt and ships nothing), never an omitted zero - otherwise a dead-idle producer is indistinguishable from a healthy idle one, and a fabricated dormant 0 would claim a live fresh producer. Reference implementation: emit_guardian_journal_age_tags + GuardianEngine::journal_age_stats() (the staleness pair emits including 0 while the drain worker is live; the whole family is absent while dormant).
• A fleet rollup should be table-driven, and its meta-signals are the exception to absent-not-zero. yuzu_fleet_guardian_journal_* is the reference shape: one {tag, gauge, help} row per signal drives the clear_gauge_family(), the accumulate, the publish AND the describe() registration, so a new signal cannot ship with a gauge but no HELP, or be cleared but never published. Follow it for new rollups rather than the hand-listed net/spark form. Its two meta-signals (..._reporting, ..._tag_rejected) sit deliberately OUTSIDE the table and publish on every completed sweep including at 0: they are server-owned counts with a true value every cycle, so a 0 is a measurement rather than a fabricated zero. Note the corollary - because they are never cleared they retain their last value, so neither their absence nor a 0 detects a stalled sweep; use the sweep-duration histogram for that.
• Health endpoints (/livez, /readyz, /healthz) reflect every component’s health and are scrapable.
• Plugin heartbeat-tag bridge. A plugin publishes bounded facts into its own KV namespace under heartbeat.<key> (existing yuzu_ctx_storage_set; no ABI change). The agent reads them on EVERY heartbeat (agent.cpp, inside its own try/catch, because the heartbeat loop has no enclosing one) and emits yuzu.plugin.<plugin>.<key> status tags. Caps: at most 8 keys per plugin, key [a-z0-9_]{1,32}, value at most 64 bytes; violations are skipped. Values are agent-controlled, so the server must parse them defensively (bounded length, digits only, fleet-wide label caps) and keep rollups absent-not-zero. The read is a small SQLite KV access on the heartbeat thread (busy timeout applies), so publish only when a value changes. Reference: Echo’s heartbeat.db_corruption_total / heartbeat.db_quarantine_last -> yuzu_fleet_echo_db_corruption_agents and the echo.db.corruption_quarantined audit row. The pure bridge is agents/core/src/plugin_heartbeat_tags.hpp.
• Security events route to the SIEM via Prometheus, not directly. We do not write security events to a SIEM sink from the app. Emit them through the Prometheus counter with an event="security" label; Splunk (and every other SIEM) has a Prometheus receiver and filters on event. This keeps a single egress path and lets the SIEM, not the app, decide what to ingest. Examples: yuzu_grpc_subscribe_peer_mismatch_total{event="security",gateway_mode="false"} (IP-binding stolen-session signal, #1059) and yuzu_grpc_subscribe_identity_mismatch_total{event="security"} (mTLS-identity stolen-session signal, #1118 — no gateway_mode label, since the mTLS check is gateway-mode-independent). The gateway has its own: yuzu_gw_heartbeat_session_mismatch_total{event="security"} (a heartbeat for a held session arrived on a different connection than the one that opened it; the sender is an unresolved principal, so it is metric + rate-limited summary log only, no audit row). Pair the metric (real-time SIEM/alerting) with the audit row (forensic detail): the metric is the signal, the audit row is the evidence. Carve-out: an event with no resolvable principal (e.g. a pre-auth ingress rejection like yuzu_onbehalf_rejected_total) has nothing to attribute an audit row to — it emits metric + (possibly sampled) log only, and the surface’s doc must say so explicitly.

yuzu_grpc_subscribe_peer_advisory_total{event="security",reason="mtls_identity_match|trusted_nat_cidr"} (#1128) is the tolerated counterpart of _peer_mismatch_total: a Subscribe peer-IP mismatch that was downgraded to advisory (not rejected) under a NAT-aware accommodation. It pairs with a session.peer_mismatch audit row carrying result="ok" and outcome=advisory (vs the reject row’s result="denied"). Read the two metrics together — a spike in _advisory_total alone is benign multi-egress churn; a spike in both _advisory_total and _mismatch_total is worth investigating. The reason label distinguishes which accommodation fired.

yuzu_auth_lockout_applied_total / yuzu_auth_lockout_blocked_total (SOC 2 CC6.3, account lockout) follow the same metric-is-the-signal / audit-row-is-the-evidence split, but with a deliberate asymmetry SIEM consumers must know. _applied_total increments once when an account crosses the failed-login threshold and is locked — paired one-to-one with a single auth.lockout.applied audit row. _blocked_total increments on every subsequent login attempt against an already-locked account; those blocked attempts are metric-only (no per-attempt audit row) to prevent audit-log flooding during a sustained brute-force. Read them together: a sustained spike in _blocked_total after a single _applied_total increment is the expected brute-force-against-a-locked-account shape; a spike in _blocked_total with no preceding _applied_total is anomalous (counter corruption or a pre-check bypass) and worth investigating. The matching admin-unlock and successful-login-reset both emit auth.lockout.cleared.

yuzu_auth_local_disabled_total{target=break_glass|other} (SOC 2 CC6.3, hardened mode) is the only signal for a local-password login rejected under --auth-mode=sso-only — that denial is deliberately metric-only, never a per-attempt audit row (same anti-flood rationale as _blocked_total: a credential spray under sso-only would otherwise grow audit.db without bound, and the account never locks on this path to short-circuit it). The CC6.3 activation evidence is the boot-posture log line, not a per-attempt row. The bounded {target} label (cardinality 2) lets a SIEM alert specifically on target=break_glass — repeated probing of the named break-glass account, a higher-signal event than generic other spray.

yuzu_auth_break_glass_login_total (SOC 2 CC6.6) increments after the break-glass password is verified (a wrong password takes the auth.login_failed path and never increments it), paired with a Severity::kCritical auth.breakglass.login audit row. Because break-glass use is rare by design there is no flood risk; alert on any increment. The richer evidence chain for SIEM correlation is the audit row (auth.breakglass.armed → auth.breakglass.login/auth.breakglass.denied); wire an alert on the counter as the cheap always-on signal.

yuzu_auth_mfa_step_up_gate_total{outcome,source,surface} (SOC 2 CC6.6) counts every evaluation of a step-up kRequired route (37 routes) by the central step-up check (step_up_gate.cpp), run from the pre-request stage (run_pre_request_stage in step_up_wiring.hpp, installed by server.cpp); a route classified kExempt, kOwnGate or kPendingRuling is not evaluated there. The one exception is the unwired-store refusal, which the in-handler path (refuse_step_up_store_unwired) also counts as unavailable for the kOwnGate elevation routes, always with surface=rest; every other in-handler refusal is audited but not counted. It is a bounded-label counter pre-seeded to 0 over the full closed set: outcome is pass|refused|unavailable|handler_skipped; source is the session’s auth source (local|oidc|saml|api_token|mcp_token|engine_token, anything else other, no session none); surface is rest (/api/v1/), dashboard (every other path) or mcp (reserved; the MCP arm does not evaluate it yet). refused is a 401 challenge, a SAML 403, an engine principal’s 403 (reason=engine_principal_denied; evaluated after the unwired-store check, so an engine on an unwired store counts as unavailable) or an enrolment-read refusal (a 401 for a local session whose enrolment read failed with a non-store error); unavailable is a 503 (store or decrypt failure, no usable auth store, an unexpected exception while the check itself ran, or a session that could not be fingerprinted) and is the signal YuzuStepUpGateUnavailable keys on. Both are paired with an mfa.step_up.required audit row (written best-effort: an audit write failure never turns a refusal into a pass) whose detail carries cid=<id> and stage=pre_authz when the central check wrote it (the in-handler gate of the elevation routes writes neither), so the counter gives the rate and the audit row the evidence; a refusal with no resolved session (the central check’s anonymous branch, normally answered earlier by pre-routing) has a counter increment (source=none) but no audit row. The gate’s OWN refusal log lines are sampled one in 100 per site (kStepUpLogSample), while the counter and the audit rows are not. That sampling does not cover the audit write itself: AuditStore::log logs each failed audit insert at error level while the store is open (its not-open branch counts the failure and is silent) and the route layer’s audit_log wrapper warns once per failed audit write, both unsampled, so during an audit-store outage a refusal storm still produces about two log records per refusal (measured: 200 refusals gave 400 records, with the store open and failing; a store that is not open produces only the wrapper’s one). The reliable signal under such a storm is the counters, yuzu_auth_mfa_step_up_gate_total and yuzu_server_audit_emit_failed_total (exposed as a counter, but its value is refreshed by a sweep every 15 seconds, so it lags a burst), not the log volume. The gate counter records only refusals the step-up check itself decides: a refusal decided before the gate, such as the 401 unauthorized a cookie session gets once its validation cache lapses during a database outage, or an engine credential that does not resolve, is not counted by it. For an audit-only outage (the authentication store healthy, the audit store failing) no step-up refusal is unavailable, so YuzuStepUpGateUnavailable cannot fire; the alert that covers it is YuzuAuditPersistFailures (increase(yuzu_server_audit_emit_failed_total[5m]) > 0 held for 2 minutes). handler_skipped is a report-only runtime tripwire raised after the handler returns: a kRequired row answered successfully and the handler’s own step_up_fn never consumed the per-request verdict, which can only mean the table disagrees with the handler; it never changes the response. The in-handler second check is not counted again. yuzu_auth_mfa_step_up_window_seconds (gauge, no labels) is the configured --mfa-step-up-window-secs; a value of 0 or less means the stale-proof requirement is disabled on every kRequired route (no stale proof is refused, so none is recorded, and the SAML refusal and the store read are skipped too, so SAML sessions pass; the elevation routes substitute a 300 s window, kElevationStepUpWindow; engine denials and a null store’s 503 still apply and are recorded; approve and reject, POST /api/approvals/{id}/approve and /reject, are the exception: they keep their own approver gate whatever this value is) and is what YuzuStepUpWindowDisabled keys on. YuzuStepUpGateUnavailable keys on sum by (job, instance, surface) (increase(...{outcome="unavailable"}[10m])) >= 3 held for 5 minutes: the >= 3 debounce means one or two stray 503s never page, and a 401 store refusal (counted refused) is not covered by any alert.

yuzu_auth_service_token_role_capped_total (no labels) increments every time synthesize_token_session floors a service-scoped API token’s session role to user, i.e. on every service-scoped session — not just when the creator’s live role would otherwise have differed. It’s deliberately operational, not event="security": this is confinement working as designed on every legitimate service-scoped request, not an anomaly, so there is no expected-vs-unexpected split to alert on (a “did the cap actually change the outcome” signal would need the creator’s live role, i.e. the get_user_role() lookup this branch deliberately skips for the common case). Purpose is debuggability, not detection — an operator investigating why a service-scoped token behaves as user can confirm the cap is active from this counter alone, with no audit row needed (there is nothing to audit — the session was never anything other than user).

yuzu_auth_service_scope_default_denied_total{permission,path_class} (#2298 PR 3 — “the flip”) increments every time a service-scoped API token is refused a (securable_type, operation) pair because it isn’t on the seeded-empty kServiceScopeGlobalSafe allow-list — at three increment sites: AuthRoutes::require_permission (REST, path_class via resolve_body_cap), MCP’s C8 denied-class short-circuit (path_class="mcp", wired separately since C8 returns before require_permission ever runs — see docs/adr/1006-service-scope-default-deny.md Consequences), and (#4981) RestApiV1::deny_fleet_wide_service_scoped (REST, the shared ~9-route blanket-deny chokepoint used where a route has no single (securable, operation) pair to admit-then-filter against — e.g. /api/v1/scope/preview), which uses a fixed literal path_class="rest" rather than resolve_body_cap’s per-route derivation, and falls back to permission="unspecified" for a caller that passes no specific permission (most of this chokepoint’s callers do, since a blanket deny has no single securable to name). This third site’s fixed label collapses several distinct REST routes that share the same default permission argument into one series — read (permission="GuaranteedState:Read", path_class="rest") as an aggregate across those routes, not a single-route signal, until/unless a per-route label is added. It’s deliberately operational, not event="security" — a hit is the flip working as designed on a not-yet-migrated route, not an anomaly — and its purpose is Phase 2 migration prioritization: permission names which (securable, operation) pair to consider re-admitting first, ranked by real call volume rather than guesswork. No alert rule exists for it (tracked as a follow-up). AuthRoutes::require_list_read refuses a service-scoped token without incrementing it (the audit row is written; MCP’s C8 pre-gate still counts), so a route that moves onto that gate also leaves this series, like the require_fleet_read migrations below. The pairing evidence for any individual denial is the auth.permission_required/mcp_audit("denied", ...) audit row (metric-is-the-signal, audit-row-is-the-evidence, same split as elsewhere on this page). Once a (permission, path_class) pair’s LAST route migrates onto require_fleet_read (#3290 Phase 2 — Inventory:Read first), that series stops incrementing entirely for the migrated route(s) — the migrated route no longer reaches require_permission’s service branch at all. This is exact for MCP’s path_class="mcp": query_installed_software was the only Inventory:Read MCP tool, so (Inventory:Read, mcp) genuinely goes flat/absent post-migration. It is NOT yet true for REST’s path_class: GET /api/v1/inventory/tables and POST /api/v1/inventory/query still gate via require_permission on the same Inventory:Read permission and resolve to the same catch-all path_class="default" (body_cap_policy.hpp’s kBodyCapTable has no /api/v1/inventory prefix entry) that /inventory/software used pre-migration — so (Inventory:Read, default) keeps incrementing from those still-unmigrated siblings, and a flat REST series should NOT be read as “the migration didn’t take” or a real one dismissed as noise. Read the series per-route only once every route sharing a (permission, path_class) pair has migrated. No production fleet exists yet, so this metric currently has no real traffic to prioritize by; Phase 2 migration order is documented reasoning instead (docs/security-reviews/service-scope-phase2-migrations-2026-08.md), revisited once real traffic exists.

yuzu_server_quarantine_gate_total{outcome} and yuzu_server_system_reserved_push_total{capability,result} (#881 / #3402) are both bounded-label, pre-seeded-to-0 families driven from their own constexpr arrays (kQuarantineGateOutcomes; kSystemReservedPushes x kSystemReservedPushResults), so an emit site cannot add a label value without it appearing in the boot seed. The gate family is deliberately operational, not event="security": outcome="exempt_control_plugin" and outcome="fresh" are the containment gate working on every dispatch, and the value worth alerting on is fail_closed, which means instruction dispatch is refusing the whole fleet because containment state is unreadable — a dispatch outage, not a quarantine event. The push family meters the four server-internal pushes that deliberately BYPASS that gate; its result="undelivered" series is the one that matters (an undelivered __guard__.push_rules leaves a device unenforced) and is pre-seeded precisely so it can be alerted on before it ever fires. Both follow the standard split: the per-denial evidence is the quarantine.dispatch_denied audit row, the metric is the signal. Note the deliberate exception on the neighbouring yuzu_server_dispatch_target_rejected_total{route,reason} family — its reason="quarantined" value (and, since #3424/#3511, reason="unknown_plugin", and since #5294 reason="kill_switched_os") is EXCLUDED from the YuzuDispatchTargetRejected alert, because correct containment would otherwise page (one looping automation against a contained host clears the >3/15m threshold, and a fail-closed episode increments by the whole fleet at once); a rule that fires on the control working gets silenced, taking the genuine near-miss signal with it. unknown_plugin follows the same reasoning: a caller retrying against an absent plugin is a mistake worth withholding, not a fleet outage worth paging.

yuzu_server_quarantine_reapply_total{result} and yuzu_server_quarantine_endpoint_unconfirmed{reachability} (#3425) extend the same #881 family with the reconnect-reconciliation half: the gate above stops Yuzu reaching a contained device, this pair covers the device’s OWN firewall converging after it reconnects. The counter is another bounded-label, pre-seeded-to-0 family driven from its own constexpr array (kQuarantineReapplyResults), same discipline as kQuarantineGateOutcomes. The gauge is the one exception to this page’s counter-heavy convention: it measures CURRENT divergence (an active record not yet confirmed contained), which must be able to go back down when a reconcile succeeds, so a monotonic counter cannot express it — seeded to 0 for both label values at boot for the identical “absent must not read as never ran” reason. It is also the one metric on this page that is explicitly per-replica, never summed: each server instance’s view of “is this agent’s session live” is local to the gRPC connections it holds, so sum()ing the gauge across instances double-counts every agent connected to a DIFFERENT replica than the one publishing the metric. The reachability label is the SAME exclusion shape as yuzu_server_dispatch_target_rejected_total{reason="quarantined"} two paragraphs up, applied to a gauge instead of a counter: reachability="offline" is excluded from the shipped alert (YuzuQuarantineEndpointUnconfirmed) because a device quarantined while off is legitimately unconfirmed for its whole offline duration, and paging on that is paging on correct behaviour — the alert fires only on reachability="connected", the population where non-confirmation is a genuine reconciliation failure rather than an unreachable device. The evidence trail is the quarantine.reapply audit row (system principal, trigger=tick|heartbeat or a CONFIRMED detail), the same audit-row-is-the-evidence/metric-is-the-signal split as everywhere else on this page. yuzu_server_quarantine_reconciler_tick_healthy (added in the same #3425 line, governance round 11) is a THIRD shape again: an unlabelled boolean freshness gauge, pre-seeded to 1 rather than 0 — the divergence gauge above is only ever .set() on a tick that reaches its normal publish, so on a tick that aborts early it silently FREEZES instead of reading absent or zero; this gauge is what distinguishes “genuinely confirmed” from “haven’t been able to check,” and it is the one gauge on this page where the optimistic pre-seed is the deliberately correct choice, not the 0 convention used everywhere else.

yuzu_server_gateway_peer_denied_total{rpc,reason,event="security"} meters every gateway-upstream call the peer-authorization guard refuses, and is the standard signal-via-Prometheus shape for a bounded-label security counter: the rpc set is the five unary RPCs in snake_case, like every sibling label (proxy_register, batch_heartbeat, proxy_inventory, notify_stream_status, forward_guardian_message) and the reason set is the closed eight-value DenyReason enum (null_context, not_authenticated, no_cert, bad_cert, no_server_auth_eku, not_pinned, outside_validity, internal_error), so a caller cannot mint a series; every rpc x reason pair (5 x 8 = 40) is pre-seeded to 0 at boot while the service is enabled, so increase() sees the first denial. It is UNSAMPLED and is the complete count; the caller receives one fixed UNAUTHENTICATED message whatever the reason, so the label is not an oracle to the caller. The reasons are split by meaning, and the alert follows the split: YuzuGatewayPeerDenied (docs/prometheus/yuzu-alerts.yml, group yuzu-security; rule tests in tests/prometheus/yuzu-alerts.test.yml) lists the five reasons that are not anonymous refusals (bad_cert, no_server_auth_eku, not_pinned, outside_validity, internal_error), and the three reasons with no usable client certificate (null_context, not_authenticated, no_cert) are deliberately NOT alerted, because an anonymous probe of a listener that does not require a client certificate is background noise; a test per anonymous reason pins that. YuzuGatewayPeerDeniedBatchHeartbeat (critical, same group) restricts the same five reasons to rpc="batch_heartbeat" for 5 minutes: a gateway sends that RPC while at least one agent heartbeats through it (an empty-buffer flush is a no-op), so a sustained certificate-reason denial on it means the gateway itself is refused (its registrations and stream-status notices fail; established agent streams to the gateway stay up), and it is active together with the warning rule (an operator-side Alertmanager inhibit rule with source YuzuGatewayPeerDeniedBatchHeartbeat, target YuzuGatewayPeerDenied and equal: [instance, reason] pages only on the critical one). Two gateway-side rules in group yuzu-gateway (scraped from the GATEWAY) cover what the server counter cannot: YuzuGatewayUpstreamAuthRefused (yuzu_gw_upstream_rpc_errors_total{code="16"}, warning) and YuzuGatewayUpstreamTlsHandshakeFailures (yuzu_gw_upstream_tls_handshake_failures_total, warning although the same cause can be a total outage of that gateway; a handshake refused below the guard never reaches the service and is not in the server counter, and the gateway-side alerts have the gateway’s instance, so the inhibit rule above cannot cover them). Those two gateway counters are not pre-seeded, so a label set that first appears at 1 and never moves is invisible to increase() (pinned by a promtool case); a persistent cause keeps incrementing. For the server-side rules the alert stays active for about 6 minutes after the last event, and for the two gateway-side rules (YuzuGatewayUpstreamAuthRefused, YuzuGatewayUpstreamTlsHandshakeFailures, whose window covers the circuit breaker’s maximum open period, 30 s by default since #5695) for about 11 minutes; YuzuGatewayUpstreamCircuitOpened (yuzu_gw_upstream_circuit_transitions_total, pre-seeded for every state) fires only when the breaker opened and has not closed for 3 minutes, or closed three or more times in 15 minutes (a flapping server), and its first step is to ask whether the server is up (if down, the server’s own alerts cover it). YuzuGatewayUpstreamSlow (yuzu_gw_upstream_slow == 1 for 5 minutes, warning, gateway-scraped) covers the case the breaker cannot: a server that answers every call slowly never fails one. The gauge is the gateway’s own trailing-60 s backpressure duty cycle at 0.5 or more (definition and lanes in the gateway manual’s “Slow server”; counters yuzu_gw_upstream_backpressure_seconds_total and yuzu_gw_upstream_lane_saturated_seconds_total{lane}, whose lanes overlap and are never summed); it is pre-seeded to 0, four promtool cases pin the for: and the reset on a dip, the rule is per gateway so it should be grouped by alert name in Alertmanager, and the threshold and window are initial values. The companion gauge yuzu_server_gateway_peer_authz_mode{mode} (exactly one of enforce/insecure_ack/insecure_ack_tls/disabled is 1, fixed at boot) carries the posture: insecure_ack is deliberately not alerted fleet-wide, because a plaintext development rig runs it by design (a production operator who wants it can add yuzu_server_gateway_peer_authz_mode{mode="insecure_ack"} == 1 on their own side; no shipped alert covers it), whereas insecure_ack_tls, acknowledged-disabled on a TLS server with a client CA, is alerted (YuzuGatewayPeerAuthzDisabledWithTls), and either acknowledged mode also writes a boot audit row server.gateway_peer_authz_disabled (system principal; listed in docs/user-manual/audit-log.md among the session.* and gateway rows next to session.gateway_peer_denied, not with that file’s other startup-posture rows (server.viz_disabled, server.unsigned_packs_allowed, server.unsigned_definitions_allowed)). The denial audit budget is itself observable: yuzu_server_gateway_peer_denial_audit_suppressed_total counts the rows it declined to write. The audit half, including what is deliberately NOT audited, is described under “Audit events” below. See metrics.md.

yuzu_server_leader_* (HA WS-11, #4014) is the fenced leader elector’s family: is_leader, epoch, acquired_total, acquire_failures_total{cause}, lost_total{cause}, last_ok_age_seconds and loop_errors_total. All names, HELP text and the closed cause sets live in ONE header (server/core/src/leader_metrics.hpp, with the cause arrays in leader_elector.hpp), and every series is pre-seeded to 0 from those arrays (a counter that moves before the first scrape is first seen at its non-zero value, so increase() does not count that first event: a single replica’s own restart loop is usually not visible to YuzuLeaderFlapping; YuzuServerRestartLoop is the rule for it, with its own documented limits) EXCEPT last_ok_age_seconds, which is absent until the dedicated connection first answers (absent, not a fabricated 0). It is published at SCRAPE time from a lock-free LeaderElector::Stats copy, with counters synced by delta under a mutex (the NVD precedent: two concurrent scrapes must not double-count), and not pushed from the election loop: the loop is the sole writer and is exactly the thing that can wedge (a blocking libpq call against a frozen-but-ACKing backend has no client-side bound; one measured pause lasted about 100 s), so a loop-pushed age would freeze when it matters. Two rules follow. (1) A scrape path must never call a mu_-taking elector accessor (is_open()): the publisher takes a Stats VALUE, never a LeaderElector&, and test_health_routes.cpp pins deps.leader_elector->stats() as the only call. (2) The election loop must never touch the MetricsRegistry (its mutex and std::string key can throw inside a catch block, which would be a std::terminate); it counts into a plain atomic that the scrape pulls. There is deliberately no holder_id label (a per-boot random value would be unbounded series churn; attribute a leader_state.holder_id from the acquired leadership ... (holder_id=...) log line). Every yuzu-ha-leader rule needs a successful scrape; when the whole primary stalls the /metrics handler’s own pool queries hang and every leader series goes stale together, so an up == 0 alert on the server job is the backstop (none ships; #2956 tracked it and was closed without building it). The four yuzu-ha-leader alerts aggregate sum by (job) over max by (job, instance), which REQUIRES scraping each replica individually (never through the load balancer) and separate jobs per deployment; their thresholds are WS-9 calibration claims, and tests/test_leader_alert_metric_names.py keeps the rules, their promtool cases and docs/user-manual/metrics.md on exactly the names the header exports. Leadership is deliberately not a /readyz input.

The NVD CVE sync exposes three server metrics — yuzu_nvd_total_cves (gauge), yuzu_nvd_backfill_complete (gauge), and yuzu_nvd_sync_failures_total{reason} (counter, reason ∈ {connection, http_429, http_403, http_other, parse}). The counter follows the standard bounded-label convention: every reason series is initialised to 0 at startup so the family (and its HELP/TYPE) is present on a healthy server and absent()-style alerts stay meaningful. See metrics.md → NVD CVE sync metrics for the full per-metric reference and reason-label semantics.

yuzu_server_principal_quota_exhausted_total{side,limit} (PR 4.4, ADR-1005 class engine principals) registers the same bounded-label family: all 4 series in the side (engine|operator) x limit (concurrency|rate) closed cross-product are pre-seeded to 0 at startup. It is deliberately operational, not event="security" — hitting a per-principal quota cap is expected steady-state behaviour for a busy engine principal, not an attack signal — and it carries no principal_id label (bounded cardinality only; a per-principal breakdown belongs in a future dashboard join against the engine-principal store, not a Prometheus label). side="operator" is pre-seeded but dormant until Phase 5 delegation debits the operator side too. Its companion yuzu_server_principal_quota_admits_total{side} (governance hardening round) registers the same bounded-label, pre-seeded-to-0 family for the admit path — every request that clears both quota dimensions increments it, making the exhaustion rate (exhausted / (exhausted + admits)) computable rather than just the raw exhaustion count. See metrics.md → Per-principal quota metric for the full reference and the audit-row carve-out rationale.

yuzu_server_csrf_denied_total{path_class,reason} (central same-site gate; counts refused cookie-session writes and refused guarded reads, the cookie-session requests that dispatch work, persist state or force a refresh of server state, with the same closed reasons) registers the same bounded-label, pre-seeded-to-0 shape: path_class is kBodyCapTable’s label set (never the raw path), reason is the closed kSameSiteDenyReasons set (no_origin_no_referer, origin_mismatch, referer_mismatch, gate_error). It is operational, not event="security": a hit is the control working (a script without the header, a proxy without --csrf-trusted-origin) as often as an attempt. Two alert rules ship with it in the yuzu-web-writes group of docs/prometheus/yuzu-alerts.yml: YuzuCookieWritesRefusedSameSite (the name predates the guarded requests and is kept so existing consumers keep working; origin_mismatch plus referer_mismatch, 3 or more in 15 minutes, the Host-rewriting-proxy signature) and YuzuSameSiteGateInternalError (any gate_error in 15 minutes); no_origin_no_referer is deliberately not alerted. The window and threshold are tentative until tuned against real traffic (the yuzu-ota precedent). Choosing operational over event="security" is hard to reverse, because adding an event label later changes the series identity and breaks continuity for every rule and dashboard already built on it. See metrics.md for the reference row.

yuzu_engine_principal_confirm_total{surface,result} (#2404) registers the same bounded-label, pre-seeded-to-0 family: the surface (rest|mcp) x result (success|conflict|client_error|transient) closed cross-product (8 series) is initialised at startup so a 409-conflict or 503-transient retry storm on the credential-confirm endpoint is alertable (it has no per-route label on yuzu_http_requests_total). Unlike the access-review counters below, it IS wired on both surfaces (a distinct surface label prevents double-count) because an agentic MCP client and a REST client can each drive the replay-storm shape #2404 fixes. Scope contract: it counts only confirm calls that reached the credential store or found it unavailable at the store-open guard — pre-store denials (permission, input-validation, the MCP approval-gate consumed-ticket replay) are deliberately excluded, so the label set stays a fact about store outcomes rather than a catch-all endpoint tally; the increment is stamped before the audit emission so an audit-store failure cannot suppress it. Deliberately operational, not event="security" — a replay conflict is an expected agent-retry shape, not an attack signal — and carries no principal_id label; pair it with the engine_principal.credential.confirm audit rows for per-principal forensics (same metric-is-the-signal / audit-row-is-the-evidence split).

yuzu_api_token_confirm_total{surface,result} (P2 #11, SOC 2 CC6.3) registers the human-owned twin of yuzu_engine_principal_confirm_total immediately above: the same surface (rest|mcp) x result (success|conflict|client_error|transient) closed cross-product (8 series), pre-seeded to 0 at startup for the same absent()-alert reason, and exported as a constexpr symbol (rotation_sweep_naming.hpp’s kApiTokenConfirmTotalMetric) rather than a string literal so the registration site and the increment sites can’t silently diverge into a shadow series. It shares the sweep-driver’s kind-discrimination header alongside yuzu_api_token_rotation_auto_revoked_total and yuzu_api_token_rotation_events_total{reason} (the human twins of the two yuzu_engine_principal_rotation_* sweep counters — the T12 maintenance sweep in server.cpp routes each swept row to the engine or human family, never both, keyed on the row’s own principal_kind; see rotation_sweep_naming.hpp’s header comment for the full chokepoint rationale). yuzu_api_token_confirm_total is live — both transports increment it: rest_api_v1.cpp’s POST /api/v1/tokens/{id}/confirm handler (surface="rest") and mcp_server.cpp’s confirm_api_token_rotation tool handler (surface="mcp"), both through the shared kApiTokenConfirmTotalMetric symbol. It follows the identical scope contract and stamp-before-audit discipline as its engine twin, pairing with whatever audit action the confirm-rotation piece emits for per-principal forensics (distinct from the api_token.rotation.auto_revoke / .successor_unused rows this piece DOES ship — see audit-log.md, those are the sweep-driven auto-revoke/successor-unused half, not confirm). See metrics.md → Human API-token confirm metric for the full reference.

yuzu_mcp_poll_total{tool,result} (#3344) registers the same bounded-label, pre-seeded-to-0 family: the tool (get_execution_status|query_responses|get_bundle_result) x result (ready|not_ready) closed cross-product (6 series) is initialised at startup. These three tools are the only MCP surfaces that emit a success-shaped retry_after_ms hint (pending execution/incomplete bundle/rows-still-in-flight is not an error, so the hint lives in the tool’s own structured payload, not the A4 error.data); not_ready means this exact response carried the hint, ready means it was served terminal/complete without one. Scope contract: it counts only calls that reached a served verdict — pre-verdict denials (tier, permission, invalid-params, not-found) are deliberately excluded, already visible via the existing denial counters and A4 envelopes, so this series stays a fact about poll outcomes rather than a catch-all tool tally. Deliberately operational, not event="security" — a high poll rate is expected agentic-worker behaviour, not an anomaly. Its purpose is the same data-driven tuning role as the NVD sync failures above: the named retry_after_ms floor constants (server/core/src/mcp_retry.hpp) were shipped as a mechanical derivation (no dispatch-to-first-result latency histogram exists to measure from) rather than a measured one, and this series’ not_ready fraction is what lets a future PR re-tune them from real data instead of guessing again. See docs/mcp-server.md → retry_after_ms floors for the full constant table.

yuzu_server_response_param_rejected_total{surface} and yuzu_server_response_export_truncated_total{surface,cause} (#4644 / #4703) register the same bounded-label, pre-seeded-to-0 shape for the response routes: surface is rest|rest_v1|mcp on the first and rest|rest_v1 on the second, cause is row_cap|byte_cap (byte_cap is reported when both fired), and seed_response_metrics in response_export_metrics.hpp owns the closed sets so an emit site cannot add a value the seed lacks. Two things differ from the sibling surface counters above and are deliberate to name: rest here is the legacy /api/responses/... routes and rest_v1 the v1 routes (yuzu_api_token_confirm_total uses rest for the v1 surface), and cause is a family-specific label, not one of the cross-cutting set. Both counters are operational, not event="security": a rejected ?limit=100abc is a client bug and a cut export is a bounded-result signal, not an attack. A byte-cap cut also logs one spdlog::warn, rate-limited to one a minute per surface per process, and never logs the instruction or agent id (neither is ever a label). The counter is incremented once the response body has been built into the reply, not after the socket write, which the handler cannot observe. Scope of both counters: the first covers the response routes, MCP query_responses and MCP list_issued_certs only (the strict-parse 400/-32602 rejections on about a dozen other route families and MCP query_audit_log have no counter and show only in generic request metrics); the second covers the two response EXPORT routes only, so the cuts on the execution visualization route, the dashboard results fragment’s unfiltered read and the Echo retention-paused scan page are NOT counted (visualization logs a warning with the cause, the dashboard fragment and the scan page log nothing) and the in-band signals are the only evidence. A sibling ..._read_truncated_total{surface,cause} family would be the shape if a count is wanted; it does not exist today.

yuzu_server_response_facets_dropped_total{reason} (#5623) registers the same bounded-label, pre-seeded-to-0 shape for the response facet ingest path: reason is value_too_long|cap|batch_failed and the value counts distinct (column, value) facets of a stored response that did not persist. seed_response_facet_metrics in response_store.cpp owns the closed set and ServerImpl calls it at startup. The counter is published only after the response transaction commits, so a response that was never stored (including a failed COMMIT) moves no facet series. It is a data-completeness signal, not a security event, and no alert rule ships for it (see the metrics.md row and #5630).

yuzu_server_rotation_initiator_disagreements_total / yuzu_server_rotation_pair_resolve_failures_total (#2961) are ApiTokenStore-owned gauges, gauge-published on every /metrics scrape (no counter pre-seed needed — same shape as yuzu_server_token_cache_size). _initiator_disagreements_total is a tamper/corruption signal on an authorization input, not an operational one: resolve_rotation_initiator fails a confirm closed when its RAM grace-cache copy of requesting_user and the durable api_tokens.rotation_initiator column disagree, which is not reachable through any live code path (both are written from the same value in the same locked mint transaction) — alert on any nonzero value, not a rate threshold. _pair_resolve_failures_total is operability- only (a swallowed post-revoke partner-clear failure; stale metadata, not a lockout risk). See metrics.md → Rotation-durability tamper/corruption gauges for the full reference.

Periodic access reviews (SOC 2 CC6.2, hardening round) added four bounded-label series wired at the REST handlers only (MCP twins not double-counted): yuzu_access_review_export_total{format} (format ∈ json|csv, pre-seeded to 0), yuzu_access_review_export_duration_seconds (histogram, the build_access_review grant-population read latency), yuzu_access_review_campaigns_opened_total (no labels), and yuzu_access_review_attestations_total{decision} (decision ∈ attested|flagged_revoke, pre-seeded to 0). All four are operational, not event="security" — a review export or a recorded decision is expected, self-audited operator activity (own audit rows: access_review.exported, .campaign_opened, .attested/.flagged), not an anomaly signal; pair the counters with the audit rows for evidence, same metric-is-the-signal/audit-row-is-the-evidence split used elsewhere on this page. See metrics.md → Access review metrics for the full reference.

The Guardian heartbeat reconcile admission family (#5513) follows the same rules. yuzu_server_guardian_reconciles_total{result} has the closed label set sent|rate_limited|shed|degraded|no_session, all five pre-seeded to 0, and yuzu_server_guardian_reconcile_invalidated_total (no labels, also pre-seeded) is a separate counter rather than a sixth result because a push withheld after a rule or policy change is not a store failure and must not move the degraded series. yuzu_server_guardian_reconcile_exceptions_total (no labels, pre-seeded to 0) counts an exception thrown by the reconcile hook and caught at its heartbeat-ingestion call site (both catch sites; the quarantine reconcile hook is not counted). It is also kept out of result, because a throw is not a store read that reported itself degraded; YuzuGuardianReconcileDegraded nevertheless alerts on it as a second expression, so a reconcile that keeps throwing pages like one that keeps aborting. Three saturation gauges, yuzu_server_guardian_reconcile_in_flight, yuzu_server_guardian_reconcile_budget_balance_ms and yuzu_server_guardian_reconcile_claims, are per-replica 15 s snapshots published from the health_recompute tick before the session reap and the claim sweep, in their own try block, so a throwing reap or sweep cannot freeze them. yuzu_server_guardian_reconcile_duration_seconds is a histogram of the wall time of an ADMITTED reconcile, observed once on every exit including an exception unwind, with deliberately non-default buckets from 0.1 ms to 10 s (13 bounds) instead of the default set that starts at 5 ms: an admitted push is estimated (not measured) at 3 to 8 ms, and the default set would lump the sub-5 ms part of that into its first bucket. The family is operational, not event="security", and shed writes no audit row by design (the counter plus a log line at most once a minute are its whole signal). yuzu_server_guardian_agent_generation_ahead_total (no labels, pre-seeded to 0) is a throttled flag, sampled at most once a minute per replica, that an agent reports a policy generation above the server’s; a non-zero value does not count agents, and it is not a result of the reconcile family. YuzuGuardianAgentGenerationAhead alerts on it and YuzuGuardianReconcileDegraded alerts on the degraded result and on the exception counter (both in docs/prometheus/yuzu-alerts.yml, both behaviour-tested in tests/prometheus/yuzu-alerts.test.yml); YuzuGuardianReconcileDegraded is the only alert for an unreadable policy generation or Baseline deployed set, since neither moves a read-degrade counter. No alert ships on shed.

yuzu_server_guardian_policy_generation_raise_total{transport,result} counts the outcomes of the supported policy-generation raise (transport ∈ rest|mcp, result ∈ applied|unchanged|refused|failed, all eight series pre-seeded to 0), the same shape as yuzu_server_rbac_enforcement_toggle_total. It is operational, not event="security": the raise is expected, self-audited administrator activity, and the audit rows guaranteed_state.policy_generation.raise (an attempted row, then the outcome) are the evidence while the counter is the signal.

InstructionStore (ADR-0058) registers two bounded-label, pre-seeded-to-0 families at boot: yuzu_server_instruction_read_degrade_total{reason} (reason ∈ store_not_open|pool_acquire_timeout|query_error) and yuzu_server_instruction_write_degrade_total{reason} (reason ∈ the five call sites that can fail — insert_definition_row|update_definition|delete_definition|insert_set_row|delete_set) both increment on a genuine DB/lease failure at their respective read or write call sites, distinct from a legitimate not-found/empty result (the ADR-0036 fail-open class this migration closes). Both are deliberately operational, not event="security" — ordinary Postgres contention, not an attack signal. No dedicated audit row backs either (the class is infrastructure health, not an access decision); pair with the spdlog::error/spdlog::warn log line at the same call site for the specific offending id/reason. This store has no legacy-SQLite backfill (ADR-0009’s fresh-start-by-default class) and so no backfill-outcome metric.

RuntimeConfigStore (ADR-0060) registers one bounded-label, pre-seeded-to-0 family at boot: yuzu_server_runtime_config_read_degrade_total{reason} (reason ∈ store_not_open|pool_acquire_timeout|query_error|crypto_error), incremented independently by get_all(), get(), and read_secret() (each holds its own sampler so a hot call site’s degrade cannot mask a cold one from ever logging) on a genuine DB/lease/decrypt failure, distinct from a legitimate not-found/empty result. Deliberately operational, not event="security" — a crypto_error here means an undecryptable secret (KEK/DB pairing mismatch, corruption), the same fail-closed class docs/user-manual/server-admin.md’s “Key management (secrets KEK)” section already documents for AuthDB/WebhookStore, not an intrusion signal. No dedicated audit row backs it; pair with the spdlog::warn log line at the same call site. This store has no legacy-SQLite backfill (ADR-0009’s fresh-start-by-default class) — the separate warn_if_legacy_data_present() boot check is a one-shot log line, not a metric.

UpdateRegistry (ADR-0061) registers two bounded-label, pre-seeded-to-0 families at boot, gated on --ota-enabled’s default-on state: yuzu_server_update_registry_read_degrade_total{reason} and yuzu_server_update_registry_write_degrade_total{reason} (both reason ∈ store_not_open|pool_acquire_timeout|query_error — a narrower, uniform vocabulary than InstructionStore’s per-call-site write reasons, since every one of UpdateRegistry’s five DB-touching methods shares the same three failure shapes). Plain counter increments, no rate-limited sampler — unlike RuntimeConfigStore’s hot admin-read paths, this store’s own call pattern (admin-driven writes, per-heartbeat-but-not-per-request reads) isn’t the log-flood-prone shape a sampler exists for (see docs/postgres-store-playbook.md’s “Degrade log level” note for the log-level-vs-counter distinction this store’s authoring made explicit). This is the ONLY post-boot degrade signal for this store — is_open() is construction-time-only and never re-checked at runtime, so a healthy boot followed by a sustained Postgres degrade during a live OTA rollout has no other observable trace. Deliberately operational, not event="security". These families are still the only signal for four of the store’s five DB-touching methods; the OTA rollout route is the one exception, and it now also emits ota.package.rollout_changed with result=failure and outcome=store_unavailable existence_unknown=true when a degraded read prevents it from deciding whether the package exists (#3692) — so on that path the audit row and the counter corroborate each other. Everywhere else, pair the counter with the spdlog::error log line at the same call site. This store has no legacy-SQLite backfill (ADR-0009’s fresh-start-by-default class) — the separate legacy_sqlite_probe::warn_if_legacy_rows() boot check is a one-shot log line, not a metric.

AuthManager’s enrollment/pending surface (WS-6 slice 6.2, ADR-2002 §8) registers one bounded-label, pre-seeded-to-0 family at boot: yuzu_auth_enrollment_store_degrade_total{op,reason} (op ∈ create_token|consume|list_tokens|revoke_token|add_pending|ensure_enrolled|pending_status|list_pending|approve|deny|bulk_approve|bulk_deny|remove — the full set of AuthManager enrollment calls; reason ∈ no_store|pool_acquire_timeout|query_error, the AuthDB/SessionStore vocabulary plus no_store for an unattached AuthDB). It increments when an enrollment/pending call fails CLOSED (Register -> gRPC UNAVAILABLE, admin views -> 503/unknown, never an empty or 0 result); caller-input rejections (StoreError::InvalidInput and StoreError::CharsetRefused) are deliberately not counted. Operational, not event="security"; pair with the spdlog::error at the handler. YuzuEnrollmentStoreDegraded (docs/prometheus/yuzu-alerts.yml, mirroring YuzuSessionStoreDegraded’s shape below) fires on increase(...[10m]) > 0 for 10m.

The one-time legacy enrollment .cfg import (WS-6 slice 6.2, run from Server::create right after the auth store is wired) registers two bounded-label, pre-seeded-to-0 families: yuzu_server_enrollment_import_total{kind,outcome} (kind ∈ tokens|pending; outcome ∈ imported|already_imported|fingerprint_mismatch|absent|error) and yuzu_server_enrollment_import_rows_total{kind,result} (result ∈ imported|skipped_existing|skipped_garbled|recovered_colon|id_disambiguated|unplaceable). absent is the steady state and already_imported a benign retry after a failed rename; fingerprint_mismatch (a restored old backup was refused) and error (the server refused to start) are the two that warrant attention. Also writes an enrollment.import audit row (system principal) for imported / fingerprint_mismatch / error, never for the steady-state outcomes. No alert rule ships with it.

The account-lifecycle last-Administrator guard (#4966) registers three bounded-label, pre-seeded-to-0 families at boot: yuzu_server_rbac_lifecycle_store_failure_total{op} (op ∈ deactivate|set_role|would_strand|grant_read; counted in AuthManager, so every transport shares it; a store FAULT, not a refusal) and yuzu_scim_compensation_failed_total{outcome} (outcome ∈ guard_refused_last_administrator|deactivated_but_session_wipe_failed|store_failure; written beside the scim.user.compensation_failed audit row) and yuzu_scim_group_recompute_skipped_total{cause} (cause ∈ promotion|undetermined; a promotion skipped after the first failed guarded write, or a member whose read failed so its role was left untouched; demotions are always attempted so there is no demotion series; written beside the one scim.group.recompute_incomplete audit row). The would_strand op is advisory (the request proceeds to the guarded write) and the unwired fail-closed arm counts for deactivate and set_role only. The critical analytics event that accompanies a SCIM refusal or compensation failure is a no-op under --no-analytics; the audit row and counter still fire. The refusal series yuzu_server_rbac_last_admin_guard_refused_total{transport} does NOT count a refused compensation undo. yuzu_scim_compensation_failed_total and yuzu_scim_group_recompute_skipped_total are operational counters, not SIEM-labelled event="security" series, consistent with the sibling yuzu_scim_* counters (the audit row and the critical analytics event are the security evidence). No alert rules ship with these yet; tracked in #5673.

SessionStore (HA WS-1/1a, ADR-2002 §4) registers four pre-seeded-to-0 families at boot: yuzu_auth_session_store_degrade_total{op} (op ∈ validate|create|touch|generation_refresh|reap|invalidate_user|invalidate|mark_mfa|elevate — the full set of durable-session ops), yuzu_auth_session_reap_total (plain counter), yuzu_auth_session_reap_clock_anomaly_total (plain counter — the DB-PRIMARY clock-integrity monitor, ADR-2002 §4 mitigation (a): incremented when a reap pass is declined because the DB clock — Postgres now(), read in-SQL, the same clock that authors expires_at since #3715 — is implausibly ahead of/behind the persisted anchor, i.e. a backward/forward DB-primary clock step; YuzuSessionReapClockAnomaly alerts on it), and yuzu_auth_local_clock_backward_total (plain counter — the LOCAL host-clock drift monitor, #3715/M6: incremented when a replica’s own wall clock is observed falling behind monotonic time. DISTINCT from the reap series on purpose — with DB-clock authority (#3715) session adjudication no longer rides this host’s wall clock, so a local wobble is a host-health signal, not a session-integrity one, and must never conflate with the DB-primary series; observe-only today, no dedicated alert). The degrade family increments on a genuine PostgreSQL read/write failure on the durable-session auth hot path — a validate degrade for an uncached token fails the request closed (401; the approve and reject routes also read the session’s stored proof, and a degraded read there answers 503 with retry_after_ms and increments yuzu_auth_secret_unavailable_total{route="approval"}); create/elevate/mark_mfa/invalidate/invalidate_user degrades fail the privileged op closed, with one exception: when the invalidate_user delete follows an MFA clear that has already committed (Settings POST /api/settings/mfa/disable), the clear stands and the response is partial (the mfa.disabled audit row carries result=partial, sessions_ended=failed), so the audit row, not the counter, is the signal that the account’s sessions were left live (the same database fault can fail the audit write too, in which case no row exists and the runbook says to treat a missing row as unknown; a single event does not raise YuzuSessionStoreDegraded, which needs the counter to keep rising for 10 minutes; docs/ops-runbooks/auth-db-recovery.md “Finding a partial result” has the audit query); generation_refresh/touch/reap degrades are best-effort (a sustained generation_refresh degrade drops the validate cache past the stale-serve bound so cache hits fall to the authoritative store). Deliberately operational, not event="security" — ordinary Postgres contention/failover on the session store, the same class as yuzu_auth_read_degrade_total and yuzu_server_rbac_read_degrade_total, not an attack signal; pair with the spdlog::warn/debug line at the same call site. yuzu_auth_session_reap_total is the retention-sweep proof (rows deleted by the ~15-minute clock-guarded reap) — the evidence artifact for this store’s storage-limitation control, and belongs alongside the Yuzu*Retention* series in any retention dashboard. The reap pass is single-writer on Postgres today (one server); it joins the shared clock-guarded-retention family and becomes PG-shared-state under the ADR-0012 advisory lock when a second replica lands. This store has no legacy backfill — there was never a durable session table (fresh-start).

The /crypto dashboard (ADR-0070 D13, ADR-0071) adds two sources to the shared yuzu_inventory_read_degrade_total{source,reason} family, both pre-seeded to 0 at boot: source="ca_issued" (the Yuzu-issued counts and certificate list; reason="admission_full" pre-seeded, the store-side reasons on demand) and source="crypto_layout" (layout reads and saves; store_not_open|pool_acquire_timeout|query_error pre-seeded; never admission_full). It also registers one bounded-label family of its own, yuzu_server_crypto_layout_erase_failures_total{reason} (reason ∈ store_not_open|pool_acquire_timeout|outcome_unknown|query_error|link_read_failed, each pre-seeded to 0; the set and the store-failure mapping live once in server/core/src/crypto_layout_erasure.hpp): the best-effort post-deprovision erasure of an operator’s layout row did not complete (link_read_failed: an already-inactive SCIM user’s linked identities could not be read, so only the username’s row was erased). It is the signal of a personal-data row left behind, alerted by YuzuCryptoLayoutEraseFailed (any increase in an hour); the warn line beside it names the reason only, never the principal. Deliberately operational, not event="security": the deprovision itself (the security outcome) has already committed and is audited by its own seam. The CA-store and layout-store degrade warnings are sampled (first of an episode, then every 100th); the layout store logs a SQLSTATE, never the libpq message.

The rotation sweep (#2964) now carries the full clock-guarded-retention shape, the same seven-part contract audit_store’s retention pass follows — reused via the shared audit_retention_rules.hpp::classify, not forked. yuzu_rotation_sweep_declined_total and yuzu_rotation_sweep_lock_skipped_total are the two liveness-adjacent counters this adds to the existing yuzu_rotation_sweep_capped_total family (kind-neutral naming throughout, shared across engine-credential and human API-token rotation pairs — a tick-level clock-guard verdict is not attributable to one kind’s rows). Unlike audit_store, this pass is single-writer-on-Postgres by construction: it takes a store-wide session advisory lock in its own key namespace before classifying, so lock_skipped_total is expected to be nonzero only on a multi-replica deployment, and — the operationally load-bearing fact — must stay at exactly 0 on a single-replica deployment (the default), since one thread taking one lock once per tick can never legitimately lose an election against itself; any movement there on a single replica is a fault, not routine contention. As with audit_store, an accepted, un-capped, un-declined tick increments none of this family’s counters, so a dead sweep thread is invisible to all of them — the same silence-means-healthy gap the audit family closes with its unconditional passes-attempted counter and last-pass gauge. yuzu_rotation_sweep_last_pass_timestamp_seconds (gauge, pre-seeded at 0 at boot) and yuzu_rotation_sweep_bootstrap_declines_total (counter) close that gap the same way; YuzuRotationSweepNotRunning (docs/prometheus/yuzu-alerts.yml) alerts on the gauge’s staleness. The gauge is stamped on Ok/Declined verdicts only — it proves the classification transaction ran, not that any predecessor was actually revoked, so pair it with yuzu_rotation_sweep_lost_revocations_total (an accepted tick that lost some or all of its per-pair revoke transactions to pool contention still stamps this gauge fresh). See metrics.md → Rotation-sweep clock guard metrics for the full reference, plus docs/ops-runbooks/rotation-sweep-clock-guard.md for operator triage.

CommandOutboxStore (HA WS-3 3.3, schema command_outbox_store) registers the yuzu_server_command_outbox_* family: eight bounded counters — yuzu_server_command_outbox_degrade_total{op,reason} (store-level enqueue/claim failures; op also takes occurrences_for_approval and occurrence_exists with reasons not_open/lease_timeout/db_error, the schedule runner’s spent-ticket ledger reads: a rise means approval-gated schedules are being HELD, and yuzu_schedule_fire_failures_total counts those tick attempts, not occurrences), yuzu_server_command_outbox_delivered_total, yuzu_server_command_outbox_deliver_no_agents_total, yuzu_server_command_outbox_deliver_denied_total, yuzu_server_command_outbox_deliver_retry_total, yuzu_server_command_outbox_deliver_errors_total, yuzu_server_command_outbox_deliver_decode_failed_total, yuzu_server_command_outbox_deliver_degrade_total (the delivery loop’s own outcome taxonomy, one series per terminal/retry/degrade path in CommandOutboxDelivery::deliver) — plus one gauge, yuzu_server_command_outbox_pending (the current pending-occurrence backlog), plus an additive cause-labeled companion to the retry counter, yuzu_server_command_outbox_deliver_retry_cause_total{cause} (cause ∈ containment_unreadable|route_unreadable|os_gate_unreadable, the first two added HA WS-4 4.2b Task D and the third by #5294), which breaks a systemic-gate retry down by WHICH gate/directory read degraded without redefining what the unlabeled retry counter means. The gauge is the primary stall signal: a delivery loop stuck on a lost leadership or a persistently degraded gate shows flat counters everywhere, so an unbounded rise in the gauge (or in _deliver_retry_total) is the tell until a dedicated alert ships (tracked for WS-11) — yuzu_server_command_outbox_deliver_retry_cause_total{cause="route_unreadable"} specifically now has one (YuzuGatewayRouteUnreadable, see below). Names follow the standard yuzu_server_<subsystem>_..._total counter / bare-name gauge convention. See metrics.md → Command outbox delivery metrics for the full per-metric reference.

GatewayRouteStore (HA WS-4 4.1, schema gateway_route_store) registers one bounded counter at construction: yuzu_server_gateway_route_write_failed_total{op,reason} (op ∈ register_fresh|announce_connected|deregister|renew_leases|reclaim_tombstoned_session|session_lookup (HA WS-5 batched session_id read, a READ) | cluster_affinity_check (#4669, a READ), reason ∈ store_unavailable|db_error). A degraded directory write/read is per-site, not uniformly fail-open (HA WS-4 4.2b Task B, extended by the 4.4 round-2 review’s UP-4/COMP-6 fix): register_fresh (the row-CREATING write) is fail-CLOSED — the RPC is refused with UNAVAILABLE — and so is the replay-ADOPT decision’s OWN renew_leases/reclaim_tombstoned_session calls in ProxyRegister (a degraded read there can’t distinguish “still valid, adopt” from “superseded, refuse” any better than the code can, and a wrong ADOPT actively clobbers a healthy replica’s placement — a stronger failure mode than a merely-absent CREATE). Every other writer (announce_connected, deregister, and BatchHeartbeat’s OWN separate renew_leases call) stays fail-OPEN (logged, RPC proceeds), since only the CREATE and the ADOPT-vs-REFUSE decision are otherwise unrecoverable in a way worse than a refusal. The session_lookup READ (HA WS-5 slice 2, ADR-2002 §7f) is per-site too, and the name is misleading: despite write_failed it counts a degraded directory READ. BatchHeartbeat is fail-SOFT (locally hinted pairs are still acked and renewed; an id whose read failed and that this replica holds no local hint for is neither acked nor listed in unknown_session_ids); ProxyInventory ingests a degraded read only when it carries a hint and answers received=false for an unknown id with none; NotifyStreamStatus CONNECTED on a degraded read is accepted only when this replica holds the matching local session (directory-verified at ProxyRegister) and is otherwise refused and counted as the desync pair notify_stream_status/connected_unverified below; on a SUCCESSFUL read the local-map fallback applies only to an absent/tombstoned row or a lost-race id, and a row holding a DIFFERENT non-null session id is rejected; DISCONNECTED proceeds on its local check. This counter remains the only signal a systemic Postgres write problem would otherwise leave invisible for the fail-open sites; a fail-closed-site degrade is ALSO directly observable as the RPC’s own UNAVAILABLE error return. Deliberately no companion success/rate counter; the alert rule (YuzuGatewayRouteWriteFailed, docs/prometheus/yuzu-alerts.yml) shipped with HA WS-4 4.2b Task D, once the directory became dispatch-authoritative (Task C, fallback-only — see below). See metrics.md → Gateway routing directory metrics and docs/postgres-migration-ladder.md’s GatewayRouteStore row for the full reference.

GatewayUpstreamServiceImpl (HA WS-4 4.2a, #4246 item #8-counters) registers a second bounded counter, distinct from the write-failure one above: yuzu_server_gateway_route_desync_total{op,outcome} (op ∈ renew_leases|announce_connected|deregister|notify_stream_status|proxy_register|batch_heartbeat|proxy_inventory|session_lookup, outcome ∈ shortfall|session_mismatch|unknown_session|stale_home|malformed_home_id|malformed_cluster_id|cluster_affinity_violation|session_superseded|session_held_by_other_agent|stale_connected_session|connected_unverified|malformed_session_id|ambiguous_session; unknown_session also covers proxy_inventory/renew_leases ids the durable directory does not know). session_lookup/ambiguous_session (HA WS-5 slice 2) is emitted when one presented session_id matches MORE than one non-tombstoned directory row; the id is refused (never acked, never listed, never ingested). The id is not a label and no log line is guaranteed to carry it, so the runbook starts from the directory: SELECT session_id, array_agg(agent_id) FROM gateway_route_store.agent_routes WHERE session_id IS NOT NULL GROUP BY session_id HAVING count(*) > 1;. The pair is pre-seeded at zero and YuzuGatewaySessionAmbiguous pages on any increase. The proxy_register/session_superseded pair (HA WS-4 4.4, #4246 #6) is emitted when the replay-ADOPT decision REFUSES a presented session outright (FAILED_PRECONDITION) because the agent’s OWN directory row is LIVE under a DIFFERENT session — a genuine stale/zombie replay and nothing else, not a store-write guard rejection like the other ops here. The proxy_register/session_held_by_other_agent pair (HA WS-5, pre-seeded at zero) is the same decision refusing because a DIFFERENT agent’s committed directory row already holds the presented id: expected rate zero, any increase actionable (a forged or colliding id from a pinned gateway), paged by YuzuGatewaySessionHeldByOtherAgent. Its NOT EXISTS guard is only a fast path: since migration v5 (#5683) a UNIQUE partial index (ux_agent_routes_session_id) serialises two concurrent adopts of one id, the loser fails 23505 and is counted on this same pair, and ambiguous_session remains defence in depth for a hand-dropped index only. The announce_connected/session_held_by_other_agent pair (#5683, pre-seeded at zero) is the same condition on the CONNECTED path: when the presenting agent has NO directory row, the fallback insert for an id another agent holds is refused by the unique index, the in-memory placement is unpublished, the notification is not acknowledged, and a best-effort gateway.session_held_by_other_agent audit row is written (without the session id); an agent that still has a row, including a tombstone, is not covered (its CONNECTED is a session_mismatch; the tombstoned-victim residual, ADR-2002 7f). YuzuGatewaySessionHeldByOtherAgent pages on both ops. The migration also exports yuzu_server_gateway_route_session_dedup_total (rows tombstoned by migration v5 at THIS start; present only with --gateway-upstream; durable record in route_meta key v5_session_dedup_tombstoned_rows; it keeps its migrating-start value for the whole process lifetime, so it proves nothing about timing: the migration’s session_held_by_other_agent refusals are told from a forged id only by BOTH at most N proxy_register refusals (none from announce_connected) AND all within that start’s window of roughly 2 minutes when every gateway includes #5695, or roughly 6 minutes plus 20 ms per agent if any gateway is older; inside the bound and window they are still not proof of benignness, so re-check after the window and do not silence the alert beyond it, see gateway.md “One-time dedup at upgrade”). The proxy_register/malformed_session_id pair (HA WS-5, pre-seeded) is the adopt path refusing a presented id that is not gw-session- plus 32 lowercase hex characters. The notify_stream_status/stale_connected_session pair (HA WS-4 4.4 post-build review, PR #4636 FortitudeEtc BLOCKER 2) is emitted when a CONNECTED’s session_id passes the earlier gateway_sessions_ liveness check (it IS some live session for this agent) but is no longer the CURRENTLY-installed one — AgentRegistry::set_gateway_route’s own session guard rejected the in-memory publish, and the RPC is acknowledged false rather than falling through to (a now-meaningless) announce_connected write — and, since HA WS-5, when a successful directory read finds the agent’s row naming a different non-null session; distinct from the plain unknown_session outcome below, which fires when the session isn’t in gateway_sessions_ at all. The notify_stream_status/connected_unverified pair (HA WS-5, pre-seeded) is a CONNECTED refused because the directory read degraded and this replica holds no matching local session (it also covers a delayed CONNECTED for a session this replica holds but has since superseded, which a successful read would count as stale_connected_session; that agent’s newer session is unaffected); the gateway neither retries it nor counts it in yuzu_gw_upstream_notify_dropped_total, so this pair is the only signal that, in the first case, the agent stays registered without dispatch placement until it reconnects. The batch_heartbeat/malformed_session_id pair (#1197) is emitted when a BatchHeartbeat entry’s session_id is longer than 64 bytes AND is not resolved (counted per entry, not deduped; since HA WS-5 the directory lookup drops over-64-byte ids before reading, so on a successful read such an entry is always counted here; a minted id is 43 bytes); no write is attempted, the id is never echoed in BatchHeartbeatResponse.unknown_session_ids, and the pair is pre-seeded at zero with an expected rate of zero (a sustained rate means a buggy gateway, or an agent sending a malformed session_id; the gateway forwards agent heartbeats verbatim). This is a guard-rejection, not a store-failure, signal — the write/lookup succeeded but a session-scoped guard correctly refused to touch a row it doesn’t own (renew_leases renewing fewer sessions than presented, announce_connected/deregister matching zero rows for the presented session, or NotifyStreamStatus’s own in-memory gateway_sessions_ lookup missing entirely), which is exactly the “in-memory session map vs. durable directory disagree” symptom an operator needs visibility into. unknown_session is emitted by BOTH notify_stream_status (as before) AND, since HA WS-4 4.2b Task B, renew_leases — BatchHeartbeat’s renew batch excludes a heartbeat whose session gateway_sessions_ doesn’t locally recognize (nothing to correlate an agent_id against for the #4246 #10 renew correlation) rather than silently dropping it. Since HA WS-5 slice 2 (ADR-2002 §7f, #4246 item 3) a session is resolved against the durable GatewayRouteStore directory instead of the replica’s in-memory map, so unknown_session now means the DIRECTORY does not know the id (a store-only resolution, known to the directory but not yet to this replica’s local map, is renewed and listed for replay but does NOT move this counter). The per-replica-memory limitation is closed for session resolution and remains only for the local-only surfaces tracked in #4246 (device list, agent_count(), guardian reconcile); a directory-unknown rise is reached by a PG outage past the 270 s lease+grace horizon or by a rolling deploy with an old replica still serving (see “What happens when the server restarts” in gateway.md). A low background rate is expected (stale/superseded-session notifications the guards exist to no-op on); a sustained rise is the desync signal. Deliberately excludes a register_fresh epoch-race loss (benign, tracked separately via lost_race_sessions_ so it never inflates this counter as a false session_mismatch). Two contributors that are benign on their own: (a) a brief post-restart/failover baseline rise (shortfall until agents re-announce/reap; unknown_session only for ids the durable directory itself has lost, e.g. after a PG outage past 270 s, or from an old replica during a rolling deploy). A brief rise that then clears is not an incident by itself; with both sides upgraded it clears by itself (the gateway replays the sessions the heartbeat verdict lists: within about one agent heartbeat interval plus one flush plus the agent’s drip position when the gateway’s circuit breaker is closed, and after the breaker’s remaining backoff, up to 300 s, plus the same when the outage opened it). A server-only restart SHORTER than 270 s leaves the rows live, so its sessions are store-only resolutions — listed for replay, never counted — and renew_leases/unknown_session stays FLAT whether or not the replay converges. The non-convergence signal is therefore yuzu_agents_connected / agents.online not ramping back (with a gateway that predates the verdict replay, agents.online staying at 0 is the condition described in “What happens when the server restarts” in gateway.md: on the observed rig it did not clear on its own, nor after a further server-only restart; only a full server+gateway+agent restart recovered it), plus, after an outage past 270 s, a SUSTAINED renew_leases/unknown_session rise. To find why, read yuzu_gw_heartbeat_verdict_dropped_total{reason}, yuzu_gw_heartbeat_unknown_truncated_total, yuzu_gw_registration_replay_triggered_total{trigger} and yuzu_gw_registration_replay_total; (b) a redelivered/duplicate DISCONNECTED notification, which counts as deregister/session_mismatch the second time it lands. This counter has no general alert, apart from three expected-zero pairs: session_lookup/ambiguous_session (YuzuGatewaySessionAmbiguous), and proxy_register/session_held_by_other_agent and announce_connected/session_held_by_other_agent (both YuzuGatewaySessionHeldByOtherAgent, expression op=~"proxy_register|announce_connected") — its other guard-rejection outcomes are expected at a low background rate and a threshold that doesn’t page on the two benign contributors above needs real fleet data to set (same reasoning as the #913 OTA-bounds lesson on alert thresholds). HA WS-4 4.2b Task D instead ships YuzuGatewayRouteUnreadable (docs/prometheus/yuzu-alerts.yml) keyed on yuzu_server_command_outbox_deliver_retry_cause_total{cause="route_unreadable"} — a degraded directory read during dispatch, the closest available signal to #4246 item #8’s alert-rule half, distinct from (and narrower than) this desync counter’s guard-rejection outcomes. See metrics.md → Gateway routing directory metrics.

GatewayUpstreamServiceImpl (HA WS-5 slice 2, ADR-2002 §7f) registers a third bounded counter: yuzu_server_gateway_route_degraded_renewal_total{op} (op ∈ batch_heartbeat, a closed set of one, pre-seeded at zero at construction). It is the fail-soft companion to write_failed{op="session_lookup"}: it counts BATCHES, not sessions — one increment per BatchHeartbeat whose directory read degraded and that still renewed at least one locally hinted (agent, session) pair, taken only after renew_leases returned success. A rising rate therefore means Postgres reads are failing while writes succeed and leases are being kept alive from this replica’s local hints; a batch whose renewal write also fails moves write_failed{op="renew_leases"} instead and does not count here. No alert: it rises together with YuzuGatewayRouteWriteFailed, which already pages on the degraded read. See metrics.md → Gateway routing directory metrics.

GatewayUpstreamServiceImpl (#5686) also registers yuzu_server_gateway_batch_heartbeat_duration_seconds{outcome}, a handler-latency histogram with the standard buckets (the 0.5/1.0 boundaries resolve the 500 ms directory-read and lease-renewal acquire bounds) and a closed outcome label (ok|degraded, both pre-seeded; degraded = the directory read, the renewal or a presence upsert degraded in that batch; outcome mirrors the sibling desync family in the same file rather than introducing status), and yuzu_server_agent_presence_upsert_skipped_total, an unlabelled counter pre-seeded at zero that counts the presence upserts a batch skipped after a presence-store failure, except half-open probes (the entry right after the first failure is attempted and the spacing doubles on each further consecutive failure; under total saturation, every attempt failing, a batch of N makes at most floor(log2 N)+1 attempts, each waiting at most its 250 ms pool acquire (an attempt stalled on a row lock is bounded by the pool lock_timeout instead, so the attempt count is bounded but the time per attempt is not always 250 ms); under partial brownout a probe that succeeds closes the trip and resets the spacing, so waits can exceed that count but never the old per-entry N (a strictly alternating failure pattern can still approach about N/2 failed waits, about 41 s at N=330 with 250 ms acquires), and once the store recovers a later probe or the next batch refreshes a skipped row). Failed attempts stay in yuzu_server_agent_presence_store_failed_total{op="upsert"}, so that family keeps counting only real attempts (on the gateway batch path at most about floor(log2 N)+1 per saturated batch, not one per entry). Both are operational. No latency alert yet: the threshold needs fleet data. Route-directory step failures (session_lookup, renew_leases) raise YuzuGatewayRouteWriteFailed; presence-store failures and skips have no dedicated rule yet (same posture as presence_store_failed_total above) and are covered only at pool level by YuzuPgPoolSaturated and YuzuPgAcquireWaitHigh; the decision is recorded in docs/prometheus/yuzu-alerts.yml. See metrics.md.

The gateway-side counters added for the heartbeat verdict (#1197) follow the same rules as every other bounded family: yuzu_gw_registration_replay_triggered_total{trigger} (trigger ∈ breaker|heartbeat) and yuzu_gw_heartbeat_verdict_dropped_total{reason} (reason ∈ malformed|not_local|circuit_open|queue_full) have closed label sets, every series is pre-seeded to 0 at start, no label is derived from an agent or session id, and yuzu_gw_heartbeat_unknown_truncated_total is an unlabelled counter. The verdict replay feeds the gateway’s shared upstream circuit breaker like any other ProxyRegister (a failed replay ProxyRegister counts as a failure), so it can open the breaker, and an open breaker makes the whole /readyz answer 503 (circuit_breaker=false in its checks), which can take the node out of rotation for a load balancer that probes /readyz (a half_open -> open (probe failed ...) WARN means the server is still failing: look at the server, do not restart the gateway); see “What happens when the server restarts” in gateway.md. queue_full means the replay queue is at its cap, or the upstream process already holds more than 100 unhandled messages so the ids were not handed to it; the truncated-verdict WARN is logged at most once per 60 s per heartbeat buffer process (the limit resets when that process restarts, so the first WARN after a restart says suppressed 0) while yuzu_gw_heartbeat_unknown_truncated_total counts every occurrence. The gateway’s heartbeat buffer adds two more bounded families under the same rules (#1197): yuzu_gw_heartbeat_buffer_dropped_total{reason} (reason ∈ buffer_full|snapshot_oversize|snapshot_evicted|heartbeat_oversize|heartbeat_invalid|chunk_rejected, a closed set, every series pre-seeded to 0 at start) and yuzu_gw_heartbeat_coalesced_total (unlabelled); no label is derived from an agent or session id. The buffer keeps one entry per session, flushes in chunks of at most about 3 MiB (estimated) so a request stays under the server’s gRPC receive limit of 4194304 bytes, and bounds the retained buffer by max_heartbeat_buffer (sessions) and max_heartbeat_buffer_bytes (default 64 MiB); buffer_dropped counts older heartbeat state shed under that bound (and, for heartbeat_oversize and heartbeat_invalid, a heartbeat forwarded without its status tags, or with the invalid bytes of a tag replaced by replacement characters, so its liveness is unaffected; heartbeat_oversize also covers a heartbeat with one tag key or value over about 64 KiB or more than about 256 KiB of tag bytes in total, which is dropped of its tags before the UTF-8 repair so that the repair cost stays bounded; and, for chunk_rejected, a single server-rejected heartbeat dropped instead of retried; heartbeat_invalid also counts tags removed because a tag was not a binary and a heartbeat dropped for a bad or empty session id, which the short HELP text does not say; a transient failure keeps heartbeats buffered), a flush sends at most 8 chunks (about 24 MiB) per cycle, and coalesced counts a heartbeat replaced by a newer one of the same session. coalesced is expected to rise while the server is unreachable and has no alert; because merged heartbeats arrive as fewer heartbeats, the server’s yuzu_heartbeats_received_total{via="gateway"} under-counts the heartbeats the agents sent while a backlog is coalesced, so do not alert on a fall in it alone; see “Heartbeat Batching” in gateway.md. The gateway also registers yuzu_gw_session_limit_rejected_total (unlabelled, pre-seeded to 0): registrations refused because one connection already holds the configured number of agent sessions (max_sessions_per_connection, default 8, read once at registry start; it counts session rows, pending plus live, and the slot is reserved atomically when a Register is admitted; a refused call is answered UNAVAILABLE); it has no agent label, and see “Per-connection session cap” in gateway.md.

The gateway’s pending stream-status notification metrics (#4632) follow the same rules. yuzu_gw_upstream_notify_dropped_total{reason} has the closed label set {at_capacity, unknown}, where at_capacity means an admission loss: the pending store was at its cap, that is a new key is admitted only while the held keys plus the key of a half open probe in flight stay below notify_pending_max (it does not count delivery loss after admission, such as a failed worker or a gateway stop) and unknown is the overflow bucket that any other reason value folds into, which the gateway does not emit today. Of the notify and heartbeat families, only this one pre-seeds its unknown series: verdict_dropped and buffer_dropped fold an out-of-set value into an unknown series that is not pre-seeded, so it first appears when it first fires (seeding those is a follow-up). circuit_open is retired for new emissions, because a notification arriving while the breaker is not closed is now held, so an older gateway may still export samples with that value and a dashboard or rule that sums the family should not read it as a current cause. Both series {reason="at_capacity"} and {reason="unknown"}, yuzu_gw_upstream_notify_coalesced_total (unlabelled; it counts a held notification replaced by a newer event for the same agent session and stream home, or a failed probe notification superseded by one, so it measures suppression, not delivery), and the two gauges yuzu_gw_upstream_notify_pending and yuzu_gw_upstream_notify_inflight (label node) are all pre-seeded to 0, so a healthy gateway is distinguishable from an older one that exports nothing. No label is derived from an agent or session id. The gauges count the current upstream client process only: after it restarts they start at 0, the held entries are gone, and workers of the previous process are not counted. See “Stream Status Notification” in gateway.md.

The gateway’s yuzu_gw_upstream_rpc_errors_total{rpc_name,code} pre-seeds only the two locally produced values of code, encode and exception, to 0: encode for register, inventory, notify_stream and guardian, and exception for those four plus batch_heartbeat (a marshal failure on the heartbeat buffer’s flush is counted as code="exception" and is not covered by the encode alert); the server-status and transport values are open-ended and appear at their first occurrence, so they are not seeded. The encode series is seeded at 0 so that YuzuGatewayUpstreamEncodeErrors (increase(...{code="encode"}[10m]) >= 10, so one rejected request stays quiet and a systematic fault pages on a gateway fronting about five or more agents; it is per rpc_name, so a small gateway or a low-rate RPC can stay under it) sees the first increase of a series that would otherwise first appear at 1, which matters because a local encode failure never opens the breaker or fails /readyz, and the older YuzuGatewayUpstreamErrors rule fires only above 0.1 errors per second with no code filter. A circuit_open rejection never reaches the call and is not counted in this family. yuzu_gw_register_refused_total{reason} counts Register calls the gateway edge refused before any upstream call; its closed label set is {agent_id_length, unknown} (unknown is the overflow bucket for any other reason value, not emitted today), both pre-seeded to 0, and no label is derived from agent data.

OfflineEndpointStore (HA WS-5 slice 1, ADR-2002 §7a, governance hardening 2026-09-22; label set widened #4981 PR-1) registers one bounded counter, mirroring GatewayRouteStore::record_route_store_failure’s naming/label convention exactly: yuzu_server_agent_presence_store_failed_total{op,reason} (op ∈ upsert|query_live_ids|query_stale_within|fleet_state_counts|remove_if_session, reason ∈ store_unavailable|db_error|truncated), incremented from record_presence_store_failure (offline_endpoint_store.cpp). truncated (#4981 PR-1 Finding B1, query_live_ids-only) fires when a live-presence read returns more than kQueryRowCap rows — the query is issued at LIMIT kQueryRowCap + 1 specifically so this case is DETECTED rather than silently served as a partial, under-counted set. upsert (the sole writer of cross-replica liveness, called on every heartbeat — external review finding, 2026-09-22, closed a real coverage gap where only the read/delete paths were metered and a saturated/failing writer would age out presence fleet-wide with the counter reading zero throughout) and remove_if_session remain fail-SOFT by design — a degraded remove_if_session leaves a departed agent counted present until the ordinary TTL read-filter expires it, instead of being cleared immediately. query_live_ids is NO LONGER fail-soft for AgentRegistry::evaluate_scope as of #4981 PR-1: query_live_ids returns std::expected<..., PresenceReadError>, and a degraded/truncated read now makes evaluate_scope (the fleet-wide, presence-consulting entry point) ABORT with ScopeEvalError::Kind::PresenceDegraded rather than silently narrowing to local-only — see docs/adr/0036-result-set-store-postgres-migration.md’s dated Update section and agent_registry.hpp’s evaluate_scope doc comment. AgentRegistry::all_ids() (the confined-broadcast/empty-scope-Guardian-push consumer) still degrades to local-only on a failed read — presence only widens visibility for this caller, it grants no send authority (see AgentRegistry::live_presence()’s doc comment). evaluate_scope_local (the Guardian rule-scoping entry point) never reads presence at all, so it never has a failed presence read to degrade FROM in the first place — it cannot return ScopeEvalError::Kind::PresenceDegraded by construction (see its own doc comment). This counter exists so that fail-soft does not mean fail-invisible now that the store is load-bearing for cross-replica scope-evaluation correctness, not merely the /viz/fleet stale-cube render it originally served. A zero-row RETURNING result from remove_if_session (a legitimate session mismatch — the guard the method exists to enforce) is NOT counted here; only a genuine store-unavailable/query-error/truncated path is. Deliberately no dedicated alert rule yet (same posture as yuzu_server_gateway_route_desync_total above) — a background rate worth paging on needs real fleet data once a multi-replica deployment exists to generate any. Rate caveat (#4981 PR-1): live_presence() negative-caches a failed read for the same kPresenceCacheTtl window a successful one gets, so during a SUSTAINED outage this counter increments at most once per TTL window (currently 3s) — not once per evaluate_scope/all_ids()/policy-tick call in that window. This is the correct choice for a boolean “is presence degraded right now” signal (and avoids a thundering herd of redundant queries against an already-failing store), but the counter’s rate deliberately under-represents actual call volume during an outage; do not use it to size blast radius. This caveat is query_live_ids-only: query_stale_within (#5531, the roster read; never truncated) has no cache and counts and warns once per failed read (one per Hardware render / offline-device CI-record open / Software hostname-map render / every fleet-topology request — the 60 s topology cache does not cover the roster read), so its rate approximates operator click volume. Its consumers: Hardware list/REST/CI record show the banner / 503; viz and MCP render live-only with offline_roster_degraded. fleet_state_counts (the SQL fleet tally behind the Dashboard readout) likewise has no cache and counts once per failed read; it is a typed error to its caller, never fail-soft.

HA WS-4 4.2b Task C wired GatewayRouteStore::lookup_routes into confined dispatch as a FALLBACK-ONLY consult (dispatch_route_fallback.hpp): one batched read per arm walk, fired only for candidates the local AgentRegistry has no live session for, with every send still passing the existing per-device confinement intersection first. This changes zero routing outcomes on today’s single-replica monolith (a direct-connected agent never has a directory row) — see docs/adr/2002-high-availability-architecture.md §7’s 4.2b status paragraph.

server.cpp’s background maintenance tick (HA WS-4 4.2a governance fold) registers a third bounded counter: yuzu_server_gateway_route_reap_total{outcome} (outcome ∈ ok|ok_capped|recovered|declined|skipped|error), incremented once per reap_stale_routes() tick. This is the compensating observable signal for the clock-guarded-retention part-1 carve-out and the decline-once/drain-on-repeat part-4 guard gateway_route_store.hpp’s reap_stale_routes header documents — a declined (clock-anomaly) or failed (store/query error) reap pass was previously spdlog::warn-only with no metric, distinct from the write-failed/desync counters above which cover the write path, not a reap pass’s own outcome. recovered (PR #4299 round-2 review) is distinct from a routine ok pass — it means the pass ran via the decline-once/drain-on-repeat recovery branch, i.e. a clock anomaly persisted across a full decline pass at the SAME (anchor, direction) and this pass drained the resulting backlog under the usual caps; a sustained non-zero recovered rate means the reaper is recovering from a clock anomaly or a genuine gap and is worth an operator look, even though nothing failed. skipped (PR #4299 round-2 external review) means another replica already held the gateway_route_store:reap advisory lock this tick (pg_try_advisory_xact_lock, not blocking) — routine on a multi-replica deployment, never a failure. ok_capped (PR #4299 round 4) is an accepted pass that hit kReapCap on a sweep AND a same-txn EXISTS probe confirmed a remaining backlog — observability only (NOT a cadence/re-arm change; see the non-acceleration rationale in docs/clock-guarded-retention.md); a sustained ok_capped rate means the reaper is chronically behind. Note also (round 4) that declined now covers a same-direction anomaly that has NOT yet PERSISTED a real-time-plausible interval ([kMinReapRecoveryGapMs, kMaxReapRecoveryGapMs]), not just an implausible reading. See metrics.md → Gateway routing directory metrics and docs/clock-guarded-retention.md’s GatewayRouteStore entry.

yuzu_exec_tracker_bookkeeping_failed_total{op,surface} (#4982) registers a bounded, pre-seeded-to-0 family for two ExecutionTracker calls that were previously log-only on failure at every call site: op ∈ set_agents_targeted|mark_cancelled, surface ∈ rest|mcp|workflow|schedule|outbox (10 pre-seeded series; rest_api_v1.cpp’s run_async result-set producer, mcp_server.cpp’s result-set and execute_instruction handlers, workflow_routes.cpp’s instruction-execute route, schedule_runner.cpp’s fire path, and command_outbox_delivery.cpp’s delivery loop — the last three surfaces added in a fix round after the same swallowed-failure pattern was found, uninstrumented, at their equivalent call sites. {op=set_agents_targeted,surface=outbox} is permanently unreachable, by construction — that write moved into the atomic CommandDeliveryFinalizationOwner::mark_sent_with_target, counted instead via yuzu_server_command_outbox_deliver_degrade_total. {op=set_agents_targeted,surface=schedule} is likewise permanently unreachable — schedule_runner.cpp never calls set_agents_targeted at all; a schedule-originated dispatch’s target count is set, if at all, via the outbox delivery path above, which counts under surface=outbox — so 8 of the 10 series are the live set). Every one of these calls sits on a dispatch-then-bookkeeping tail — a pool exhaustion or a failed statement there was previously visible only as an spdlog::error line (or, at one command_outbox_delivery.cpp site, neither), with no Prometheus signal distinguishing a one-off blip from a sustained degrade. No retry is attempted inline: retrying inside an already-degraded-store request handler or background worker risks doubling request latency or delaying the next tick for no reliability gain. Which failure is more consequential INVERTED once the stuck-execution reap sweep described below shipped. Before that, mark_cancelled failing was the more consequential of the two — a dispatch refused before it reached any agent that ALSO failed to cancel its execution row left that row stranded at status='running'/agents_targeted=0 forever, since refresh_counts’s terminal transition requires agents_targeted > 0 to ever fire. The stuck-exec reap sweep below now recovers exactly that population. The UNRECOVERABLE case today is the opposite: set_agents_targeted failing on a dispatch that genuinely reached one or more agents (so a real agent_exec_status response exists) leaves that row at agents_targeted=0 forever AND permanently excluded from the sweep’s candidate set (the same BLOCKING fix’s own exclusion clause, by design — never acting on a row with a real agent response is correct, since there is no repair path today that re-derives agents_targeted from agent_exec_status). refresh_counts’s terminal transition still never fires for that row, and nothing else ever will. Deliberately operational, not event="security" — no dedicated audit row backs either failure; pair with the spdlog::error line at the same call site for the specific execution_id.

yuzu_exec_tracker_stuck_reap_total{outcome} (#4982 Part B) is the recovery-side counterpart of the metric above — outcome ∈ cancelled|not_cancelled|would_wipe|clock_anomaly|degraded|capped|skipped, a bounded, pre-seeded-to-0 family incremented once per ExecutionTracker::reap_stuck_running_executions() pass (~15-minute cadence, server.cpp). cancelled/not_cancelled increment by the per-row count for an accepted pass — since a fix round (post-adversarial-review, closing a TOCTOU window and a cross-replica cap-exceed gap) the pass’s candidate select and its cancel are ONE atomic UPDATE ... RETURNING id inside the SAME advisory-lock-held transaction, so not_cancelled now covers a candidate the atomic recheck excluded at mutation time — a genuine dispatch landed, its outbox entry went pending, a real agent response landed, or the row otherwise left running, all within a single-statement-boundary window, never the cross-replica race this field used to also have to cover; would_wipe/clock_anomaly/degraded each increment once per DECLINED/failed pass. would_wipe is this sweep’s own addition to the clock-guarded-retention shape (see docs/clock-guarded-retention.md’s entry) — unlike this file’s two DELETE-based siblings (reap_command_execution_mappings/reap_event_outbox, which deliberately decline part 1’s would-wipe probe because their tables drain to 100% expiry as routine behaviour), this sweep MUTATES live execution rows, so a pass whose candidates are an implausibly large fraction of all currently-running executions declines to act rather than risk mass-cancelling live work on what is more likely a systemic bookkeeping bug than a genuine backlog. clock_anomaly also covers a same-direction repeat anomaly that has not yet earned this sweep’s (anchor, direction) wedge-recovery match — see the doc entry for the recovery mechanism that replaced the sweep’s original permanent-wedge-after-any->24h-gap behaviour. capped fires once per ACCEPTED pass whose true, uncapped backlog exceeded kStuckExecReapCap (500) — mutually exclusive with would_wipe by construction (the latter always returns before capped is computed), same meaning as the Gateway routing directory metrics’ ok_capped above; a sustained non-zero rate means check yuzu_exec_tracker_bookkeeping_failed_total{op=set_agents_targeted|mark_cancelled} for the root cause and the per-pass info log for the specific ids. skipped fires when another replica already held the advisory lock this tick. Unlike a lock held for a whole tick interval, this one is acquired and released inside a single short transaction, so on a healthy deployment where replicas’ maintenance ticks are not phase-aligned, simultaneous contention is rare — a healthy rate is close to 0 on every replica, not a fixed (N-1)/N fraction. A sustained non-trivial rate, not only a reading near 100%, is itself worth a look; a rate approaching 100% ACROSS EVERY replica means a lock holder is stuck, worth inspecting via pg_locks.

Audit events

Structured JSON envelope:

{
  "timestamp": "2026-04-18T12:34:56.789Z",
  "principal": "alice",
  "action": "noun.verb",
  "target_type": "User",
  "target_id": "bob",
  "result": "ok|denied|error",
  "detail": "free-form context"
}

• Suitable for direct delivery to Splunk HEC or generic webhook sinks.
• Indexed by timestamp and principal for efficient queries.
• detail is SANITISED on read, not delivered verbatim. AuditStore::query() applies sanitized_detail() at the single point where a stored row becomes a value a caller can hold, so every reader (REST v1, the legacy REST route, MCP, Settings - six call sites) is covered by construction rather than by each remembering. Today the rule replaces the value= segment of a target_type="RuntimeConfig" row naming a secret-valued key: those rows recorded the plaintext credential before the write path was fixed, and they are still on disk. A reader therefore never receives the stored bytes of such a row; every other detail is returned as stored (the registry.write bullet below says so for its registry values). Any new audit reader must go through AuditStore::query() - reading audit_events directly re-opens the disclosure.
• Never add a detail filter or free-text search to AuditQuery. It has fields for principal, action, target_type, target_id, since/until and action_prefixes, and deliberately none for detail. A LIKE/search parameter over detail would let any AuditLog:Read holder binary-search the stored bytes and recover exactly the credentials the read-time sanitiser exists to withhold - converting the redaction into a disclosure oracle. The absence of that filter is what makes read-time redaction sufficient rather than cosmetic, so it is a standing constraint, not an unimplemented feature.
• A raw database extract will NOT match API or export output, for rows predating the read-time sanitiser. The rows are deliberately left byte-unchanged - rewriting audit history to remove embarrassing content is a worse posture than declining to disclose it, and there is no hash chain over audit_events that would make such a rewrite distinguishable from concealment. The consequence is a real discrepancy: an auditor given DB-level access sees the stored plaintext, while the same row sampled through GET /api/v1/audit shows value=<redacted>. Disclose that discrepancy proactively rather than letting it be discovered - it is a compensating disclosure control operating at the read layer, not a claim about what is stored.
• Denied operations MUST emit an audit event — spdlog::warn alone breaks the SOC 2 CC7.2 evidence chain. The one deliberate exception: an interactive cookie session refused on an auto-loading dashboard panel fragment (probe-gated; AuthRoutes::has_permission / probe_permission / probe_permission_unavailable_aware / probe_admin / probe_fleet_read / probe_global_list_read) writes no denial row, because nothing was attempted by the operator; token and engine callers always do. Example: the MCP Streamable HTTP transport (track 2f) emits mcp.session.open / mcp.session.close (result="success") and mcp.session.reject (result="failure", detail="reason=origin|protocol_version|unknown_session|missing_session_header|per_principal_cap|global_cap|id_generation|shutdown") on target_type="McpSession" — the reject verb fires on every transport denial (origin, protocol-version, unknown/expired session, cap), routed through the shared try_persist_audit kernel so a persist failure is logged, never silently swallowed. (Note: mcp.session.reject itself uses result="failure" for every transport-layer denial, not "denied" — tools/call (the mcp.<tool_name> verb) instead uses "denied" for most client-caused rejections (tier, read-only, schema, input-bounds, per-submitter cap, approval mismatch/replay — see docs/user-manual/audit-log.md’s “Result vocabulary”), with several known exceptions still client-caused but audited "failure" (approve_request/reject_request, set_tag’s validation-failure rejection, execute_bundle’s out-of-scope rejection, and the access-review campaign family among them — not an exhaustive list, tracked for a full sweep in #3176; delete_tag’s not-found rejection uses its own third token, "not_found", not "failure"). So the mcp.* verbs are not one surface-wide convention, and even within tools/call "denied" is the dominant pattern rather than an invariant. Do not author a mcp.* SIEM rule assuming one token family covers every verb; check the specific verb.)

MCP Streamable HTTP streams (track 2f PR 2) add two verbs — mcp.stream.attach and mcp.stream.close (the latter carrying reason= from the closed set client_disconnect|superseded|session_terminated|credential_revoked|auth_unavailable|internal_error|cancelled|cap_expired|record_gone|completed — the last four are the streamed-POST surface, produced by its pump — live wherever --mcp-enable-streamed-post is set, which is the default (opt out with --no-mcp-streamed-post)) — and widen the mcp.session.reject reason= set with not_acceptable, per_principal_stream_cap, global_stream_cap, stream_handover_pending, terminal_poisoned, and replay_window_exceeded (the last on mcp.session.close), plus the streamed-POST admission denials post_per_principal_cap, post_global_cap, post_record_cap, post_pin_slots, post_duplicate_request_id and post_unknown_session (2f PR 3b; post_record_cap added #2918). Both mcp.stream.* verbs carry surface=post on a streamed POST, so a GET attach and a POST attach stay separable in the log rather than both reading as “a stream”. Every value here is a static literal, no label ever derived from caller input — but only most are pre-seeded on the matching yuzu_mcp_stream_rejects_total{reason} / yuzu_mcp_stream_closes_total{reason} counter: terminal_poisoned (from the 2f PR 3a kPoisoned close reason) is missing from server.cpp’s hand-written pre-seed list (a pre-existing gap, not introduced here), so that series stays absent, not 0, until the first poisoning event — the same class of drift the kPostRejectReasons derived-list pattern exists to prevent for the streamed-POST family, just not yet applied here.

• Shutdown-time teardowns are evidence-silent only while what they reap stays independently recoverable (#2489 comp-S1, #2517). The MCP progress bridge audits every record it reaps on a live server (mcp.bridge.forced_expire, mcp.bridge.session_dead). A record shutdown()’s own walk finds already CLEANLY resolved (its terminal published, or a real POST-wire delivery, or normal projector settlement) is reclaimed silently: the process is terminating, the audit sink is itself tearing down, and the client’s result stays durably fetchable by execution_id, so the absent row evidences nothing that was actually lost. That is the exemption this convention describes, and it still applies to the ordinary case. A record shutdown() finds claimed but with its terminal disposition never resolved - a pressure-visitor claim (torn_down) whose teardown_claimed call never reached (or never completed) Step 1, most often because a concurrent sweep() lost the race against shutdown_started_ - does NOT get the exemption: shutdown() poisons the stream (any currently-connected client gets an honest close instead of heart-beating past process exit) and emits ONE aggregate mcp.bridge.shutdown_reap audit row naming the count, rather than per-record rows (publishing per-record evidence here would itself violate teardown_claimed’s exactly-once arbitration - the abandoned claim is never re-entered, only poisoned). Read a gap between yuzu_mcp_bridge_records_active at its last scrape and the reap rows in the log, with no shutdown_reap row either, as a clean shutdown; a shutdown_reap row names exactly how many were not. This inference holds only while the reap walk itself completes without throwing - the whole walk still runs inside one swallowing catch (...) (#2515, unaddressed by this fix), so a throw partway through can leave both an absent row AND an unresolved record, which is NOT the clean-shutdown case. No metric backs this - the process is exiting and the series would never be scraped, so the audit row is the only durable evidence. The convention generalises: a background reaper may skip its audit on the shutdown path only while what it reaps stays independently recoverable OR was never left in an unresolved state to begin with. One that reaps something a client cannot re-fetch, or whose disposition was never decided, owes an aggregate “N records reaped at shutdown” row instead.
• HA WS-3 3.3 splits a scheduled fire’s audit trail into two verbs. instruction.schedule_fired (result="queued") now marks the poller’s enqueue onto the durable command outbox, not the actual dispatch; a new verb, command.outbox_delivered (result="success"|"failure"|"denied"), marks the delivery loop’s real send outcome — success on dispatch (detail carries the agents-reached count), failure for a recorded-and-skipped no-agents occurrence or a decode failure, denied when the send-time re-authorization check refuses an occurrence whose authority was revoked since enqueue. A SIEM rule correlating “one audit row per scheduled fire” must be updated to expect the pair (docs/user-manual/server-admin.md’s WS-3 3.3 Upgrade Notes entry has the full operator-facing description).
• session.gateway_peer_denied (gateway-upstream peer authorization) is audited for a denial that names a principal, and only metric-and-log for one that does not. The convention that denied operations MUST emit an audit event holds for a denial from an authenticated certificate holder: a refused call whose certificate parsed far enough for its public-key hash to be computed (no_server_auth_eku, not_pinned, outside_validity, and internal_error when the key was computable) writes one row, result="denied", principal="gateway-peer:<spki8>" (the first 8 hex of the hash), target_type="GatewayRpc", target_id the snake_case RPC name, detail="reason=<reason> rpc=<rpc> spki=<16 hex>" (the first 16 hex of the hash), principal_role="unverified_peer" (a refused peer is by definition not an authorized gateway), and source_ip the transport peer. The action is deliberately in the session.* family so it falls inside the CC7.2 auth-sample export. Carve-out: a denial with no resolvable authenticated principal writes NO audit row (null_context, not_authenticated, no_cert, where no certificate reached the guard, bad_cert, where one did but did not parse, and an internal_error raised before the certificate’s key could be computed, so there is no key to attribute). That is the same split the OTA identity gate already makes for reason=no_client_identity (agent_service_impl.cpp, with docs/user-manual/audit-log.md’s session.ota_identity_rejected entry): there is no principal to attribute, and a synchronous audit write on that path would let an unauthenticated or certificate-less caller turn its own refusals into unbounded audit-store writes (that volume argument covers null_context, not_authenticated and no_cert; bad_cert is reached only for a transport-authenticated peer whose certificate does not parse, so for it the reason is that there is no key to attribute a row to). These denials are still evidenced: the unsampled yuzu_server_gateway_peer_denied_total counter is complete, and the server logs the refusal. Because even a keyed caller could otherwise flood the store, the rows go through a fixed-window budget (bounded_audit_budget.hpp) keyed by reason and key prefix: 10 rows per key per 10 s window, and beyond 64 distinct keys in a window new keys are charged to one overflow bucket per audit-eligible reason (4 reasons, so at most 680 rows per window per replica), so a flood of one reason cannot use up another reason’s overflow bucket; individual rows can then be hidden, and yuzu_server_gateway_peer_denial_audit_suppressed_total says how many. The budget bounds ROWS only: the refusal is decided first and unconditionally, and a budget refusal or an audit write failure never changes it. Audit-write failure posture: set-and-proceed. A failed write is logged (gateway peer guard: audit write failed for a denied call (rpc=<rpc>)) and the call is still refused; the guard deliberately does NOT set the x-yuzu-audit-failed trailer the other gRPC audit paths use, because the deny is already final and the complete counter exists whether or not the row lands. A budget-suppressed row is counted and appears only in the per-window summary warning; a failed write has no counter of its own, only the audit write failed warning. The log line for a denial with no key carries the transport peer (gateway peer denied: rpc=<rpc> reason=<reason> peer=<ip>, rate-limited per reason), the only attribution an anonymous caller has. Nothing is written in --insecure-gateway-peer mode, where no call is refused.
• Store-availability 503 guards do NOT audit. A request rejected because a store’s database never opened (is_open() gate, e.g. the /api/v1/tokens routes per #347 CH-3) is an operational event, not a principal action: the guard runs before any token/principal interaction, so there is no operation to evidence. The CC7.2 evidence for the outage is the boot-time spdlog::error plus the store’s entry in the /readyz conjunction (and, once #1385 lands, the store-readiness gauge). This is the platform-wide convention for every store-down guard — do not add per-request audit rows to these paths. One declared exception: an audited READ. GET /api/v1/ca/issued and MCP list_issued_certs already write a crypto.ca.certificates.view row for every served page, so a request the server could not serve (the CA store is not open, list_issued failed, or the access check could not be completed) writes a result=failure row under the same verb, as the filtered form and crypto.ca.stats.view do: the read is itself evidence, and an attempted read that left no row would be an invisible gap next to rows for the pages that were served. During an outage that is one REST row per call (two on MCP, with mcp.list_issued_certs); if the audit store shares the failing pool those writes are lost too and move yuzu_server_audit_emit_failed_total. A store-down guard in front of a route that has no audit verb of its own still writes nothing.
• registry.write (the registry plugin’s mutating actions) uses the existing AuditEvent fields only, and is written by the registry plugin 2.0.0 and later, on agents that stamp the action: no column or envelope change; result comes from the whole line’s first result= token (never dry_run=). The evidence it does NOT record is counted by yuzu_server_registry_audit_dropped_total{reason} (suppressed and persist_failed count rows, unmintable_id and truncated count frames, so do not sum across reason), for the losses the server sees while its audit store is up; the log lines are for a human. Rows are minted only for a response the server’s own dispatch record labels registry: a registry command with no record (stored ~unverified) mints no row and moves no reason of this family (see yuzu_server_response_provenance_total{outcome}). Each agent id is budgeted to 600 rows per fixed 60-second window on each server process (bounded_audit_budget.hpp), because a bound per frame is not a bound on frames. No alert rule ships for this family yet, deliberately: the thresholds want fleet data. Of the four reasons only suppressed is deliberately alert-less; unmintable_id, truncated and persist_failed are counted and not alerted until the registry alert rules of #6296 land, and a row the store itself fails to write also moves yuzu_server_audit_emit_failed_total (alerted as YuzuAuditPersistFailures, whose text speaks of behavioural-data routes returning 503: the registry hook does not, it sets and proceeds); a row never attempted because shutdown closed the hook mid-frame, or lost to the hook’s own exception arm, is counted as persist_failed only and does not move it. A failed store write ends that frame’s remaining rows, is logged, moves yuzu_server_audit_emit_failed_total and, on the direct session, adds the x-yuzu-audit-failed trailer once per stream (delivered when the stream closes; the gateway path has no trailer, so the one log line the hook writes for the failed frame is its signal). The detail is the server’s agent=<id> and cut=0|1 tokens (cut=1 for a cut line) followed by the plugin’s line (old and new registry values, as emitted apart from the bounds, raw hex that can contain secrets: the product owner accepted this risk on 2026-10-20 and masking is a planned later change; readers are the AuditLog:Read and Response:Read holders and the legacy /events stream): NOT read-redacted. Binding, bounds, the budget and the join to the operator’s row: docs/user-manual/audit-log.md.
• response.forensics.read (reads of Forensics-tier stored responses) is a separate, usage-class verb written only when Forensics rows are actually served (#4990). target_type="Execution", result="success", detail="forensics_rows=<n> plugins=<a,b> surface=<s>" (every token through audit_token; surface is the closed set rest, legacy, mcp, dashboard, drawer, executions_page, hardware, bundle, chosen by the server and never taken from the request). It is emitted by the row-returning response surfaces (legacy /api/responses/{id} and /export, REST v1 execution/response/export/visualization, MCP query_responses, the dashboard results fragment, the executions drawer, the Hardware Actions-lens result poll /fragments/hardware/ci/result, bundle collation GET /api/v1/bundles/{id} and MCP get_bundle_result with surface=bundle), by forensics_read_audit.hpp’s ForensicsServed/emit_forensics_read_audit, which inspects the served rows’ plugin (no second store query); a caller who is filtered out of the tier, or whose result holds only ordinary-plugin rows, leaves no response.forensics.read row (an API-token, MCP-token or engine caller without a global Forensics:Read still leaves its auth.permission_required denial row per read attempt; an interactive cookie session does not). Persist semantics follow rest_audit.hpp per surface: REST fails closed (503 + Sec-Audit-Failed), MCP reports audit_persisted:false, dashboard HTML (including the Hardware result poll) proceeds with the header. Aggregate, facet and count reads write nothing (nor does the dashboard filter-bar fragment GET /fragments/results/filter-bar, which serves facet option values to a Forensics:Read holder unaudited). The verb counts read requests, not viewings or rows: an open executions drawer on a running execution re-fetches after each state change and writes a row per refresh that serves a Forensics row, and an export’s forensics_rows is taken before the byte cap trims the file. Bundle collation counts one served row per RESPONDED step run by a Forensics-tier plugin (summarize_forensics_served(const BundleAggregate&, ...)): a step the caller’s tier filter excluded collates as pending and is not counted, and a bundle with no served Forensics step writes nothing; each poll that serves such a step writes the row again. The Forensics plugin set is derived from the capability catalogue, never hand-kept.
• yuzu_server_forensics_tier_withheld_total{reason} counts reads whose Forensics-tier visibility could not be resolved (#4990), and a definitive refusal is never counted. reason is the CLOSED set rbac_unavailable, unwired (kForensicsUnknownReasons, pre-seeded to 0 by seed_forensics_tier_metrics), so the series is flat on a healthy server whatever the callers’ roles and any increase means a real fault. There is no scoped (management-group) read tier and so no reason for one. An unavailable global Forensics:Read check is Unknown for sessions, API tokens, MCP tokens and engine principals (the engine branch decides through engine_grant_checked, which refuses a degraded view without reading a grant, so an unconfirmed enforcement state is Unknown on the opt-in mapping); a service-scoped token is default-denied instead (a definitive refusal, not counted), and when a whole-store outage fails the surface’s earlier admission gate first, that gate’s own answer is returned and this counter does not move. A failed role query is counted as yuzu_server_rbac_read_degrade_total{reason="query_error"} and is a breaker failure. Operational, not event="security". The rule YuzuForensicsTierUnresolved (docs/prometheus/yuzu-alerts.yml) is sum by (instance, reason) (rate(yuzu_server_forensics_tier_withheld_total[5m])) > 0 with for: 10m: a single refusal, or a burst shorter than about 5 minutes, does not fire, and a failure that keeps recurring fires after about 10 minutes (the edge is the 5m rate window: a refusal that recurs only every 5 minutes or longer leaves the 5m rate false between events and does not fire, by design, while one recurring about every 285 seconds still fires). The same decision also logs one warning per reason per minute (the counter records every occurrence).
• yuzu_server_response_provenance_total{outcome} counts every command response frame by how its plugin label was resolved (#4990). outcome is the CLOSED set verified, unrecorded, lookup_degraded, pre-seeded to 0 at boot (server.cpp); keepalive frames (claim-renewal heartbeats) are never counted. The label comes from the server’s dispatch record (command_execution.plugin), never from the command id the agent echoes. unrecorded and lookup_degraded frames are stored as ~unverified, a member of the Forensics plugin set, so they are withheld from callers without Forensics:Read: a sustained rate means failed dispatch-record writes (yuzu_exec_correlation_write_degrade_total), a record reaped under a command that outlived its 24 hour window, or an endpoint echoing ids the server never issued. Operational, not event="security". Alert rules: YuzuResponseProvenanceLookupDegraded (sum by (instance) (rate(yuzu_server_response_provenance_total{outcome="lookup_degraded"}[5m])) > 0 for 10 minutes) and YuzuDispatchRecordWriteFailing (the same shape on yuzu_exec_correlation_write_degrade_total). There is deliberately no rule on unrecorded: it is non-zero on a healthy fleet (an id the server never issued, a record reaped after 24 hours, an empty id), and a failed write that causes it is already covered by the write rule.
• yuzu_exec_correlation_write_degrade_total counts dispatch-record writes that did not land (#4990), including writes a per-loop latch skipped. It is unlabelled and pre-seeded to 0. A write that failed (no pool lease within the budget, a lock or statement timeout, a query error, no tracker wired) counts once. In a loop that writes one record per target (today only the operator Guardian rule push, DispatchRecordLatch in dispatch_record_policy.hpp), the first failed or timed-out write trips the latch and every later write in that loop is skipped and counted too, so one real write attempt during an outage yields one degrade per target, and the counter size is the number of affected commands, not the number of store attempts. Each skipped command is still dispatched and its responses are stored ~unverified. Single dispatches and all other callers are never skipped. Alert rule: YuzuDispatchRecordWriteFailing, whose expression is unchanged by the latch. Operational, not event="security".

Event format (envelope)

All events — lifecycle, compliance, audit, system — follow a common envelope so downstream parsers can demultiplex by event_type without per-source schemas:

{
  "event_type": "...",
  "timestamp": "...",
  "source": "server|agent|gateway",
  "payload": { ... }
}

Response schemas

All instruction response data must be typed for downstream consumption (ClickHouse, Splunk, etc.). Permitted column types:

• bool, int32, int64, string, datetime, guid, clob

clob is reserved for large text fields with configurable truncation. See docs/data-architecture.md for the analytics-integration view.

Where to find dashboards

• Grafana dashboards: deploy/grafana/*.json (operational set, auto-provisioned into the UAT/full-UAT Grafana — Prometheus-backed yuzu-{dashboard,fleet,gateway} plus the ClickHouse-backed yuzu-analytics); docs/grafana/yuzu-overview.json (standalone Prometheus import template). See docs/grafana/README.md.
• Prometheus alert rules: docs/prometheus/yuzu-alerts.yml; scrape-config example: docs/grafana/README.md
• Alert-rule unit tests: tests/prometheus/yuzu-alerts.test.yml, run under promtool by tests/prometheus/run_promtool_tests.py (the docs-lint workflow’s prometheus-rules job and meson test --suite docs both call it; that script’s docstring carries the skip-vs-fail contract and the reasoning). Changing a rule’s expr means updating these — write the new case, watch it fail against the old rule, then change the rule. Cross-metric rules use PromQL’s default all-label matching; an explicit on(...) narrows the join and needs a stated reason, because any other series agreeing on the narrowed labels can then stand in for the one you meant (#2553).
  • What the gate proves, and what it does not. Every rule is parsed (promtool check rules). Only rules carrying cases in the test file are behaviour-tested — the audit-retention liveness family (YuzuAuditRetentionNotRunning, YuzuAuditRetentionNeverRan, YuzuAuditRetentionMetricMissing; #2553/#2854) plus the process-health YuzuServerRestartLoop (#2854 rung C), the YuzuResponse* reap/ingest family, the yuzu-ha-leader group (YuzuLeaderSplitBrain, YuzuLeaderNoLeader, YuzuLeaderFlapping, YuzuLeaderCoordinationStale; #4014), the Guardian reconcile pair YuzuGuardianReconcileDegraded and YuzuGuardianAgentGenerationAhead, and YuzuMcpStreamedPinSlotsWedged’s for: 15m window (#2792) — a minority of the file’s alerting rules (grep - alert: for the current total rather than trusting a count written here; an earlier revision hardcoded “3 of 64” and both numbers went stale). A green run on the untested rest proves they parse, not that they can fire. The run additionally fails if it would be vacuous (no rules, no cases, no assertions, a stale rule_files: glob) and if the behaviour suite is not exercising the shipped rules at all: a canary breaks a copy of the rules file and requires the suite to redden, because check rules and test rules are handed different paths and can otherwise disagree silently while reporting a rule count.
  • The job is not a required status check. Branch protection is configured outside this repo, so a red prometheus-rules can merge today. Do not describe alert rules as “gated” without confirming that setting.
  • Testing a not-yet-enabled (commented-out) group. promtool test rules can’t test commented YAML, and run_promtool_tests.py can’t un-comment a group in production just to test it. The pattern (#2340 PR-4): mark the block with paired # PROMTOOL-EXTRACT-BEGIN: <group> / # PROMTOOL-EXTRACT-END: <group> comments (see docs/prometheus/yuzu-alerts.yml’s yuzu-guardian-journal group for the live example), and the script’s extract_marked_block pulls it out, strips exactly the # prefix (two-space indent, load-bearing — a group nested at a different depth fails closed rather than extracting wrong) from each line, and runs the result through the SAME check/test/canary machinery as the shipped rules — off a generated temp file, never written back to the production rules file. Fails closed on a missing/duplicated marker, an empty extraction, or a non-comment line caught between the markers (each has its own --selftest case). Use this for the next not-yet-enabled candidate group rather than reinventing a parallel mechanism.
• ClickHouse ingest setup: docs/clickhouse-setup.md