Metrics

Yuzu exposes a Prometheus-compatible /metrics endpoint on the server and provides real-time event streaming for dashboard live updates. This page covers how to scrape metrics, what is exposed, and how to connect Grafana or other monitoring tools.

Metrics endpoint

GET /metrics

Returns all server and connected-agent metrics in Prometheus exposition format. Localhost requests (127.0.0.1, ::1) are always unauthenticated for Prometheus scraping. Remote requests require authentication by default. Use --metrics-no-auth (env: YUZU_METRICS_NO_AUTH) to allow unauthenticated remote access for external monitoring infrastructure.

curl -s 'http://localhost:8080/metrics'

Example response (excerpt):

# HELP yuzu_http_requests_total Total HTTP requests by method, status, and principal_class
# TYPE yuzu_http_requests_total counter
yuzu_http_requests_total{method="GET",status="200",principal_class="human"} 1542
yuzu_http_requests_total{method="POST",status="200",principal_class="agent"} 87
yuzu_http_requests_total{method="GET",status="404",principal_class="none"} 12
yuzu_http_requests_total{method="POST",status="200",principal_class="engine"} 34

# HELP yuzu_command_duration_seconds Command execution latency in seconds
# TYPE yuzu_command_duration_seconds histogram
yuzu_command_duration_seconds_bucket{le="0.005"} 12
yuzu_command_duration_seconds_bucket{le="0.01"} 47
yuzu_command_duration_seconds_bucket{le="0.025"} 180
yuzu_command_duration_seconds_bucket{le="0.05"} 540
yuzu_command_duration_seconds_bucket{le="0.1"} 980
yuzu_command_duration_seconds_bucket{le="0.25"} 1320
yuzu_command_duration_seconds_bucket{le="0.5"} 1480
yuzu_command_duration_seconds_bucket{le="1.0"} 1530
yuzu_command_duration_seconds_bucket{le="2.5"} 1541
yuzu_command_duration_seconds_bucket{le="5.0"} 1542
yuzu_command_duration_seconds_bucket{le="10.0"} 1542
yuzu_command_duration_seconds_bucket{le="+Inf"} 1542
yuzu_command_duration_seconds_sum 198.74
yuzu_command_duration_seconds_count 1542

# HELP yuzu_agents_connected Number of currently connected agents
# TYPE yuzu_agents_connected gauge
yuzu_agents_connected 47

yuzu_http_requests_total also counts the responses served on the plain-HTTP redirect port (--web-port, when HTTPS is on): its 301s, 400s and the four health-probe answers. A plain-port 503 (https_listener_not_serving) while the HTTPS listener is down is counted in yuzu_http_requests_total, but it never reaches an error-ratio alert such as the example YuzuHighErrorRate: while that listener is down the plain /metrics is itself a 301 to it, and the listener binds once per process, so recovery is a restart that resets the counter and those 503s are never scraped (use --healthcheck or up == 0 to catch that state). The method label is limited to the HTTP methods the server accepts (GET, HEAD, POST, PUT, DELETE, CONNECT, OPTIONS, TRACE, PATCH, PRI); any other request-line token is reported as UNKNOWN, so a malformed request cannot create new series. Accepted methods keep their exact label, so existing dashboards and YuzuHighErrorRate (which uses status only) are unchanged.

Naming conventions

All Yuzu metrics follow a consistent naming scheme.

Internal-CA / default-certificate metrics

Gateway certificate metrics

Scraped from the gateway (:9568, job yuzu-gateway). Full description, the label table and the runbook: Certificate rotation and expiry in gateway.md.

Gateway upstream permit and held-notification metrics (#5695)

Scraped from the GATEWAY (job yuzu-gateway). Every label is a closed set and every series below is created at 0 when the gateway starts. The design and the numbers are in the gateway manual: “Upstream call permits”, “Stream Status Notification” and “Planned stop”.

yuzu_gw_registration_replay_triggered_total{trigger} gained the values notify_loss (a full replay started because a held CONNECTED was lost), server_lost_state (a heartbeat verdict showed the server lost its state) and manual; breaker is no longer incremented, because a breaker recovery no longer replays every agent. yuzu_gw_upstream_rpc_errors_total{code} gained the value connection_lost (a call ended because its upstream connection closed or went away; the breaker counts one such connection once, however many calls it ended), which replaces stream_error and connection_error for those calls, and the value too_many_streams (the gateway did not open the stream because the connection was at the server’s concurrent-stream limit; nothing was sent; a proxied call or replay waits for a permit again and does not count against the breaker). The pressure lane serial of yuzu_gw_upstream_lane_saturated_seconds_total{lane} is now permits (a live Register or inventory report waiting for a permit). Apart from YuzuGatewayNotifyUnacknowledged (refused notifications still waiting after 10 minutes), no alert ships for these; a sustained rise of yuzu_gw_upstream_notify_lost_total{event="disconnected"} or of yuzu_gw_upstream_permit_refused_total{class="register",reason="queue_full"} with a healthy server is worth a look.

Gateway runtime metrics (#5695)

Scraped from the GATEWAY (job yuzu-gateway). These say whether the gateway itself is keeping up: whether one of the processes every agent’s traffic passes through is backing up, whether a listener or the process is close to the most connections it can hold, and how often the HTTP/2 server closed a connection on its own. They are sampled every telemetry_gauge_interval_ms (10 s by default) by yuzu_gw_gauge, at a cost that does not grow with the fleet: a fixed list of nine named processes, one ranch:info/1 per listener, one call per management connection, never a walk of every process or connection. The router’s fan-out gauge is set by the router itself whenever the number changes, and the close counter is incremented when the close happens. Every label is a closed set, and every series of a closed label set exists at 0 from start.

yuzu_gw_beam_scheduler_util now counts the normal schedulers only. It used to average the dirty schedulers in as well, which are mostly idle, so a gateway whose normal schedulers were all busy read about 0.3; it now reads about 1.

Per-process actions (YuzuGatewayProcessBacklogged, or a high yuzu_gw_process_queue_wait_seconds):

• upstream, upstream_channel, heartbeat_buffer: the server is slow or unreachable. Check the server and its database, and YuzuGatewayUpstreamSlow and the circuit breaker; restarting the gateway does not help and disconnects its agents.
• registry, agent_sup, pg: registrations and disconnects are arriving faster than they are handled. Look for agents reconnecting in a loop (rate(yuzu_gw_agents_connected_total[5m]) far above the fleet’s normal churn) and for what makes them reconnect.
• router: commands are fanned out faster than they complete. Slow the dispatching side (scheduled instructions firing together).
• upstream_regulator: callers waiting for upstream call permits; read yuzu_gw_upstream_permits_waiting and the server’s health.
• logger: the gateway is writing too many log lines. Raise its log level; OTP’s handler drops lines rather than grow without bound, so this alone does not stop the gateway.

Gateway agent owner restarts (#5695)

Scraped from the GATEWAY (job yuzu-gateway). Each agent session is held by one owner process on the gateway; one that dies abnormally is restarted, sends its session’s DISCONNECTED and recovers the session on the agent’s open stream (gateway manual, “When an agent’s owner process crashes”). The series exist at 0 from start.

Gateway metric index

Every other metric the gateway exports, so this page names each one. The gateway manual’s Prometheus Metrics section has the full description of each; the sections above cover the certificate, permit, held-notification and runtime families. node is the gateway’s Erlang node name. A family marked “first event” has no series until its first event, because its label values are not known at start (a plugin name, a TLS alert, a peer node) or because it is a histogram; every other family exists from start.

A test (yuzu_gw_metrics_scrape_tests) starts the gateway from its shipped configuration, fetches /metrics over HTTP and fails if any metric the gateway declares is missing from the scrape or from this page, if this page names a yuzu_gw_ metric the gateway does not declare, or if an alert rule reads one.

Crypto dashboard metrics

The /crypto Yuzu-issued certificate reads (GET /api/v1/crypto/ca/stats, MCP get_crypto_ca_stats, the dashboard cards and the filtered GET /api/v1/ca/issued drill) and the per-operator layout reads and saves (GET/PUT /api/v1/crypto/dashboard/layout and their MCP twins) count every refusal and failure on the inventory read-degrade counter, under their own source values. No label carries a user name or a widget id.

The same counter, its other sources and the per-degrade log sampling are described in Inventory. Two alerts read it: YuzuInventoryReadDegraded (any source, per reason, for 5 minutes) and YuzuCryptoCaReadDegraded (sum(rate(yuzu_inventory_read_degrade_total{source="ca_issued"}[5m])) > 0 for 15 minutes, warning): the Yuzu-issued widgets have answered 503 for long enough to need a person. Split it by reason: admission_full means too many concurrent /crypto reads (a poll storm) or a pool of 3 connections or fewer, so raise the pool or find the caller; pool_acquire_timeout and query_error are database faults, so check the yuzu_pg_* pool metrics and the server’s CaStore warnings (they carry the database error; sampled: the first of an episode, then every 100th). A query_error whose warning reads canceling statement due to statement timeout on every read is not a transient fault: the counts statement has outgrown its 5 s bound (it is sized for about a million live certificates, ADR-0053 “Update (2026-10-20)”), and retrying does not help; store_not_open means the CA store did not open at boot. See Crypto: Failure behavior, which is also the alert’s runbook_url.

Inventory purge metrics (typed-source hygiene)

Executions event-outbox metrics (HA WS-2a)

The durable event_outbox (ADR-2002 §5) shadows the execution-history SSE events and, on a multi-replica deployment, is drained cross-replica so a subscriber on one server replica sees live execution progress driven from any replica. See docs/executions-history-ladder.md.

Execution bookkeeping + stuck-execution reap metrics (#4982)

ExecutionTracker::set_agents_targeted/::mark_cancelled failures on the synchronous dispatch path were previously log-only; a separate clock-guarded background sweep, reap_stuck_running_executions() (~15-minute cadence, see docs/clock-guarded-retention.md’s entry for the full seven-part design), recovers an execution row left permanently stranded at status='running' as a result.

No dedicated alert rule ships for either family in this change — same posture as the Gateway routing directory reap counter below, which also has none; revisit once real fleet data exists to size a threshold.

Command outbox delivery metrics (HA WS-3 3.3)

Scheduled instruction fires enqueue a durable pending occurrence to CommandOutboxStore (schema command_outbox_store) instead of dispatching inline; a leader-gated delivery loop drains it and performs the actual wire dispatch. See docs/user-manual/instructions.md “Execution semantics” and docs/postgres-migration-ladder.md’s CommandOutboxStore row.

There is still no dedicated alert on this family: monitor it manually by alerting on yuzu_server_command_outbox_pending growing without bound, or on yuzu_server_command_outbox_deliver_retry_total climbing steadily. The most common cause of a stalled delivery loop (no replica holds leadership) IS covered, by YuzuLeaderNoLeader (see “Leader election metrics” below).

Leader election metrics (HA WS-11, #4014)

Each server replica reports its own view of the fenced background-work leader elector (ADR-2002 §3/§6/§10) on its own /metrics. The series exist only on a server that has an elector (a Postgres-backed server); every series except yuzu_server_leader_last_ok_age_seconds (absent until the first success) is seeded at 0 at boot so a later event shows as a rise.

Because the counters are published at scrape time, a counter that already moved before the first scrape (the leadership acquired in the first seconds after boot) is first seen at its non-zero value, so increase() does not count that first event. They are published when /metrics is scraped, from a lock-free snapshot, so a stalled dedicated coordination connection can never stall a scrape and the age gauge keeps growing if the election loop wedges.

These series are per replica. Scrape each replica directly (never through the load balancer) and put unrelated Yuzu deployments that share one Prometheus in separate jobs. Do not sum them yourself except as the shipped alerts do (sum by (job) (max by (job, instance) (yuzu_server_leader_is_leader))). /metrics follows the access rule at the top of this page: a loopback source address is unauthenticated (the same posture as yuzu_server_pg_reachable), while a remote scraper needs a token or --metrics-no-auth. The series carry no identity, only counts and one epoch number.

What these do not cover, and why /readyz ignores the elector. Leadership is not readiness: a replica that is not the leader, or whose elector never opened, is a ready follower and still serves every operator-synchronous path, so /readyz and /health deliberately do not consult the elector. A paused background loop is therefore visible only through these series and the yuzu-ha-leader alert group. Those alerts need a successful scrape: when the whole Postgres primary stalls (its kernel still acknowledging packets) the /metrics handler’s own database queries hang, the scrape fails and every leader series goes stale together. No server up == 0 alert ships (#2956 tracked it and was closed without building it), so configure one on your own scrape job as the backstop. Not yet shipped (tracked): remediation-claim lifecycle counters (#6526) and a CRL nextUpdate freshness gauge (#6527).

Plugin-enablement store metrics (ADR-3006)

The durable enablement store (EnablementStore, over the plugin_config_store schema) and its continuity anchor enablement-anchor (see docs/user-manual/server-admin.md, “Enablement continuity anchor”). Every label set is closed and pre-seeded, so each family exists, at zero, on a healthy server and an absent() alert stays meaningful. The alert group is yuzu-enablement in docs/prometheus/yuzu-alerts.yml; only the regression and anchor-write alerts can fire until the later change that issues desired state.

Postgres reachability metrics (HA WS-8)

Each server replica runs a dedicated reachability probe (its own connection, not the pool) that feeds /readyz’s gating pg_reachable row — see docs/user-manual/server-admin.md “Health Endpoints”. These metrics are that probe’s view, per replica.

Agent presence metrics (HA WS-5 slice 1)

OfflineEndpointStore (schema endpoint_state) is the cross-replica agent presence source AgentRegistry::all_ids()/evaluate_scope() merge into scope-evaluation visibility (ADR-2002 §7a). See docs/postgres-migration-ladder.md’s OfflineEndpointStore row for the v3 migration and docs/observability-conventions.md for the fail-soft posture this counter covers.

There is deliberately no success/rate counter and no dedicated alert rule yet — same posture as yuzu_server_gateway_route_desync_total below: a background rate worth paging on needs real fleet data once a multi-replica deployment exists to generate any.

Gateway routing directory metrics (HA WS-4 4.1)

GatewayRouteStore (schema gateway_route_store) is the fenced agent→cluster routing directory written on the gateway-upstream connect/disconnect/heartbeat paths (register_fresh/announce_connected/deregister/renew_leases). See docs/postgres-migration-ladder.md’s GatewayRouteStore row for the full epoch-fence design. The desync counter below was added in slice 4.2a; the reap-outcome counter was added alongside it in the same slice’s governance fold.

yuzu_server_gateway_route_degraded_renewal_total{op="batch_heartbeat"} (pre-seeded to zero) counts BatchHeartbeat BATCHES, not sessions: it rises by one for each batch whose directory read degraded and that still renewed at least one locally hinted (agent, session) pair, incremented only after that renewal write succeeded. A rising rate means Postgres reads are browning out while writes succeed and heartbeat leases are kept alive from local hints; a degraded read whose renewal write ALSO fails moves write_failed{op="renew_leases"} instead.

yuzu_server_gateway_batch_heartbeat_duration_seconds{outcome} (histogram, #5686) records how long each BatchHeartbeat call takes, from after the on-behalf check to the response. outcome is a closed set of two, both pre-seeded: degraded when at least one Postgres-backed step of that batch degraded (the session_lookup directory read, the renew_leases write, or a presence upsert), ok otherwise. No label carries a session, agent or gateway id. Buckets are the standard latency set (0.005 to 10 s); the 0.5 and 1.0 boundaries matter because the directory read and the lease renewal each bound their pool acquire at 500 ms, so a batch that waited out one of them lands in the 1.0 bucket and one that waited out both lands above it. A useful SLO query is histogram_quantile(0.99, sum by (le, outcome) (rate(yuzu_server_gateway_batch_heartbeat_duration_seconds_bucket[5m]))). No alert ships for it yet (there is no fleet data to set a latency threshold against; the decision is recorded in docs/prometheus/yuzu-alerts.yml). A batch slowed by route-directory failures (session_lookup, renew_leases) also raises YuzuGatewayRouteWriteFailed, but a batch slowed by presence-store trouble does not: for that, watch yuzu_server_agent_presence_upsert_skipped_total, the outcome="degraded" series, and the pool alerts YuzuPgPoolSaturated and YuzuPgAcquireWaitHigh. The histogram also times the inline per-heartbeat reconcile hooks (quarantine, guardian) and fleet-snapshot work, so a batch slowed by them is labelled ok. After a failed presence upsert, under total saturation (every attempt fails), a batch makes at most floor(log2 N)+1 attempts on a failing presence store (probes after the first failure, see yuzu_server_agent_presence_upsert_skipped_total), each waiting at most its 250 ms pool acquire (an attempt stalled on a row lock is bounded by the pool lock_timeout instead); under partial brownout or flapping, a success resets the probe spacing, so the number of waits can exceed that bound but never exceeds what the old per-entry behaviour would have waited (a strictly alternating failure pattern can still approach about N/2 failed waits, about 41 s at N=330 with 250 ms acquires, still half of the old N).

There is deliberately no success/rate counter for either the write-failed or the desync counter. Two alert rules shipped with HA WS-4 4.2b (two more, YuzuGatewaySessionAmbiguous and YuzuGatewaySessionHeldByOtherAgent, with HA WS-5 slice 2, on expected-zero desync pairs: one for the first, two for the second, proxy_register and announce_connected), in docs/prometheus/yuzu-alerts.yml’s yuzu-gateway group: YuzuGatewayRouteWriteFailed (rate-based on yuzu_server_gateway_route_write_failed_total, #4246 item #1’s alert-rule half) and YuzuGatewayRouteUnreadable (on yuzu_server_command_outbox_deliver_retry_cause_total{cause="route_unreadable"} — see “Command outbox delivery metrics” above — the closest available signal for #4246 item #8’s alert-rule half; a degraded directory read during dispatch, not every guard-rejection outcome yuzu_server_gateway_route_desync_total tracks). The desync counter’s own guard-rejection outcomes (shortfall/session_mismatch/unknown_session/malformed_session_id) remain without a dedicated alert — they’re expected at a low background rate (see the row above) and a threshold that doesn’t page on a routine post-restart baseline rise needs real fleet data to set. /health agents.online (or yuzu_agents_connected) not ramping back after a server restart means the reconcile is not converging (an older gateway, or a verdict that never reaches the replay); after an outage past 270 s a SUSTAINED renew_leases/unknown_session rise is a second sign, but after a shorter server-only restart that counter stays flat whatever the replay does. Both are described under What happens when the server restarts in gateway.md; neither is a routine baseline. The two HA WS-4 alerts use conservative/INITIAL thresholds for the same reason (the #913 OTA-bounds lesson: alert thresholds are the top source of external-review findings), to be tuned once fleet data exists.

The directory read the route_unreadable alert covers is FALLBACK-ONLY as of 4.2b (HA WS-4 4.2b Task C, dispatch_route_fallback.hpp): confined dispatch consults GatewayRouteStore::lookup_routes only for a target the local AgentRegistry has no live session for, and it changes zero routing outcomes on today’s single-replica monolith — see docs/adr/2002-high-availability-architecture.md §7’s 4.2b status paragraph.

SSO login metrics

Every SAML callback outcome, and every OIDC callback outcome except the two group-provisioning denials below, increments its provider’s login counter (success or failure). The two OIDC group-provisioning denials (the asserted-group count over the cap, and a failed group reconcile) are counted only in yuzu_auth_sso_group_provision_total{result="error"}, not in yuzu_auth_oidc_login_total. (A callback that finds no OIDC provider configured at all answers 404 and is not counted either.) Both counters carry a uniform {result, role} label set on every series (including error paths), so a dashboard can group by either label without hitting an unlabelled/labelled split.

Local password metrics (#5342, #5274)

The self-service change route’s lockout check shares /login’s instrumentation: a fail-closed auth-store refusal there — the lockout counter could not be recorded, or the account could not be read to verify the current password — increments yuzu_auth_secret_unavailable_total{route="password_change"} and yuzu_auth_read_degrade_total{route="password_change",reason} (both pre-seeded to 0); at /login the same read failure now answers 503 instead of a 401 with a lockout strike (see docs/auth-architecture.md, login/Postgres decoupling).

yuzu_auth_secret_unavailable_total also carries route="approval" (#6079, pre-seeded to zero at boot with the other routes): approving or rejecting an approval request reads the approver’s own MFA factor, and when that read fails closed (the auth store is unavailable, the stored TOTP secret is missing or cannot be decrypted, or the session’s stored proof cannot be read from the durable session store, which also increments yuzu_auth_session_store_degrade_total{op="validate"}) the route answers 503 with retry_after_ms, the ticket is untouched, and this counter is incremented. A non-zero rate means approvers are being turned away by a store fault, not by a missing enrolment (an unenrolled approver gets 403 mfa_not_enrolled and is not counted here). This series counts the approver rule’s own refusals only (an unreadable stored proof, or an MFA-store fault at the approver’s own read: the step-up window at 0 or less, or a fault between the central check’s read and that one). With the central step-up check active (window above 0), a local approver whose MFA state is unreadable is refused by that check first: 503 without retry_after_ms, counted route="mfa_stepup" and yuzu_auth_mfa_step_up_gate_total{outcome="unavailable"}, so route="approval" stays flat in an ordinary store outage. The clock-skew refusal of an OIDC approver (the stored MFA proof is dated after the database clock; 503 with retry_after_ms null) is not counted here, so this series stays a store-fault signal; find it by the audit detail=approver_assurance:mfa_proof_clock_skew (result=error) or by the server’s error log line (see docs/ops-runbooks/auth-db-recovery.md, “OIDC approvers refused with a clock-skew 503”).

Session revocation metric

SCIM deprovision-linkage metrics (ADR-2001, CC6.8)

Two detective counters for the SCIM↔OIDC identity-link revoke seam — a SCIM deprovision revokes API/MCP tokens across a slug’s linked federated (SSO) identities as well as the slug itself. Both are always-on (no feature flag) and both pair with a dedicated audit action, scim.user.deprovision_role_refused_with_link — see docs/auth-architecture.md “SCIM ↔ OIDC identity linkage for deprovision” and docs/user-manual/scim-provisioning.md “SCIM ↔ OIDC identity linkage” for the full operator runbook. Six further ADR-2001 counters — the deny-at-login backstops firing at OIDC/SAML login rather than at SCIM deprovision time — live in “SSO login metrics” above, next to their siblings yuzu_auth_oidc_login_total/yuzu_auth_saml_login_total: yuzu_auth_oidc_deprovisioned_denied_total (§4/PR3) and yuzu_auth_saml_deprovisioned_denied_total (§4/PR4b) are each the TOTAL of a genuine-deny and a store-unavailable sub-counter (yuzu_auth_{oidc,saml}_deprovisioned_denied_genuine_total / yuzu_auth_{oidc,saml}_deprovisioned_denied_store_unavailable_total, #3069) — alert on the genuine sub-counter, since a Postgres outage alone can bump the total. Four more (#3072, SAML D2 observability) are below, alongside their OIDC D2 sibling yuzu_scim_deprovision_unlinked_total — yuzu_scim_saml_link_unmatched_total, yuzu_scim_saml_link_ambiguous_total, yuzu_scim_saml_link_lookup_failures_total (all three login-time, POST /saml/acs), and yuzu_scim_deprovision_saml_unlinked_total (deprovision-time). See docs/auth-architecture.md “SAML D2 observability (#3072)” for the honest split between what the login-time signals and the deprovision-time tripwire can each attribute.

Durable session-store metrics (HA WS-1/1a, ADR-2002 §4)

Operator sessions are durable Postgres rows (SessionStore), authored and adjudicated against the DB clock (Postgres now(), #3715). All four families are pre-seeded to 0 at boot so increase() alerting is meaningful.

DEX live-read metrics

The synchronous live-read endpoint (POST /api/v1/dex/devices/{id}/live) is bounded by a server-wide concurrency cap; these metrics surface its saturation and outcomes.

Engine-principal revalidation-cache metrics

Every held-open SSE stream re-validates its credential on each ~3 s pump tick. For a stream authenticated by an engine principal that check consults EnginePrincipalStore, and these metrics show whether it is being served from the short-TTL liveness cache (#2367) or reaching Postgres. The cache exists to keep that per-tick load off the connection pool, where it was previously self-amplifying under a pool brownout.

This cache covers only the ENGINE half of stream re-validation; the API-token half has its own (yuzu_server_token_cache_*).

MCP transport metrics

Request-body rejections at the /mcp/ ingress (#2437). The label set is CLOSED and pre-seeded at boot, so absent() alerting stays meaningful.

Pre-auth body-cap metrics (#2407)

Request-body rejections at the general pre-routing chokepoint, every route and every method — including GET/HEAD (D2 hardening, #2407: the earlier GET/HEAD exclusion was removed because it was factually wrong — a GET declaring a Content-Length still buffers a body regardless of method) — not just /mcp/, which keeps emitting the metric above too, for compatibility. Since the body-cap-post-read stage (#2407), the SAME metric also carries rejections from a second, later chokepoint — see reason=over_cap_post_read below. Seeded per table entry at boot from body_cap_policy.hpp’s kBodyCapTable, so the label set is CLOSED and absent() alerting stays meaningful.

Cookie-session write gate metrics

A cookie-session request refused by the server-wide same-site check (a write, or a guarded read: a request that dispatches work, persists state or forces a refresh of server state; see CSRF protection). Label sets are CLOSED and pre-seeded at boot, so absent() alerting stays meaningful.

MCP input-bounds metrics

Argument rejections on the MCP surface (#2405, #2437). Label sets are CLOSED and pre-seeded at boot, so absent() alerting stays meaningful.

Audit-store metrics

The audit store is the SOC 2 evidence chain, so both its write path and its retention path are scraped. The retention clock guard these describe is documented in docs/user-manual/audit-log.md.

yuzu_server_audit_backfill_total{result} (the one-time legacy-audit.db → PostgreSQL backfill outcome) was retired along with AuditStore::migrate_from_sqlite() itself (chore/retire-migrate-from-sqlite-auditstore, #3623, ADR-0009 hard-cutover Update) — the last of 19 stores to lose the metric; no production Yuzu fleet has ever run a pre-Postgres build, and the YuzuAuditBackfillFailing alert retired alongside it. A leftover audit.db still gets a boot-time WARN if it holds rows (legacy_sqlite_probe::warn_if_legacy_rows), logged, no metric.

Alert on absence, not just on rising counters. Most of the retention alert rules fire on something going wrong; ..._retention_passes_total is what catches the reaper not running at all, which is the state in which none of the other counter-driven rules can fire. The YuzuAuditRetentionNotRunning rule covers it. Its companion YuzuAuditRetentionMetricMissing covers the one case it structurally cannot: if this series is absent entirely - a server predating the metric, or a scrape config dropping it - increase() returns an empty vector, so YuzuAuditRetentionNotRunning selects nothing and can never fire. That rule keys on absent(...) instead, and is fleet-wide by construction (it cannot see one server among many going quiet). The current rule set is docs/prometheus/yuzu-alerts.yml.

The skips and failed counters must be alerted on separately and never collapsed: both leave rows undeleted, so an audit table that never shrinks looks identical either way. Only the pair distinguishes “the guard is protecting the table” from “cleanup is broken”.

Response-store metrics

The response store (PostgreSQL schema response_store, ADR-0039) persists agentic command/instruction results for the executions drawer and the /echo dashboard. Its ingest is fail-soft (a dropped result is re-derivable operational telemetry — the executions ladder still tracks the command), its reads are degrade-distinguishable (nullopt on a store/pool failure, never a false-empty), and its TTL retention runs the same clock-guarded sweep the audit store uses.

All reason/result/surface/cause dimensions are seeded to zero at boot, so absent-series alerting stays distinguishable from a scrape failure.

Echo dashboard metrics

The Echo metrics were named yuzu_tar_* (and yuzu_fleet_tar_db_corruption_agents) before 2026-10. There is no alias: dashboards, alert rules and recording rules that use the old series names must be updated to the yuzu_echo_* names (for example the series below).

MCP progress-bridge metrics

The MCP Streamable-HTTP progress bridge projects live notifications/progress onto a session’s GET stream when execute_instruction is called with a _meta.progressToken (see docs/user-manual/mcp.md). It is in-memory and bounded (256 correlation records).

All reason-label sets are closed (every value is a static literal seeded to 0 at boot), so absent()/rate() alerting is meaningful on a healthy server.

Pre-flight runner metrics

Webhook / offload delivery metrics (#3261, extended ADR-0057)

WebhookStore and OffloadTargetStore (see REST API §Webhooks / §Offload Targets) dispatch deliveries through a bounded worker pool; these counters cover delivery outcomes and pool backpressure. Every no-label counter below is pre-seeded to 0 at boot, so absent()-based alerting stays meaningful on a fresh server before the first delivery ever fires. Neither store has a backfill metric today: OffloadTargetStore never had one (fresh-starts unconditionally per ADR-0009’s amendment, no migrate_from_sqlite() to report an outcome for); yuzu_server_webhook_backfill_total was retired along with WebhookStore::migrate_from_sqlite() itself (chore/retire-migrate-from-sqlite-batch-a, #3623) — see ADR-0057’s Update.

Two more, added with the ADR-0059 Postgres migration (also pre-seeded to 0 at boot; this store has no backfill metric — ADR-0009’s fresh-start-by-default amendment means there is no legacy migration to report an outcome for):

Patch-manager write-outcome metrics (ADR-0062)

PatchManager (see REST API §Patch Management) records one outcome per write call. Pre-seeded to 0 at boot for every op×result combination, so absent()-based alerting stays meaningful before the first call ever fires.

The fleet-visualization REST surface (PR 3 of feat/viz-engine ladder; see REST API §Fleet Visualization) exposes the following metrics. Routes share one FleetTopologyStore cache; all metrics are process-global.

Gateway-upstream peer authorization metrics

The gateway-upstream service (--gateway-upstream) requires an authorized gateway peer before it processes any request: the caller’s leaf public key (SPKI SHA-256) must match a pin that was fixed at server boot and the leaf must carry serverAuth, or the operator must acknowledge disabled authorization with --insecure-gateway-peer. See server-admin.md for the flags.

A denial writes a session.gateway_peer_denied audit row only when its certificate parsed far enough for its key to be computed, under a bounded per-key budget; denials with no resolvable principal (null_context, not_authenticated, no_cert, bad_cert, and an internal_error with no computable key) are metric and log only (see audit-log.md). The row’s principal is gateway-peer:<spki8> and its principal_role is unverified_peer, because a refused peer is by definition not an authorized gateway; its detail is reason=<reason> rpc=<rpc> spki=<16 hex>. The log line for a denial with no key is gateway peer denied: rpc=<rpc> reason=<reason> peer=<ip> (a rate-limited warning; peer is the transport peer address, or unknown when there is none), and for a keyed denial gateway peer denied: rpc=<rpc> reason=<reason> spki=<16 hex>. The counter, not the audit table, is the signal.

Audit-write failure posture. A refusal is decided before its audit row is written, so a failed or suppressed audit write never changes the refusal: it is set-and-proceed. When a write FAILS the server logs gateway peer guard: audit write failed for a denied call (rpc=<rpc>); a row the budget suppresses logs nothing of its own and only increments the suppression counter and appears in the per-window summary warning. Either way the call is still refused with the same UNAUTHENTICATED status; the guard deliberately does not set the x-yuzu-audit-failed trailer, because the deny is already final and the denial counter is complete whether or not the row lands. A row the per-key budget declined is counted in yuzu_server_gateway_peer_denial_audit_suppressed_total; a row whose write failed has no counter of its own, only the warning line, so a persistently failing audit store shows up as missing rows plus those warnings, not as a metric.

The shipped alerts are in groups yuzu-security and yuzu-gateway of docs/prometheus/yuzu-alerts.yml. In yuzu-security (scraped from the server): YuzuGatewayPeerDenied (warning; the five certificate reasons bad_cert, no_server_auth_eku, not_pinned, outside_validity and internal_error; the three anonymous reasons deliberately do not fire it, because they are what a probe or a misdirected client produces on a listener that does not require a client certificate); YuzuGatewayPeerDeniedBatchHeartbeat (critical; the same five reasons restricted to rpc="batch_heartbeat" for 5 minutes, which means the gateway itself is refused: its registrations and stream-status notices fail while established agent streams to the gateway stay up; a gateway sends BatchHeartbeat only while at least one agent heartbeats through it, because an empty-buffer flush is a no-op; active together with YuzuGatewayPeerDenied, and an Alertmanager inhibit rule with source YuzuGatewayPeerDeniedBatchHeartbeat, target YuzuGatewayPeerDenied and equal: [instance, reason] pages only on the critical one; the gateway-side alert has the gateway’s instance and cannot be inhibited that way); and YuzuGatewayPeerAuthzDisabledWithTls (warning; mode="insecure_ack_tls" held for 5 minutes). In yuzu-gateway (scrape the GATEWAY, job yuzu-gateway): YuzuGatewayUpstreamAuthRefused (warning; increase(yuzu_gw_upstream_rpc_errors_total{code="16"}[10m]) > 0 for 2 minutes, the gateway-side view of the same refusal) and YuzuGatewayUpstreamTlsHandshakeFailures (warning; increase(yuzu_gw_upstream_tls_handshake_failures_total[10m]) > 0 for 5 minutes, covering a handshake that fails below the guard and so never reaches yuzu_server_gateway_peer_denied_total; it is a warning although the same cause can be a total outage of that gateway). The two upstream-refusal gateway counters are not pre-seeded to 0, so a label set that first appears at 1 and never moves is not seen by increase(); a persistent cause keeps incrementing and fires. After the cause is fixed, each server-side alert stays active for about 6 minutes after the last event (the 5-minute increase window plus an evaluation step), and each of the two gateway-side alerts for about 11 minutes (their 10-minute window covers the circuit breaker’s maximum open period, 30 s by default since #5695 and 300 s before, during which the counter moves only at the breaker’s probes): check that the counter is flat rather than waiting for the alert to clear. YuzuGatewayUpstreamSlow (warning; yuzu_gw_upstream_slow == 1 for 5 minutes) covers a server that answers every call slowly, which the circuit breaker cannot see because no call fails: the gauge is the gateway’s trailing-60 s backpressure duty cycle at 0.5 or more (see the gateway manual’s “Slow server”). Two more yuzu-gateway rules cover the listener’s client-address attribution (yuzu_gw_grpc_origin_ignored_total, pre-seeded to 0 for every mode and reason): YuzuGatewayOriginUnusable (warning; increase(...{reason=~"no_claim|bad_claim"}[10m]) > 0 for 10 minutes, meaning a trusted proxy keeps giving the gateway no usable client address, so those registrations are audited with origin_observed=false) and YuzuGatewayOriginSourcesMismatch (warning; reason="untrusted_source" for 30 minutes, meaning the proxy connects from an address the listener’s sources does not list, or something other than the proxy reaches the port). The origin is resolved per RPC stream: an L4, TCP, TLS or PROXY LOCAL health check opens no stream and is never counted, and only an HTTP/2 health check that makes a call would be. Both thresholds are initial values not yet tuned on a real deployment. No shipped alert covers the gateway counter yuzu_gw_upstream_calls_dropped_total{rpc_name} (proxied register or inventory calls not sent because the caller’s deadline passed while it waited for a permit, pre-seeded to 0); the ratio rate(yuzu_gw_upstream_calls_dropped_total{rpc_name="register"}[5m]) / (rate(yuzu_gw_upstream_calls_dropped_total{rpc_name="register"}[5m]) + rate(yuzu_gw_upstream_rpc_duration_ms_count{rpc_name="register"}[5m])) is the share of registrations given up while waiting for a permit, relative to those forwarded successfully (upstream errors are not in the denominator). YuzuGatewayUpstreamCircuitOpened (warning) fires when the breaker opened and has not closed for 3 minutes, or closed three or more times in 15 minutes (a flapping server); a server that is back within 3 minutes does not fire it. insecure_ack is deliberately not alerted: a plaintext development rig runs it by design, and no shipped alert covers an insecure_ack mode on a deployment that should be enforcing; a production operator can add the rule yuzu_server_gateway_peer_authz_mode{mode="insecure_ack"} == 1 on their own side.

Subscribe peer-binding security counters

The per-session peer-IP binding for the agent Subscribe RPC (#826/#1058/#1059, NAT-aware relaxation #1128) emits two paired counters. Both carry the event="security" SIEM-routing label and should be read together — a spike in one alone vs both together carries very different meaning.

Read-together interpretation. Both counters share the event and gateway_mode labels so an analyst can join them by operator-mode dimension. A spike in _peer_advisory_total alone is expected multi-egress churn in NAT-relaxed deployments and is not actionable. A spike in BOTH simultaneously is the actionable signal: it can indicate a stolen-session replay landing inside a trusted range, where the legitimate agent triggers the reject and the attacker (also in-range) is admitted via advisory. The AgentSubscribePeerAdvisoryCorrelatedSpike alert below encodes exactly that pattern.

Recommended alerts

- alert: VizFleetDispatchTimeoutsRising
  expr: rate(yuzu_viz_agent_dispatch_timeout_total[5m]) > 5
  for: 5m
  annotations:
    summary: "Fleet topology fetcher is timing out per-agent (partial fleet outage)"

- alert: VizFleetSlowRequests
  expr: histogram_quantile(0.99, sum(rate(yuzu_viz_topology_request_seconds_bucket[5m])) by (le)) > 5.5
  for: 10m

- alert: VizFleetSlowAgentDispatch
  expr: histogram_quantile(0.99, sum(rate(yuzu_viz_topology_fetch_duration_seconds_bucket[5m])) by (le)) > 5.0
  for: 10m
  annotations:
    summary: "Fleet topology agent-dispatch p99 above the 5 s fetcher deadline; agents may be unresponsive"

- alert: VizFleetRefillOversizeDrops
  expr: increase(yuzu_viz_refill_oversize_drops_total[10m]) > 0
  annotations:
    summary: "FleetTopologyStore is dropping refills above 256 MiB cap; raise --max-snapshot-bytes or scope down the fleet"

- alert: VizFleetPushRejections
  expr: increase(yuzu_viz_topology_push_rejected_total[10m]) > 0
  annotations:
    summary: "Fleet-snapshot pushes rejected by the IP-spoof guard — spoofing campaign or NAT/DHCP misconfiguration"

- alert: VizFleetCapEvictions
  expr: increase(yuzu_viz_pushed_cap_evictions_total[10m]) > 0
  annotations:
    summary: "FleetTopologyStore is evicting agents at the 100000-entry hard cap — fleet outgrew the cap or a cap-flood attack is in progress"

- alert: YuzuMcpEngineRevalidateStoreUnreachable
  expr: increase(yuzu_server_engine_revalidate_backoff_suppressed_total[10m]) > 0
  labels:
    severity: warning
  annotations:
    summary: "Engine-principal liveness re-checks are being answered from the failure backoff - the principal store is unreachable and engine streams are riding their grace windows"
    description: "This counter only moves on CONFIRMED unreachability (the store closed, or a query actually ran and failed, or PgPool's own connect-failure breaker is open) - a bare pool-lease-acquire timeout under a healthy database does not arm this backoff, so firing already rules out ordinary pool saturation. First rule out real impact: a flat yuzu_mcp_stream_closes_total{reason=\"auth_unavailable\"} means the backoff is absorbing the blip and no streams have ended. Then correlate with yuzu_pg_acquire_wait_seconds and yuzu_pg_pool_in_use for genuine connection exhaustion vs. a connect-level failure. Runbook: docs/ops-runbooks/engine-principal-store-recovery.md."

- alert: VizFleetPushedMapNearCap
  expr: yuzu_viz_pushed_map_size > 80000
  for: 10m
  annotations:
    summary: "FleetTopologyStore pushed_ map above 80% of the 100000 hard cap; evictions imminent"

# Stolen-session signals (event="security" — routed to the SIEM via Prometheus;
# see observability-conventions.md). The audit rows session.peer_mismatch /
# session.identity_mismatch carry the forensic detail.
- alert: AgentSubscribePeerMismatch
  expr: increase(yuzu_grpc_subscribe_peer_mismatch_total{event="security"}[5m]) > 0
  annotations:
    summary: "Subscribe peer-IP mismatch (#1059) — possible stolen session_id replayed from a new IP"

- alert: AgentSubscribeIdentityMismatch
  expr: increase(yuzu_grpc_subscribe_identity_mismatch_total{event="security"}[5m]) > 0
  annotations:
    summary: "Subscribe mTLS identity mismatch (#1118) — possible stolen session_id replayed with a non-matching client cert"

# NAT-aware tolerated mismatch (#1128). yuzu_grpc_subscribe_peer_advisory_total
# {event="security",reason="mtls_identity_match|trusted_nat_cidr"} counts peer-IP
# mismatches that were DOWNGRADED to advisory (not rejected) under a NAT
# accommodation. A spike here ALONE is expected churn in multi-egress NAT
# deployments — do NOT alert on it bare. The actionable signal is the
# correlated form: advisory AND rejected mismatches rising together can mean a
# stolen-session replay landing inside a trusted range.
- alert: AgentSubscribePeerAdvisoryCorrelatedSpike
  expr: >
    increase(yuzu_grpc_subscribe_peer_advisory_total{event="security"}[5m]) > 0
    and increase(yuzu_grpc_subscribe_peer_mismatch_total{event="security"}[5m]) > 0
  annotations:
    summary: "Tolerated NAT peer-mismatches (#1128) coincide with rejected mismatches — investigate the reject events for a possible stolen-session replay inside a trusted range"

# Operator-visibility guard for --nat-trust-mtls-identity. Sustained
# mtls_identity_match advisories mean the (opt-in, off-by-default) mTLS-identity
# NAT accommodation is active — confirm it was enabled intentionally AND that
# client certs are PER-AGENT (a shared fleet cert makes this a replay bypass,
# gov UP-2). Long window: this should be rare; a steady stream is worth a look.
- alert: AgentSubscribeMtlsIdentityAdvisoryActive
  expr: increase(yuzu_grpc_subscribe_peer_advisory_total{event="security",reason="mtls_identity_match"}[30m]) > 0
  for: 30m
  annotations:
    summary: "--nat-trust-mtls-identity is relaxing peer-IP binding via mTLS-identity match — verify it was enabled deliberately and that client certs are per-agent (shared cert = session-replay bypass)"

- alert: AgentRegisterDeniedFlood
  expr: rate(yuzu_register_denied_total{event="security"}[5m]) > 1
  for: 5m
  annotations:
    summary: "Admin-denied identity repeatedly attempting Register (#1067) — credential-abuse / denied-flood signal"

Agent registration input-shape rejections

Labels

Metrics carry a standard set of labels for filtering and grouping in queries.

On-behalf-of rejection metric (ADR-1005)

# HELP yuzu_onbehalf_rejected_total Requests rejected for carrying a reserved on-behalf-of header/metadata key (ADR-1005) by surface
# TYPE yuzu_onbehalf_rejected_total counter
yuzu_onbehalf_rejected_total{surface="http",event="security"} 0
yuzu_onbehalf_rejected_total{surface="grpc",event="security"} 0

Both series are pre-seeded to 0 at startup, so absent() alerts stay meaningful. Any non-zero value means a client asserted it was acting on another principal’s behalf via a reserved header (HTTP 403) or gRPC metadata key (call cancelled) — see docs/auth-architecture.md (“On-behalf-of assertions rejected”) for the reserved-name list. This event deliberately has no audit row: the rejection fires pre-authentication, so there is no resolved principal to attribute — the metric (with event="security", SIEM-routable per the observability conventions) is the signal. Alert on increase(yuzu_onbehalf_rejected_total[1h]) > 0 if you want notification of any attempt.

Per-principal quota metric (PR 4.4, ADR-1005 class engine principals)

# HELP yuzu_server_principal_quota_exhausted_total Per-principal quota-cap exhaustions by side and limit dimension
# TYPE yuzu_server_principal_quota_exhausted_total counter
yuzu_server_principal_quota_exhausted_total{side="engine",limit="concurrency"} 0
yuzu_server_principal_quota_exhausted_total{side="engine",limit="rate"} 0
yuzu_server_principal_quota_exhausted_total{side="operator",limit="concurrency"} 0
yuzu_server_principal_quota_exhausted_total{side="operator",limit="rate"} 0

# HELP yuzu_server_principal_quota_admits_total Per-principal quota-cap admits (successful try_acquire) by side
# TYPE yuzu_server_principal_quota_admits_total counter
yuzu_server_principal_quota_admits_total{side="engine"} 0
yuzu_server_principal_quota_admits_total{side="operator"} 0

All 4 exhausted_total series (the closed side × limit cross-product) are pre-seeded to 0 at startup, so absent() alerts stay meaningful. The gate applies only to engine-classed sessions (stored principal_kind engine or an engine_token authentication source; username engine:<slug>) at the server’s single pre-routing chokepoint — human/agent/anonymous traffic never touches it. side distinguishes which principal’s budget was debited (engine is the only value emitted in PR 4.4; operator is pre-seeded but dormant — it activates in Phase 5 when delegation debits the delegating operator’s side too). limit distinguishes which of the two independent caps rejected the request: concurrency (in-flight requests, tunable via --principal-max-concurrency/YUZU_PRINCIPAL_MAX_CONCURRENCY, default 16) or rate (token-bucket requests/second, tunable via --principal-rate-limit/YUZU_PRINCIPAL_RATE_LIMIT, default 20.0/s, burst = 2x rate). Neither label carries principal_id — bounded cardinality only. This is an operational counter, not event="security" — a quota cap hit is expected steady-state behaviour for a busy engine principal, not an attack signal — and it deliberately has no audit row for the same high-frequency-operational reason (see docs/user-manual/audit-log.md).

yuzu_server_principal_quota_admits_total{side} is the admits companion — every request that clears both the concurrency and rate checks increments it once. It exists so the exhaustion rate is computable rather than just the raw exhaustion count: exhausted / (exhausted + admits). A raw exhaustion counter alone can’t distinguish “an engine principal is hitting its cap on a small fraction of a huge request volume” (healthy, cap doing its job) from “almost everything this engine principal sends is being rejected” (self-brick, usually right after a config change that set the cap too low) — the ratio against admits is what tells them apart. Both series are pre-seeded to 0 at startup for the same reason; side="engine" is the only value emitted today (side="operator" is pre-seeded but dormant until Phase 5, same as the exhaustion counter). Streaming requests are included — a streaming request now takes a concurrency slot (held for the stream’s lifetime) plus the rate debit, same as any other engine request (UP-1 hardening fix; see the streaming note below) — so a streaming request is counted here exactly like any other admitted engine request.

Streaming/SSE note (UP-1). Streaming/SSE requests are not exempt from either dimension of this cap — they take a concurrency slot held for the stream’s lifetime (released when the stream ends), plus the rate debit, the same two caps as any other engine request. An earlier revision of this primitive treated streaming as rate-only and concurrency-exempt, which left an engine principal able to open an unbounded number of concurrent streams; that gap is closed.

Per-instance caveat: the cap is enforced in one process’s memory. A multi-replica deployment’s effective ceiling is configured_cap x replica_count, not the configured cap alone — each replica enforces the cap independently against its own share of traffic, so N replicas give each engine principal up to N x the configured per-replica cap in aggregate, not one fleet-wide budget. Durable, cross-instance quota is a Phase-8 follow-up. When alerting or sizing a cap for a multi-replica deployment, account for this multiplier — a cap sized for the single-instance default may be far too generous fleet-wide. Alert on increase(yuzu_server_principal_quota_exhausted_total{side="engine"}[1h]) > 0 if you want notification that an engine principal is regularly hitting its cap (may indicate the cap is tuned too low for its workload, or a runaway caller).

Engine-credential confirm metric (#2404)

# HELP yuzu_engine_principal_confirm_total Engine-credential rotation confirm outcomes by surface (rest|mcp) and result (success|conflict|client_error|transient); store-reaching calls only, pre-store denials excluded (#2404)
# TYPE yuzu_engine_principal_confirm_total counter
yuzu_engine_principal_confirm_total{surface="rest",result="success"} 0
yuzu_engine_principal_confirm_total{surface="rest",result="conflict"} 0
yuzu_engine_principal_confirm_total{surface="rest",result="client_error"} 0
yuzu_engine_principal_confirm_total{surface="rest",result="transient"} 0
yuzu_engine_principal_confirm_total{surface="mcp",result="success"} 0
yuzu_engine_principal_confirm_total{surface="mcp",result="conflict"} 0
yuzu_engine_principal_confirm_total{surface="mcp",result="client_error"} 0
yuzu_engine_principal_confirm_total{surface="mcp",result="transient"} 0

All 8 series (the closed surface × result cross-product) are pre-seeded to 0 at startup so absent() alerts stay meaningful. The family makes a credential-confirm retry storm alertable, which yuzu_http_requests_total cannot show (it has no per-route label). result mirrors the shared store-error taxonomy (engine_store_error_class.hpp): conflict is a terminal 409/kInvalidParams (a replay after the rotation already resolved, a moved-on pin, or unresolved rotation metadata — the client must not blindly retry), client_error is a 400 (more-than-two or non-engine active credentials), transient is a retryable 503 (store unavailable, lock contention, a persist failure, or the ambiguous empty/malformed-pair “no in-flight rotation”), and success is a completed confirm.

Scope contract. Counted only when a confirm reaches ApiTokenStore::confirm_rotation or trips the store-open guard (which increments result="transient"); pre-store denials — permission, MFA step-up, input validation, and the MCP supervised approval-gate’s consumed-ticket replay (approval already used) — are deliberately excluded, so the label set stays a fact about store outcomes rather than a catch-all endpoint tally. The increment is stamped before the audit emission, so an audit-store failure cannot suppress it. Wired on both surfaces (the surface label prevents double-count) because either a REST or an agentic MCP client can drive the #2404 replay shape.

Operational, not event="security". A replay conflict is an expected agent-retry pattern, not an attack signal; the counter carries no principal_id label (bounded cardinality only) — pair it with the engine_principal.credential.confirm audit rows for per-principal forensics (metric-is-the-signal / audit-row-is-the-evidence). A useful alert is increase(yuzu_engine_principal_confirm_total{result="conflict"}[15m]) > <n> to catch a client stuck replaying a resolved confirm, and increase(...{result="transient"}[15m]) for a genuine store-health storm.

Semantics for alert authors: this counts confirm attempts that reach the store, not logical rotations — a client retrying a transient blip 20 times before it succeeds adds 20 to transient for one rotation. Pick the threshold <n> relative to your fleet’s typical retry count, not your rotation cadence. Also cross-reference yuzu_engine_principal_rotation_sweep_failures_total: a result="conflict" cluster correlating with a non-zero, sustained sweep-failure count is the prolonged-sweep-outage signature (a predecessor’s overlap window never closes, so a confirm surfaces the unresolved rotation metadata state) — inspect the credential before revoking, since in that state the sole active credential is the good survivor.

yuzu_rotation_sweep_capped_total (governance UP-5/UP-6, no labels, shared across engine-credential and human API-token rotation pairs — deliberately kind-neutral naming, unlike yuzu_engine_principal_rotation_sweep_failures_total above, because a capped tick is a tick-level property, not attributable to one kind’s rows; the sibling failures counter keeps its engine-scoped name only because renaming an already-shipped series would break existing alerts): a rotation sweep tick found MORE eligible predecessors than its per-tick auto-revoke cap (kMaxAutoRevokesPerTick, api_token_store.cpp) and processed only the cap’s worth, deferring the rest to later ticks — the clock-guarded-retention routed concern’s mandatory unconditional cap, applied here (a single forward NTP step must degrade to a bounded multi-tick drain, never a fleet-wide cutover in one tick). A sustained non-zero rate means the fleet has more concurrent in-flight rotations than the cap comfortably drains — raise the cap, or find out why so many rotations are in flight at once, rather than ignore it.

Rotation-sweep clock guard metrics (#2964)

The T12 sweep now carries the full seven-part clock-guarded-retention shape (the routed concern in CLAUDE.md), not just the cap above. These are the remaining series it registers, all kind-neutral and shared across engine-credential and human API-token rotation pairs for the same reason yuzu_rotation_sweep_capped_total is — the clock guard’s verdict is a tick-level property of the sweep itself, not attributable to one kind’s rows. Background and the decision rule: audit_retention_rules.hpp::classify (reused, not forked) and ApiTokenStore::sweep_expired_rotations (api_token_store.cpp), pinned by tests/unit/server/test_api_token_store.cpp.

A server predating this metrics family, or a scrape config dropping the yuzu_rotation_sweep_* series, reads as a silently-healthy sweep — none of these are paired with their own “series is absent” companion alert today (unlike YuzuAuditRetentionNotRunning/YuzuAuditRetentionMetricMissing); that gap is tracked, not closed, by this round.

Human API-token confirm metric (P2 #11)

# HELP yuzu_api_token_confirm_total Human API-token rotation confirm outcomes by surface (rest|mcp) and result (success|conflict|client_error|transient); store-reaching calls only, pre-store denials excluded - the human-owned twin of yuzu_engine_principal_confirm_total
# TYPE yuzu_api_token_confirm_total counter
yuzu_api_token_confirm_total{surface="rest",result="success"} 0
yuzu_api_token_confirm_total{surface="rest",result="conflict"} 0
yuzu_api_token_confirm_total{surface="rest",result="client_error"} 0
yuzu_api_token_confirm_total{surface="rest",result="transient"} 0
yuzu_api_token_confirm_total{surface="mcp",result="success"} 0
yuzu_api_token_confirm_total{surface="mcp",result="conflict"} 0
yuzu_api_token_confirm_total{surface="mcp",result="client_error"} 0
yuzu_api_token_confirm_total{surface="mcp",result="transient"} 0

The human-owned twin of yuzu_engine_principal_confirm_total directly above — same closed surface x result cross-product (8 series), pre-seeded to 0 at startup for the same absent()-alert reason, and the same intended scope contract: counted only when a confirm reaches the credential store or trips the store-open guard (result="transient"), excluding pre-store denials (permission, input validation, an MCP approval-gate replay) so the label set stays a fact about store outcomes. result mirrors the same store-error taxonomy as the engine family.

Live — both transports increment this family. It is registered and pre-seeded (rotation_sweep_naming.hpp’s kApiTokenConfirmTotalMetric symbol, shared by the registration site and both increment call sites so a typo can’t silently create a second, uncounted shadow series) at rest_api_v1.cpp’s POST /api/v1/tokens/{id}/confirm handler (surface="rest") and mcp_server.cpp’s confirm_api_token_rotation tool handler (surface="mcp"). Treat it exactly like the engine family: increase(yuzu_api_token_confirm_total{result="conflict"}[15m]) > <n> for a client stuck replaying a resolved confirm, paired with whatever audit action the confirm-rotation piece emits for per-principal forensics — the metric is the signal, the audit row is the evidence. See audit-log.md for the api_token.rotation.auto_revoke / .successor_unused rows this piece DOES ship (the sweep-driven auto-revoke and successor-unused-warning half of human token rotation, distinct from confirm).

MCP poll-rate metric (#3344)

# HELP yuzu_mcp_poll_total MCP result-poll tool calls by verdict (get_execution_status, query_responses, get_bundle_result). not_ready: the success payload carried a retry_after_ms poll hint; ready: served terminal/complete without one. Excludes pre-verdict denials.
# TYPE yuzu_mcp_poll_total counter
yuzu_mcp_poll_total{tool="get_execution_status",result="ready"} 0
yuzu_mcp_poll_total{tool="get_execution_status",result="not_ready"} 0
yuzu_mcp_poll_total{tool="query_responses",result="ready"} 0
yuzu_mcp_poll_total{tool="query_responses",result="not_ready"} 0
yuzu_mcp_poll_total{tool="get_bundle_result",result="ready"} 0
yuzu_mcp_poll_total{tool="get_bundle_result",result="not_ready"} 0

Closed tool (the three success-shaped result-poll tools) x result (ready|not_ready) cross-product (6 series), pre-seeded to 0 at startup (server.cpp), symbol shared between the describe() site and the increment site via mcp::kMcpPollTotalMetric (mcp_retry.hpp) so the two cannot silently diverge into a shadow series — same precedent as kApiTokenConfirmTotalMetric above. Scope contract: counted only when a call reaches a served verdict — pre-verdict denials (tier, permission, invalid-params, not-found) are excluded, already visible via the existing denial counters and A4 envelopes, so the label set stays a fact about poll outcomes. For query_responses specifically, a call whose in-flight-ness could not be determined (an instruction_id-only query, or an execution_id the tracker can’t resolve) increments neither series — it was never checked, so folding it into ready would understate the not_ready fraction. Deliberately operational, not event="security" — a high poll rate is expected agentic-worker behaviour, not an anomaly.

Purpose: data-driven re-tuning, not alerting. The named retry_after_ms floor constants (kMcpStoreFaultRetryMs, kMcpResultPollRetryMs, etc. — mcp_retry.hpp) were shipped with a mechanical derivation (no dispatch-to-first-result latency histogram exists to measure from — yuzu_command_duration_seconds is full command completion, not first-result) rather than a measured one. This series’ not_ready fraction is the data that lets a future change re-derive them from real evidence instead of guessing again — same “tuning signal, not an alert” posture as yuzu_nvd_sync_failures_total above; no alert rule is expected or wired. Example query for “is the floor for this tool too conservative or too aggressive”:

sum(rate(yuzu_mcp_poll_total{result="not_ready"}[15m])) by (tool)
/
sum(rate(yuzu_mcp_poll_total[15m])) by (tool)

See docs/mcp-server.md → retry_after_ms floors for the full constant table this metric exists to tune.

Rotation-durability tamper/corruption gauges (#2961)

# HELP yuzu_server_rotation_pair_resolve_failures_total resolve_rotation_pair_after_revoke partner-clear failures that were swallowed - the revoke they follow has already committed, so they cannot fail the caller. Leaves stale rotation metadata on the surviving partner; NOT a lockout risk (the sweep cannot auto-revoke a stranded partner). Non-zero means inspect manually.
# TYPE yuzu_server_rotation_pair_resolve_failures_total gauge
yuzu_server_rotation_pair_resolve_failures_total 0
# HELP yuzu_server_rotation_initiator_disagreements_total resolve_rotation_initiator RAM-vs-durable disagreements - not reachable in normal operation; non-zero means inspect api_tokens.rotation_initiator for out-of-band tampering or corruption.
# TYPE yuzu_server_rotation_initiator_disagreements_total gauge
yuzu_server_rotation_initiator_disagreements_total 0

Both are ApiTokenStore counters, gauge-published from a store accessor on every /metrics scrape (same shape as yuzu_server_token_cache_size above — no counter pre-seed needed, since the scrape callback sets the value every time rather than relying on a startup registration).

yuzu_server_rotation_initiator_disagreements_total is the tamper/ corruption signal on an authorization input: resolve_rotation_initiator (api_token_store.cpp) refuses to resolve a confirm’s initiator when the RAM grace-cache copy of requesting_user and the durable api_tokens.rotation_initiator column DISAGREE, rather than silently preferring either. Both sources are written from the SAME requesting_user inside the SAME locked mint transaction, so this is not reachable through any live code path — a non-zero value means an out-of-band write to api_tokens.rotation_initiator (direct SQL, a restored/edited backup) or a future bug, never normal operation. The corresponding confirm call fails closed with rotation confirmation unavailable (409/Conflict); this gauge is the only durable, alertable trace of why it failed closed (the error string alone does not distinguish a genuine “never stamped” pre-v3 pair from a disagreement). Alert on any nonzero value — yuzu_server_rotation_initiator_disagreements_total > 0 — rather than a rate threshold: even one occurrence warrants manual inspection of the row’s history.

yuzu_server_rotation_pair_resolve_failures_total covers a distinct, operability-only failure: resolve_rotation_pair_after_revoke failing to clear the surviving partner’s rotation state after a revoke has already committed. See the metric’s own HELP text above for the full scope — stale metadata, not a lockout risk, since the sweep structurally cannot auto-revoke a row this leaves stranded.

Access review metrics

Periodic access reviews (SOC 2 CC6.2, docs/auth-architecture.md “Periodic access reviews”) shipped with zero metrics; the hardening round added four bounded-label series wired at the REST handlers (rest_api_v1.cpp) only — the MCP twins are not double-counted.

# HELP yuzu_access_review_export_total Access review evidence exports (GET /api/v1/access-reviews/export), by format
# TYPE yuzu_access_review_export_total counter
yuzu_access_review_export_total{format="json"} 0
yuzu_access_review_export_total{format="csv"} 0

# HELP yuzu_access_review_export_duration_seconds Latency of the cross-principal grant-population read (build_access_review) behind GET /api/v1/access-reviews/export
# TYPE yuzu_access_review_export_duration_seconds histogram

# HELP yuzu_access_review_campaigns_opened_total Access review campaigns opened (POST /api/v1/access-reviews)
# TYPE yuzu_access_review_campaigns_opened_total counter
yuzu_access_review_campaigns_opened_total 0

# HELP yuzu_access_review_attestations_total Access review reviewer decisions recorded (POST /api/v1/access-reviews/{id}/attestations), by decision
# TYPE yuzu_access_review_attestations_total counter
yuzu_access_review_attestations_total{decision="attested"} 0
yuzu_access_review_attestations_total{decision="flagged_revoke"} 0

• yuzu_access_review_export_total{format} — counter, format ∈ {json, csv}, pre-seeded to 0 at startup. Increments once per successful GET /api/v1/access-reviews/export call, split by which format was requested — a sustained shift toward csv typically means an auditor/spreadsheet workflow is in play.
• yuzu_access_review_export_duration_seconds — histogram (default bucket ladder, see “Histogram buckets” below), the wall-clock time of the cross-principal grant-population read (build_access_review) behind the export, timed regardless of success/failure outcome — a slow/failing read is exactly what an operator debugging a 503 needs latency evidence for.
• yuzu_access_review_campaigns_opened_total — counter, no labels. Increments once per successful POST /api/v1/access-reviews — the rate this fires at is the empirical “how often does the org actually run a review” cadence signal (compare against the org’s stated review policy).
• yuzu_access_review_attestations_total{decision} — counter, decision ∈ {attested, flagged_revoke}, pre-seeded to 0 at startup. Increments once per successful POST /api/v1/access-reviews/{id}/attestations call, split by the reviewer’s decision. flagged_revoke counts recorded evidence only — it never itself revokes anything (flag ≠ revoke, see docs/auth-architecture.md); a rising flagged_revoke count is a worklist size for whoever acts on the flags, not an automatic remediation trigger.

None of the four carries event="security" — a review export or an attestation decision is expected, privileged, self-audited operator activity (own audit rows: access_review.exported, .campaign_opened, .attested/.flagged), not an anomaly signal. Bounded-label series (format, decision) are pre-seeded to 0 at startup per the standing convention in docs/observability-conventions.md, so absent()-style alerts stay meaningful.

Agent OTA pull metrics (#913, #911, #416)

# HELP yuzu_ota_download_admission_total Agent OTA DownloadUpdate admission decisions by outcome
# TYPE yuzu_ota_download_admission_total counter
yuzu_ota_download_admission_total{decision="admitted"} 0
yuzu_ota_download_admission_total{decision="rejected_concurrency"} 0
yuzu_ota_download_admission_total{decision="rejected_rate"} 0
yuzu_ota_download_admission_total{decision="rejected_total"} 0

# HELP yuzu_ota_download_refund_total Agent OTA rate tokens returned after a server-attributable failure, by reason
# TYPE yuzu_ota_download_refund_total counter
yuzu_ota_download_refund_total{reason="registry_unavailable"} 0
yuzu_ota_download_refund_total{reason="version_not_found"} 0
yuzu_ota_download_refund_total{reason="package_missing"} 0
yuzu_ota_download_refund_total{reason="transfer_deadline"} 0
yuzu_ota_download_refund_total{reason="chunk_deadline"} 0

# HELP yuzu_ota_download_deadline_exceeded_total Agent OTA DownloadUpdate transfers aborted by a server-imposed deadline, by phase
# TYPE yuzu_ota_download_deadline_exceeded_total counter
yuzu_ota_download_deadline_exceeded_total{phase="transfer"} 0
yuzu_ota_download_deadline_exceeded_total{phase="write"} 0

# HELP yuzu_ota_admission_key_mode_total Agent OTA admission keying actually used, by mode
# TYPE yuzu_ota_admission_key_mode_total counter
yuzu_ota_admission_key_mode_total{mode="cert"} 0
yuzu_ota_admission_key_mode_total{mode="peer_ip"} 0
yuzu_ota_admission_key_mode_total{mode="unknown"} 0

# HELP yuzu_grpc_ota_identity_rejected_total Agent OTA RPCs rejected for a missing or mismatched client certificate identity, by rpc and reason
# TYPE yuzu_grpc_ota_identity_rejected_total counter
yuzu_grpc_ota_identity_rejected_total{event="security",rpc="download_update",reason="agent_id_mismatch"} 0
...

# HELP yuzu_ota_download_peers_tracked Peers currently tracked by the OTA admission map
# TYPE yuzu_ota_download_peers_tracked gauge
yuzu_ota_download_peers_tracked 0

# HELP yuzu_ota_download_peers_capacity Configured ceiling on the OTA admission map (--ota-max-peers-tracked)
# TYPE yuzu_ota_download_peers_capacity gauge
yuzu_ota_download_peers_capacity 50000

# HELP yuzu_ota_download_peers_evicted_total Peers evicted from the OTA admission map by its cardinality ceiling
# TYPE yuzu_ota_download_peers_evicted_total counter
yuzu_ota_download_peers_evicted_total 0

Every closed series is pre-seeded to 0 at startup so absent() alerts stay meaningful. Labels are bounded by construction — a peer identity or IP is never a label value.

Operational vs security. The admission, deadline and keying counters are operational: a concurrency or rate rejection is expected steady-state behaviour during a fleet-wide update, not an attack signal (the same reasoning that keeps the per-principal quota counters off event="security"). yuzu_grpc_ota_identity_rejected_total is the exception and carries event="security" (it counts all four rejection reasons; only three of them also write an audit row — no_client_identity is metric-only, see docs/user-manual/audit-log.md), because a failed certificate bind on an already-enrolled agent’s OTA pull is an authentication event; it mirrors yuzu_grpc_revoked_cert_total and is paired with a session.ota_identity_rejected audit row (the metric is the signal, the audit row is the evidence). This counter is complete; the audit row is not. The row is rate-limited to roughly two per second per (peer, RPC, reason), because that write is synchronous, Postgres-backed, and sits ahead of the per-peer admission bound. Use this counter to measure volume during a flood, and the rows for attribution.

yuzu_ota_identity_audit_suppressed_total{rpc,reason} counts the rows that were NOT written because that bucket was exhausted, so the sampling is directly observable rather than inferred from a gap between two other numbers. The bucket key includes the denial reason deliberately: without it, a peer holding a certificate from another CA in a multi-CA trust bundle could present CN=<victim agent id> and spend the victim’s allowance. Note that verb is deliberately distinct from session.identity_mismatch, which is the Subscribe binding check and carries a different detail shape — see docs/user-manual/audit-log.md.

Reading decision. rejected_concurrency means the peer already held --ota-max-concurrent-per-peer parallel downloads — the primary bound, and the one that should fire under an actual monopolisation attempt. rejected_total means the SERVER-WIDE ceiling (--ota-max-concurrent-total) was hit — the only bound that does not scale with a caller’s address space, so it is the one that fires under a distributed pull. rejected_rate means the peer drained its token bucket, which is a much looser secondary bound; sustained rejected_rate without rejected_concurrency usually indicates the rate knobs are tuned too tightly for the fleet’s retry behaviour rather than an attack. decision is a genuine partition over calls that reach the admission gate — exactly one per such call. It is deliberately not a partition of every received DownloadUpdate: the on-behalf, revoked-certificate and positive-identity gates return before admission and are counted by their own families, so this sum is legitimately lower than yuzu_grpc_requests_total{method="DownloadUpdate",status="received"}. A divergence between the two is expected during an identity attack, not missing accounting.

Reading refunds. yuzu_ota_download_refund_total{reason} is a SEPARATE family, deliberately not a decision="refunded" label: a refunded request has already been counted as admitted, so folding refunds into that family would stop decision partitioning and make sum(yuzu_ota_download_admission_total) double-count. A refund is issued only for a server-attributable failure — a deadline the server imposed, an unavailable registry, a missing artifact, or a version this server does not hold — and it is why a slow-link or flapping agent cannot spend itself into a lockout. The reason label is what makes the invariant checkable: in steady state the two deadline reasons together must match yuzu_ota_download_deadline_exceeded_total exactly, and a growing gap means a code path is aborting without refunding (this is what the invariant a refund-divergence alert watches — the rule for it ships with the OTA alert group, separately). version_not_found is refunded on purpose even though the peer chose the version: nothing is streamed, so no capacity is consumed, and a staged rollout routinely has agents asking for versions a given server does not hold.

Reading the peer map. yuzu_ota_download_peers_capacity publishes the configured --ota-max-peers-tracked so alerts can express occupancy as a RATIO rather than hardcoding a threshold that silently becomes wrong when an operator retunes the flag. yuzu_ota_download_peers_evicted_total increments at the moment the ceiling actually bites, so approaching the cap and hitting it are distinguishable.

Reading mode. cert is the healthy case: admission is keyed on the peer’s certificate identity. peer_ip means the peer presented no client certificate, so admission fell back to source IP — expected on a default-certificate deployment where unenrolled agents are still bootstrapping, but a sustained high peer_ip share on a fleet you believe is fully enrolled is worth investigating. unknown should never appear; it means neither a certificate nor a parseable peer address was available.

Caveat — per-process. These counters describe one server process. Behind a load balancer the per-peer bounds they report are per-replica, not fleet-wide (see docs/user-manual/server-admin.md).

Agent-side signature refusals (#416/#3807)

Emitted by the AGENT, not the server, and therefore not scrapeable directly — the agent has no /metrics endpoint. They reach an operator only by the route described below.

How it reaches you. The total across all four reasons is carried on the agent heartbeat as the status tag yuzu.ota_signature_refused, and the server derives yuzu_fleet_ota_signature_refusing_agents from it. Nothing else reads the tag. If you need per-reason detail for a specific endpoint, it is in that endpoint’s own log — the update path has no status-report RPC, so the reason does not travel to the server.

Which mode an agent is in (#5249). The heartbeat also carries the status tag yuzu.ota_signature_mode: off (no trust bundle — update binaries are not signature-checked at all), bundle (bundle set, unsigned packages still accepted), or bundle+require (unsigned packages refused). It is the mode the agent process loaded at startup, which the agent also logs as OTA update signature mode: …. A refusal counter of 0 does not mean an agent is verifying: an off agent never refuses anything. The server keeps the tag in its in-memory agent-health snapshot with the agent’s other heartbeat tags; no fleet gauge, REST field or dashboard view reads it yet, so today the per-endpoint source of truth is that startup log line.

Histogram buckets

Most histogram metrics use the same default bucket boundaries (in seconds); a few carry a custom ladder called out in their own row (including yuzu_server_guardian_event_store_duration_seconds at 0.1ms-10s and yuzu_pg_acquire_wait_seconds into the 10-60s tail). The default:

0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0

These buckets are suitable for request latency tracking. The range covers sub-millisecond local operations through multi-second network calls.

Prometheus scrape configuration

Add Yuzu to your Prometheus prometheus.yml. For local scrapers, no credentials are needed. Remote scrapers need --metrics-no-auth on the server or must provide authentication:

scrape_configs:
  - job_name: 'yuzu-server'
    static_configs:
      - targets: ['localhost:8080']
    metrics_path: /metrics
    scheme: http

Multiple servers:

scrape_configs:
  - job_name: 'yuzu-server'
    static_configs:
      - targets:
          - 'yuzu-primary.example.com:8080'
          - 'yuzu-secondary.example.com:8080'
    metrics_path: /metrics
    scheme: https

Verifying the scrape

After adding the configuration, confirm that Prometheus is scraping successfully:

# Check target status
curl -s 'http://localhost:9090/api/v1/targets' | jq '.data.activeTargets[] | select(.labels.job == "yuzu-server")'

# Query a metric
curl -s 'http://localhost:9090/api/v1/query?query=yuzu_agents_connected' | jq .

Real-time event stream

GET /events

The server exposes a Server-Sent Events (SSE) endpoint for real-time updates. The dashboard uses this for live agent status, but any SSE client can consume it. This endpoint requires authentication (session cookie).

curl -s -b cookies.txt -N 'http://localhost:8080/events'

Example output:

event: agent-online
data: agent-001

event: agent-offline
data: agent-003

event: command-status
data: cmd-20240319-001|success

event: output
data: agent-001|hardware_info|{"cpu":"Intel i7","ram_gb":32}

Event types:

Note: event data is plain text (not JSON). Fields are pipe-delimited where multiple values are present.

Agent lifecycle via WatchEvents RPC

The gRPC WatchEvents RPC provides the same lifecycle events (connect, disconnect, plugin load) over a bidirectional stream. This is used internally by the gateway and can be consumed by custom integrations that prefer gRPC over SSE.

Fleet performance gauges (DEX)

Published on every fleet-health sweep (~15 s). A metric nobody reported is absent, never zero (exception: server-owned agent counts such as yuzu_fleet_echo_db_corruption_agents and yuzu_fleet_inventory_sync_skipping{source} are published every sweep, 0 included); values are validated server-side (forged non-finite / out-of-range readings are rejected).

Per-OS breakdown (C1) — the same rollup, additionally split by the agent’s OS (os = windows / linux / darwin / unknown / other, same allowlist yuzu_fleet_agents_by_os uses). Published alongside the fleet-wide families above, cleared and rebuilt every sweep (absent, never a stale 0, for an OS nobody reported this cycle). Windows and Linux both have a real heartbeat perf collector today; a darwin agent can be online (yuzu_fleet_agents_by_os{os="darwin"} > 0) while never appearing in any yuzu_fleet_perf_os_* series — that is honest absence, not a bug, until the macOS collector ships. Never blend these into a cross-OS aggregate — that is exactly what the fleet-wide families above already are:

Per-cohort export (opt-in) — published only when a cohort export tag key is configured (DEX → Alerts tab; runtime_config key dex_cohort_export_key):

Empty cohorts and cohorts beyond the top-50-by-population cap never export; a one-device cohort exports that device’s exact avg/p50/p90/max, labelled by the tag value; devices without the key export as cohort="(untagged)". Families clear on every sweep — series go absent, never stale. Alerting recipe: max_over_time(yuzu_fleet_perf_cohort_clipped[1m]) for liveness, yuzu_fleet_perf_cohort_clipped > 0 for 5m for cap pressure; with the ~15 s sweep cadence, scrape at ≤10 s intervals when the export is enabled. See the label-exposure note under Security considerations before choosing a key.

Fleet network gauges

Published on every fleet-health sweep (~15 s), fed from the yuzu.net_* heartbeat tags (device-aggregate facts only — no per-destination data). A metric nobody reported is absent, never zero. Linux, Windows, and macOS agents emit network facts — macOS throughput only (retransmit + RTT deferred; see Network Quality → Platform coverage). Windows reports throughput and an interval retransmit rate but not RTT (per-connection RTT needs ESTATS — a later slice), so yuzu_fleet_net_rtt_ms stays Linux-only for now (see Network Quality → Platform coverage). The same numbers feed the /network Overview cards, via the shared validators, so the gauges and the page cannot disagree.

Caveat — RTT is coarse. yuzu_fleet_net_rtt_ms is a device-aggregate median across whatever TCP connections are open (loopback / LAN / internet blended), so treat it as a rough signal, not per-flow truth; actionable per-destination / per-app latency is a later warehouse-tier slice.

The yuzu_fleet_net_* series carry an os label (windows / linux / …, matching yuzu_fleet_agents_by_os) so reporting, RTT, and throughput are split per OS, never blended — query and alert per OS (sum(...) a count for a fleet total). The retransmit rate is the exception: the Windows rate is system-wide, biased low, and not yet loss-validated (#1465), so it is withheld from the gauge entirely (it still shows on the /network page + REST, caveated) — yuzu_fleet_net_retrans_pct carries only loss-validated OSes (Linux today).

Network sampling shares the --dex-disable agent flag; disabling DEX also disables the network heartbeat tags. All six yuzu_fleet_net_* families are cleared on every sweep and re-emitted per reporting OS — a (metric, os) nobody reported is absent, never stale or a fabricated 0 (_degraded{os} only appears if an agent of that OS still emits the retired tag).

SparkEngine fleet gauges

Published on every fleet-health sweep (~15 s), fed from the yuzu.spark_* heartbeat tags. SparkEngine (ADR-0021 Stage-2) is the next-generation event-driven detection engine; at rung 1 it runs observe-only (no consumer arms rules), so every mechanism- and consumer-health counter below is 0 in production today — _reporting, _disabled, _failed and _mechanisms are the four that carry live signal, and none of them depends on a consumer being armed; the rest go live when rung 2 arms real rules. A (metric, os[, mechanism]) nobody reported is absent, never a fabricated 0 (all twelve families are cleared and re-emitted every sweep, the same idiom as yuzu_fleet_net_*). An agent started with --spark-disable reports spark_running=0 + spark_disabled=1: it is absent from _reporting and counted in _disabled. An agent whose engine was enabled but threw at boot reports spark_running=0 with no spark_disabled key, and is counted in _failed — that split is what makes a fleet-wide spark boot failure visible at all.

All series carry an os label — file and registry mechanisms are Windows-only, service is Windows + Linux, macOS has none — so query and alert per OS, never sum without(os) (a cross-OS aggregate is meaningless for a single-platform mechanism). Five families additionally carry a mechanism label (file / registry / service): _mechanisms is a live per-agent capability gauge (see its own row above); _watch_rejected, _quarantined and _slow_op are fleet sums of monotonic per-agent counters, so a bare > 0 alert latches until the reporting agent restarts — the shipped counter alert templates (disabled until rung 2) use increase(...[15m]) > 0 instead. _unsupported (F7, #2298 rung 2) is the other gauge exception: a live current gauge, not a counter - it can legally decrease, so it must never be alerted on with increase()/latching logic; see its own row below. One rule ships active today: YuzuSparkBootFailed on _failed (a per-sweep state gauge, latch-free) — warning severity, 30m hold (see docs/prometheus/yuzu-alerts.yml, group yuzu-fleet-spark-rung1).

Staged-rollout example: these series only exist for agents that have been upgraded to a spark-capable (rung-1+) build. During a phased agent rollout of a 2,000-endpoint Linux fleet with 500 agents upgraded, yuzu_fleet_spark_reporting{os="linux"} reads 500, not 2,000 — the other 1,500 agents are ABSENT (pre-rung-1), which is expected and is not a failure. _reporting approaches the enrolled count as the rollout completes; only _failed indicates something wrong.

Finding an inert mechanism from the fleet series. Compare _reporting with ONE mechanism’s series, per OS, and only where every agent of that OS is expected to run that mechanism (Windows and file, for example). Match on instance as well as os: each scraped server replica publishes its own copy of these series, and matching on os alone fails with found duplicate series for the match group as soon as two replicas are scraped (on(os, instance) also works where no instance label exists). The or fallback keeps the query firing when the mechanism series is absent on every agent, where a plain subtraction would return nothing:

yuzu_fleet_spark_reporting{os="windows"}
  - on(os, instance) (yuzu_fleet_spark_mechanisms{os="windows",mechanism="file"}
                      or on(os, instance) (yuzu_fleet_spark_reporting{os="windows"} * 0)) > 0

The Linux form is the same with os="linux" and mechanism="service" (the only Linux mechanism); it counts every agent without a systemd system bus, so every containerised agent, as a gap, and is only meaningful for a fleet that has none. macOS has no per-mechanism detector: it has no mechanism by design, so no series is published and there is nothing to compare.

Do NOT compare _reporting with the SUM over mechanisms. A healthy Windows agent contributes to three series (file, registry, service), so the sum is about 3N against N reporting, and a fleet with file dropped still sums to about 2N; the comparison stays quiet until the average agent has lost more than two of its three mechanisms, so it cannot detect one inert mechanism on Windows. On Linux the sum form fires only while at least one agent still reports service: a fleet in which every agent is inert publishes no {os="linux"} mechanism series at all, and the comparison (like a plain subtraction) then returns nothing.

The fleet series cannot tell a boot-time inert from a runtime one (a File worker or Registry sweeper that failed three consecutive passes and has not yet recovered); the agent log does, see Diagnosing an inert File worker or Registry sweeper. No alert on yuzu_fleet_spark_mechanisms ships today. One written against a runtime-capable mechanism needs a for: hold of at least two heartbeats (60 s at the default 30 s heartbeat) or a self-clearing episode will fire it, and a paging alert wants at least 10 minutes, tuned on fleet data. The per-agent comparison is above zero while any one agent is in an episode, so a for: hold protects against one agent’s short episode, not against many independent short ones across a large fleet: tune the hold and a fleet-fraction threshold on fleet data. Shipping such an alert is a precondition for turning Spark detection on (docs/spark-flip-gate.md).

See Guaranteed State → SparkEngine for the observe-only migration, the --spark-disable flag, and the per-rung enforcement-posture table.

Guardian metrics

Broader Guardian metrics — rule push counts, agent apply latency, parse errors, and a fleet compliance-state distribution (compliant/drifted/error/unknown) — are on the roadmap alongside agent-side enforcement metrics.

Guardian journal fleet gauges

Published on every fleet-health sweep (~15 s), fed from the yuzu.guardian_journal_* heartbeat tags. These roll the durable lifecycle-audit journal’s per-agent integrity counters up fleet-wide, so a lost Guardian lifecycle record is visible on /metrics (and to any evidence automation that scrapes it) instead of only in one endpoint’s heartbeat.

All series are unlabelled. Unlike yuzu_fleet_net_* / yuzu_fleet_spark_*, there is no os or stat split: these are integrity and loss signals, where the fleet question is “did any endpoint lose a lifecycle record” - a flat sum answers that and a percentile obscures it. One consequence worth having: no agent-controlled label means no cardinality exposure at all.

What absence means, mechanically. For the counter families, the agent emits a journal tag only when the counter is non-zero, and every family is cleared and re-published each sweep. So an absent family means: no retained agent’s latest heartbeat carried a value for it that passed the forged-value parse. (The three _seconds_max age gauges at the end of the table have their own emission posture - see their rows; the cleared-and-republished-per-sweep rule applies to them too.) Both edges matter and both are pinned by tests - a non-conforming agent’s explicit "0" parses and publishes the family at 0, while a rejected non-zero value (over the plausibility ceiling, say) leaves it absent and increments _tag_rejected. What the rollup avoids is publishing a fabricated 0 for a family nobody reported, which would read as “checked, nothing lost” when nothing was checked.

This page deliberately stops there. It tells you what each gauge counts and when it is emitted, not what a reading proves about fleet health - the interpretation belongs with whoever enables alerting at the prefer_spark cutover, against a live journal.

The counter rows are gauge-typed fleet sums, and they are monitor-only. No churn-robust new-increment alert exists over an unlabelled fleet sum of per-agent cumulative counters: the sum cannot tell a new increment from a returning agent. The full analysis - why increase()/rate(), delta() and bare > 0 each fail, what the rising-edge template does and does not buy, and the enable-time prerequisites - is stated once, in the yuzu-guardian-journal preamble in docs/prometheus/yuzu-alerts.yml, alongside the disabled templates themselves. Graph these, review them after an incident, do not page on them. The three _seconds_max age rows are structurally different: a fleet MAX of live ages with a meaningful absolute threshold, where a plain > X for: Y IS sound (see the commented staleness examples in the same alerts file) - do not apply the counter family’s monitor-only reasoning to them wholesale, and equally do not apply an “any journal series > 0” panel to the family, since ~30 s is a healthy staleness reading.

Two meta-signals are the exception. yuzu_fleet_guardian_journal_reporting and ..._tag_rejected are server-owned counts published on every completed sweep, including at 0, unlike the 33. Two consequences, and the second is easy to miss:

• _reporting == 0 while yuzu_fleet_agents_healthy > 0 narrows the pipeline-dark question - but only once Guardian is actually deployed, since the sparse writer means 0 also covers “nothing has been journalled anywhere since restart”.
• Because they are never cleared, the registry retains their last value indefinitely, so neither their absence nor a 0 detects a stalled sweep - a stall freezes the previous sample in place. Use yuzu_server_reaper_sweep_duration_seconds for sweep liveness.

Two caveats that apply however you consume these. An agent ageing out of the 90 s staleness window drops its counters from the fleet sum without the underlying loss having healed. After a server restart the series rebuild as agents heartbeat back in, which takes longer than one sweep (the sweep is 15 s; the agent heartbeat interval defaults to 30 s). Treat the metric as a signal and the audit trail as the evidence.

Known limitations - read before citing these as assurance. The signal is self-reported by the endpoint whose audit trail is in question. Fabrication is possible (a forged plausible value from an enrolled agent sets the gauge) and so is suppression (an agent that simply stops emitting is indistinguishable from a healthy quiet one). The series are unlabelled, so a raised value cannot name the affected endpoints, and no operator surface exposes per-agent yuzu.guardian_journal_* tags today - building one is a prerequisite for enabling any alert here. Treat this as a best-effort compensating indicator, not independent evidence of audit-trail completeness.

Timing after cutover. The journal keeps 7 days of batches, so the eviction counters (_evicted_*) are structurally silent for roughly the first week after prefer_spark flips - absence in that window is guaranteed rather than evidentiary. After it, a mass reboot or OTA can produce a large step in _evicted_no_send_evidence about a retention window later that is population churn, not a new incident.

Selector caveat. Six counters in this family break the yuzu_fleet_guardian_journal_* naming pattern - yuzu_fleet_guardian_send_exceptions, yuzu_fleet_guardian_send_orphan_exceptions, yuzu_fleet_guardian_send_stalls, yuzu_fleet_guardian_send_order_hold_releases, yuzu_fleet_guardian_drain_exceptions, and yuzu_fleet_guardian_sweep_exceptions (their wire tags likewise drop the _journal_ infix, sitting beside drain_exceptions). A PromQL selector like {__name__=~"^yuzu_fleet_guardian_journal_"} silently omits all six. Match ^yuzu_fleet_guardian_ if you mean the whole family - and note that EITHER selector now also sweeps in the three _seconds_max age gauges, which are MAX-rolled live ages (not sums of counters): exclude _seconds_max$ from any panel or rule that assumes “non-zero means something was lost”.

Guardian arm-ledger fleet gauges (rung 9c PR-3)

Published on every fleet-health sweep, fed from yuzu.guardian_arm_pending / yuzu.guardian_arm_failed. Unlike the journal counters above, these are re-statable gauges - each agent reports its CURRENT ack-ledger state (GuardianArmAckLedger::arm_stats()), not a monotonic total - so the fleet gauge is a SUM of current values, and it can legally decrease. Absent means no agent reported this sweep, which today (prefer_spark off fleet-wide) means every agent, by construction - a non-spark agent never emits this pair at all.

Guardian io-ceiling fleet gauge (rung 9c PR-3, R5.1)

GuardianIoExecutor’s physical-orphan alive-worker ceiling (R5.1’s own flagged observability gap) gets one narrow, MONITOR-ONLY counter, scoped to the arm/disarm executor instance only - not #3415’s general per-class executor-fault egress, which stays a separate, still-open follow-up (see the executor bulkhead docs in guaranteed-state.md and docs/spark-legacy-delta-registry.md row D10). A windowed “currently under repeated pressure” detector was considered for this signal and deliberately DEFERRED to a later rung rather than shipping an unmeasured threshold now - this counter names the fault class only.

Guardian M1 health-stream fleet gauges

The M1 flood-guard telemetry (Guaranteed State’s errored-view staleness/priority-lane backstops), the #2993 outbox-backpressure signal, and the #4783 legacy-sink loss-visibility trio, the #5403 pending-Spark-Disarm deadline count, the #4472 outstanding-compensation deadline count, and the #5404 Spark claim-lifecycle counters and retained-tombstone count: sparse values (cumulative counters and one current count; the table below is authoritative for the set) rolled up as an unlabelled fleet sum, same absent-not-zero rule and same forged-value posture as the journal family above - a healthy fleet reads every one absent, never a fabricated 0. Two further rows, the #5403 pending-Disarm age and the #4472 outstanding-compensation age, are not counters and roll up as fleet MAX (below). Absence on its own does not mean a producer is dormant, because every counter row is sparse and a 0 ships no tag. By producer: the four M1/outbox rows and the Spark-runtime rows (the two deadline counts, the #5404 claim-lifecycle rows other than _ack_maint_exceptions, and both age gauges) are normally absent on an inert (prefer_spark off) fleet, because they read 0 or have nothing pending there. The three _legacy_sink_* rows are always live regardless of the Spark flip (the legacy IGuard sink is the current production detection path, see the rows below) and CAN fire in that mode; _ack_maint_exceptions is engine-owned and its push-apply source is not gated on the flip, so it can fire in that mode too (its heartbeat ack-drain source is live only while prefer_spark is on). Both are still sparse and absent while they read 0, so do not expect any fixed number of present series. Monitor-only, same reasons as the journal family: no churn-robust alert form exists over an unlabelled fleet sum of per-agent cumulative counters.

Three terms used by the Spark rows below. A retained tombstone is a dead, never-started Arm claim (its arm failed or was abandoned before it ran) that still holds its rule-to-key index entry because the release that should have freed it failed. An orphan key is a subscription key that no rule uses any more and that has no teardown queued: the last rule was taken off it without a Disarm being queued. A compensating teardown is the disarm of a live subscription from an arm that nobody adopted: the rule was withdrawn or superseded while its arm was in flight and the arm still succeeded, or the commit threw. A young compensation age on a healthy agent is therefore normal.

errored_rules on GET /api/v1/guaranteed-state/status and /status/{agent_id} (#2298 item 6d) is a separate, REST-surfaced count derived from the guardian_agent_rule_status census table, not from these heartbeat gauges - the two answer different questions (this family: how much flood-guard activity occurred; errored_rules: how many rules are errored right now).

NVD CVE sync metrics

The server maintains a local mirror of the NVD (National Vulnerability Database) CVE catalog (see NVD CVE sync for the sync flags). These metrics surface the catalog’s size, backfill progress, and sync-window health. The gauges are refreshed on every /metrics scrape.

Reading the reason label: http_429 is rate-limiting (backed off and retried automatically — expected during a large backfill without an API key, not a fault); http_403 is a bad or revoked --nvd-api-key (not retried — rotate the key); connection is an egress/network failure to services.nvd.nist.gov; http_other is any other non-2xx status; parse is a malformed response body. See Rate-limit and auth-error handling for operator guidance.

Alerting. Only http_403 is unambiguously operator-actionable, so page on it and merely record the rest:

# Page: bad/revoked API key — sync makes no progress until rotated
increase(yuzu_nvd_sync_failures_total{reason="http_403"}[1h]) > 0
# Do NOT page on http_429 — it is expected rate-limiting, self-heals via backoff.

Note that http_429 will climb during a first-run full backfill on a server with no --nvd-api-key (each window that exhausts its retry budget increments it before the next tick retries) — that is expected first-run behaviour, not a regression. A sustained yuzu_nvd_backfill_complete == 0 (see above) is the durable “mirror stuck” signal, not the failures counter on its own. Under --no-nvd-sync/YUZU_NO_NVD_SYNC, yuzu_nvd_backfill_complete is absent (never emitted), not 0 — an absent()-style alert built on this series should account for the deliberately-disabled case, not just the stuck case.

Management group metrics

The server exposes two gauges plus two counters for management group telemetry. The gauges are refreshed on every /metrics scrape; the counters are cumulative.

yuzu_server_mgmt_group_backfill_total{result} (the one-time SQLite→Postgres confinement backfill outcome) was retired along with ManagementGroupStore::migrate_from_sqlite() itself (chore/retire-migrate-from-sqlite-batch-a, #3623) — no production fleet ever ran a pre-Postgres build, so there was no real management-groups.db data to migrate. See ADR-0042’s Update.

Example output:

# HELP yuzu_server_management_groups_total Total number of management groups
# TYPE yuzu_server_management_groups_total gauge
yuzu_server_management_groups_total 5

# HELP yuzu_server_group_members_total Total members across all management groups
# TYPE yuzu_server_group_members_total gauge
yuzu_server_group_members_total 42

Useful PromQL queries:

# Total management groups
yuzu_server_management_groups_total

# Average members per group
yuzu_server_group_members_total / yuzu_server_management_groups_total

# Alert if no management groups exist (store may be down)
yuzu_server_management_groups_total == 0

# Confinement reads degrading → scoped operators see reduced/empty lists (DenyAll).
# A degrade denies confinement reads fleet-wide — treat any non-zero rate as a
# confidentiality/availability signal, not a fleet-size change. (Shipped as the
# YuzuMgmtGroupReadDegraded alert.)
sum(rate(yuzu_server_mgmt_group_read_degrade_total[5m])) by (reason) > 0

Custom properties metrics (ADR-0045)

The CustomPropertiesStore — operator-authored per-agent metadata (schema custom_properties_store) usable in scope expressions via props.<key> — is a migrated PostgreSQL store (authoritative posture).

yuzu_server_custom_properties_backfill_total{result} (the one-time SQLite→Postgres backfill outcome) was retired along with migrate_from_sqlite() itself (chore/retire-migrate-from-sqlite-batch-b, #3623) — no production fleet ever ran a pre-Postgres build, so there was no real custom-properties.db data to migrate. See ADR-0045’s Update.

Notification store (dashboard feed, ADR-0046)

yuzu_server_notification_backfill_total{result} (the one-time legacy backfill outcome) was retired along with migrate_from_sqlite() itself (chore/retire-migrate-from-sqlite-batch-b, #3623) — no production fleet ever ran a pre-Postgres build, so there was no real notifications.db data to migrate. This store had no other dedicated metric (set_metrics()/the metrics_ pointer were retired with it — nothing else consumed them). See ADR-0046’s Update.

Useful PromQL queries:

# props.<key> scope reads degrading → scoped dispatch/policy/push rules may be
# silently matching nobody. (Shipped as the YuzuCustomPropertiesReadDegraded alert.)
sum(rate(yuzu_server_custom_properties_read_degrade_total[5m])) by (reason) > 0

Analytics event outbox metrics (ADR-0049)

Analytics collection itself is disabled entirely with --no-analytics — no store is constructed, and the four metric families above are not registered at all (absent from /metrics, not present-at-zero; absent()-based alerting distinguishes this correctly from the degraded-but-registered case). See Upgrading for the Postgres-cutover behavior change and ADR-0049 for the store’s fail-soft/degrade-distinguishable posture.

Tag store metrics (device tags / scope-targeting substrate, ADR-0050)

yuzu_server_tag_store_backfill_total{result} (the one-time tags.db → Postgres backfill outcome) was retired along with migrate_from_sqlite() itself (chore/retire-migrate-from-sqlite-batch-b, #3623) — no production fleet ever ran a pre-Postgres build, so there was no real tags.db data to migrate. See ADR-0050’s Update.

Useful PromQL queries:

# Tag reads degrading → tag-scoped dispatch failing closed to zero agents and
# policy tag-checks silently skipping. (Shipped as YuzuTagStoreReadDegraded.)
sum(rate(yuzu_server_tag_store_read_degrade_total[5m])) by (reason) > 0

Product pack store metrics (operator-installed content, ADR-0054)

Useful PromQL queries:

# Product-pack reads degrading → GET /api/product-packs* failing closed to 503.
# (Shipped as YuzuProductPackReadDegraded.)
sum(rate(yuzu_server_product_pack_read_degrade_total[5m])) by (reason) > 0

yuzu_server_product_pack_backfill_total{result} (the one-time legacy backfill outcome) was retired along with migrate_from_sqlite() itself (chore/retire-migrate-from-sqlite-batch-b, #3623) — no production fleet ever ran a pre-Postgres build, so there was no real product-packs.db data to migrate. See ADR-0054’s Update.

Instruction store metrics (content-plane catalog, ADR-0058)

The InstructionStore (InstructionDefinition -> InstructionSet -> ProductPack, schema instruction_store) is a migrated PostgreSQL store (authoritative posture). It has no legacy-SQLite backfill (ADR-0009’s fresh-start-by-default class) — there is no *_backfill_total family for it, unlike most stores on this page.

Useful PromQL queries:

# InstructionStore reads degrading → REST/MCP catalog reads failing closed to 503.
sum(rate(yuzu_server_instruction_read_degrade_total[5m])) by (reason) > 0

# No server reporting a clean bundled-content reseed this boot → the reseed loop hit a
# genuine DB error and the boot refused to start (absent-clean shape, not
# result="errored" — a refused boot never serves /metrics).
absent_over_time(yuzu_server_instruction_bundled_content_total{result="clean"}[15m])

Per-device concurrency enforcement metrics (ADR-1007)

per-device is the only concurrency mode with real enforcement (see docs/user-manual/instructions.md §8) — a claim table (execution_tracker.concurrency_claims) excludes an agent already mid-execution of the same definition from a repeat dispatch. All six metrics below are pre-seeded to 0 at server construction except the liveness gauge, which is set unconditionally on every reconciler pass and so needs no pre-seed (absent for at most one reconciler cadence — ~5 minutes — after boot).