Audit Log

Yuzu records every operator action as a structured audit event. The audit log provides an access-controlled, retention-managed trail suitable for compliance reporting, incident investigation, and forwarding to external SIEM platforms such as Splunk or Elastic.

Fixed (#2579), and it affected upgrades only. A retention pass that began with NO stored clock reading declined nothing on its own account, so a host whose clock was ALREADY skewed forward when it first ran the guard could delete expired rows unremarked. Such a pass now declines once, warns, and anchors the reading; the next pass proceeds normally. A correct clock was never at risk, and a fresh install with nothing expired does not decline. If you ran an affected version with a wrong clock, rows deleted then are not recoverable without a backup taken before it. Detail under Limits.

Known issue in v0.9.0 (advisory YZA-2026-001): audit rows for requests authenticated via Authorization: Bearer or X-Yuzu-Token (every MCP tool call and every REST automation using an API token) had an empty principal field. Cookie-authenticated dashboard activity was unaffected. Fixed forward in v0.10.0; pre-fix rows are not backfilled. Operators auditing a window that spans the v0.9.0 → v0.10.0 boundary should expect a bimodal principal distribution. See CHANGELOG.md for full remediation context.

Storage

Audit events are persisted in the server’s PostgreSQL substrate under the dedicated schema audit_store (ADR-0006/0040) — the same database that backs the rest of the server’s stores, not a separate file. There is no SQLite fallback: if the substrate is unreachable at boot the server fails closed and refuses to start.

Because the audit log is the SOC 2 evidence chain, its write and read paths have deliberately asymmetric failure postures:

• Writes fail hard. A persistence failure is never silently dropped — the write returns an error to the caller. Behavioural-PII REST routes turn that into 503 + Sec-Audit-Failed: true and serve no data, so evidence-less PII is never recorded as audited.
• Reads degrade to DENY. A store or connection-pool failure on a query returns 503, never a false-empty 200. A reviewer or SIEM can never mistake an infrastructure blip for “no audit activity.”

Event structure

Every audit event contains the following fields:

Logged actions

The following actions are recorded automatically. No operator configuration is required.

MCP mcp.query_responses emits a pair of rows on a scope-filtered call. When an agentic worker collects responses it is only partly entitled to (some agents outside its management groups), the tool writes a result=denied row (the access-boundary event — detail carries the distinct dropped-agent count) and a result=success row (the served, possibly-empty set), both under the same principal + correlation key. A SIEM rule must treat these as a pair for one call: the denied row is informational CC6.1 access-boundary evidence (an operator reached toward agents outside its scope and was filtered), not a failed call — do not alert on mcp.query_responses denials as errors on a scoped multi-operator deployment.

The seven /crypto MCP tools write a generic mcp.<tool> row on two more paths. A rejected argument (an unknown key, a wrongly typed value, a bad cursor) writes result=denied, detail invalid arguments: <reason>; the <reason> is fixed text and never echoes the caller’s input. A burst of these from one principal is a misbehaving client, not a gap in access control. A tool whose provider, caller identity or audit sink is not wired answers a non-retryable internal error (the response says only service unavailable, or a ... unavailable data message) and, when an audit sink exists, writes result=failure: a detail ending provider unwired is a deployment fault that cannot occur on a correctly started server, while caller identity unresolved is usually a credential revoked or expired between authentication and the tool call, which can occur on a healthy server. With no audit sink there is no row to write, and the four tools that write a crypto.*.view row and set_crypto_dashboard_layout refuse the call instead of succeeding without one. A row that fails to persist on a working sink is different: the three library reads and get_crypto_ca_stats still answer, with audit_persisted:false, and so does set_crypto_dashboard_layout when only its closing row fails; if its opening attempted row cannot be written, it refuses and does not change the layout. An export_crypto_cbom random-number failure answers a retryable internal error and writes no row.

Per-principal quota-cap rejections (PR 4.4, ADR-1005 class engine principals) are deliberately NOT audited. When an engine principal (principal_kind=="engine", username engine:<slug>) exceeds its per-principal concurrency or rate cap, the server’s pre-routing chokepoint returns 429 (REST A4 envelope / MCP JSON-RPC -32010, both transports — see docs/user-manual/rest-api.md and docs/user-manual/mcp.md) and increments yuzu_server_principal_quota_exhausted_total{side,limit} (docs/user-manual/metrics.md) — it writes no audit row. This is a deliberate fail-visibility choice, not an oversight: a quota rejection is a high-frequency, pre-handler operational event — an engine principal running hot against its own cap, expected under normal steady-state load — not a forensically interesting action against a resource. Auditing every rejection would flood the audit store under a busy or misconfigured engine principal with no compliance benefit; the counter (SIEM-routable, absent()-safe pre-seeded series) is the intended observability surface. Compare with yuzu_onbehalf_rejected_total and yuzu_auth_lockout_blocked_ total (see docs/observability-conventions.md), which follow the same metric-is-the-signal / no-per-rejection-audit-row pattern for the same anti-flood reason.

Agent OTA admission rejections (#913) are deliberately NOT audited, but OTA identity rejections ARE. The two halves of the OTA gate are treated differently on purpose. When a peer exceeds its per-peer concurrency or rate bound, DownloadUpdate returns gRPC RESOURCE_EXHAUSTED and increments yuzu_ota_download_admission_total{decision} (docs/user-manual/metrics.md) with no audit row — the same fail-visibility reasoning as the per-principal quota block above: a peer running hot against its own bound is expected steady-state behaviour during a fleet-wide update, and auditing every rejection would flood the store during exactly the event it is least useful for. By contrast, a rejection from the identity half — no usable client certificate, a certificate from a foreign CA, or an agent_id that does not match the certificate — writes a session.ota_identity_rejected row, because that is an authentication failure against a specific enrolled identity and is forensically interesting exactly once. Metric is the signal; the audit row is the evidence — and the row is rate-limited per peer while the metric is not, so under a flood the counter is the complete record and the rows are a sample of it. See the row’s entry in the table below.

Reconciling approval decisions. After an audit-store fault, list the approval tickets that have more attempted rows than outcome rows, then check each one’s real status (the Approvals page, or GET /api/v1/approvals/{id}). Compare counts per target_id; do not test whether an outcome row merely exists. Timestamps are epoch seconds; set the window to the incident and leave out the last minute, where a request may be in flight. Start the window before the first decision on the tickets you care about, and re-run a suspect ticket over its whole life: a window that starts inside a ticket’s life can cut an early attempted row while keeping a later outcome row, and that surplus outcome row can offset one lost success row. Row timestamps are the writing replica’s clock, while the window is measured on the database now(), so widen the window by the observed skew when a clock anomaly is suspected. A returned ticket is suspect, not proven: an outcome row was lost, or the server stopped between the attempted row and the decision. The same query, with the surrounding procedure, is in docs/ops-runbooks/audit-store-clock-guard.md (YuzuAuditPersistFailures).

SELECT target_id,
       count(*) FILTER (WHERE result = 'attempted') AS attempted_rows,
       count(*) FILTER (WHERE result = 'success'
                           OR (result = 'denied' AND detail NOT LIKE 'approver_assurance:%'))
                                                    AS outcome_rows
FROM audit_store.audit_events
WHERE action IN ('approval.approve', 'approval.reject')
  AND timestamp >= extract(epoch from now() - interval '24 hours')::bigint
  AND timestamp <  extract(epoch from now() - interval '1 minute')::bigint
GROUP BY target_id
HAVING count(*) FILTER (WHERE result = 'attempted')
     > count(*) FILTER (WHERE result = 'success'
                           OR (result = 'denied' AND detail NOT LIKE 'approver_assurance:%'))
ORDER BY target_id;

Result vocabulary. result is predominantly success or denied, with failure for a failure the handler audits itself. partial means the change committed but could not be fully delivered or applied (api_token.rotate, engine_principal.credential.rotate, engine_principal.credential.mint, and others named in their rows); it is never failure, because the change exists. attempted is written FIRST by operations that fail closed on audit (among them the bell, the approvals approve/reject decision, the MFA clears (mfa.disabled from Settings and mfa.reset.breakglass from the --mfa-reset command: each writes an attempted intent row before the factor is touched, so every clear leaves two rows and a blocked: self-target refusal leaves one error row with no attempted row), the plugin-config writes, the instruction-set and dashboard dispatch writes, the Guardian policy-generation raise (guaranteed_state.policy_generation.raise), the DEX score policy writes (dex.score_policy.update, dex.score_policy.reset), and the Integrations writes: webhook.*, offload_target.*, directory.sync, directory.group_mapping.update, settings.mcp): an outcome row follows it (a rejected request leaves attempted then denied or failure), and an attempted row with no outcome row means “started, outcome unknown”. A “failures only” filter of result != "success" therefore also matches every attempted row; exclude it, or key on the outcome row. no_change is an outcome that is not a failure: a dex.score_policy.reset of a policy that was never written records attempted then no_change, and the same filter matches it too. On the Integrations writes, a REST 2xx that carries Sec-Audit-Failed: true means the OUTCOME row was lost after the change was made: the attempted row is then the only record that it happened. The set is not closed in practice - ok, error, warning and skipped also appear on specific verbs. denied is the usual token for an authorization or validation rejection with a real principal behind it, but that is a convention rather than an invariant: enrollment.token_consumed and engine_principal.credential.rotate both use failure for what are really rejections, and instruction.scope_resolution_failed was emitting a self-audited failure before the mcp.bridge.* family existed. Unlike those still-standing violators, the mcp.<tool_name> unknown-tool row was corrected to denied in #2445 — it used to audit failure for a client-caused rejection (the caller named a tool that doesn’t exist), inconsistent with most other denials on the tools/call surface. That surface still has several of its own violators, unchanged by #2445 and not fully enumerated here (tracked for a full sweep in #3176): set_tag’s validation-failure rejection, execute_bundle’s out-of-scope rejection, and the access-review campaign family’s not-found rejections are all client-caused but audit failure. delete_tag’s not-found rejection is client-caused too but does NOT belong on this list — it audits a distinct third token, not_found, not failure. So a SIEM rule that surfaces failure branches by filtering on denied alone will miss several families, including every background-actor row. Filter on result != "success" when you want all of them. Note GET /api/v1/audit does not currently expose a result query parameter, so that filter has to be applied by the SIEM after export rather than server-side. denied is used for RBAC rejections and for every 400/404/409 branch where the handler explicitly audits the failure — including Guardian rule.create (400 missing fields, 409 conflict), rule.update (400 invalid body, 404 not found, 409 conflict), rule.delete (404 not found), push (400 non-object body), and the Phase 8.3 offload_target.create (400 validation_failed) / offload_target.delete (404 not_found) handlers. SIEM rules that filter on result == "success" will match every completed mutation including guaranteed_state.push (which returns 202 rather than 201/200 because agent fan-out is asynchronous). To surface probe/fuzz traffic on the REST surface, filter on result == "denied" scoped to the actions you care about — every mutating branch produces a row.

The /auto pre-flight verbs add two non-denial outcomes: no_devices (a preflight.run whose scope resolved to no visible devices — an operator no-op, not a rejection) and noop (a preflight.run.delete whose run id was unknown or not owner-visible — a no-op rather than a permission denial, which keeps a non-owner from distinguishing “exists” from “doesn’t exist”). Neither is a denied; SIEM rules counting denials should not include them.

Behavioural-PII access-audit rows: result=success means “authorised and attempted,” not always “disclosed” (#4858). The per-device / per-signal PII routes (dex.device.view, dex.signal.view, guardian.device.view, dex.app_perf.devices.view, and their MCP twins) split across postures by (verb, surface), not by verb alone — a SIEM/auditor reading raw rows must know which applies to a given row before drawing a conclusion from result alone:

• Pre-read (most dashboard fragments and REST gates, e.g. dex.device.view, dex.signal.view, the MCP tool get_dex_device_app_perf’s dex.device.app_perf.view row, and the MCP tool list_guardian_events’s dex.device.view row). The audit row is written BEFORE the underlying store read runs, with a literal result=success asserting only that the request was authorised and the read was ATTEMPTED. If the store then degrades (see yuzu_server_guardian_read_degrade_total / yuzu_server_app_perf_read_degrade_total in docs/user-manual/metrics.md), the dashboard renders a placeholder or the REST/MCP caller gets a 503/retryable error — on top of an audit row that already recorded success. Do not read success on these verbs as proof any PII was actually disclosed; corroborate against the read-degrade metric for the same time window if that matters to your investigation.
• Post-read by outcome (e.g. dex.app_perf.devices.view on every surface, and guardian.device.view on REST and the MCP tool). The audit row is written AFTER the read, with result set to success/failure by the read’s actual outcome — here success does mean the read itself succeeded (though a genuinely-empty result, e.g. an unrecognised agent_id, is still success — a legitimate empty answer, not a failure). guardian.device.view is surface-dependent: on REST and MCP it is this post-read-by-outcome posture, but the device-page Guardian lens dashboard fragment (GET /fragments/device/guardian) emits the SAME verb pre-read with a constant result=success instead (the first bullet’s shape) — the verb name alone does not tell you which posture produced a given row; check which surface emitted it.
• Post-read constant success (the MCP get_dex_device_score tool’s dex.device.view row, and the MCP get_dex_device_history tool’s dex.device.view row). The row is written AFTER the read but always records result=success regardless of the read’s degraded outcome (the MCP set-and-proceed convention for the AUDIT row — there is no response-header channel). This is distinct from audit_persisted, a SEPARATE signal for a SEPARATE failure mode: audit_persisted:false appears in a successful tool response body only when the audit write itself failed to persist (the row never made it to storage); it is not present when the underlying read degraded instead (that case returns an a4_error response — get_dex_device_score/get_dex_device_app_perf both — with no audit_persisted field at all, since there is no success body to carry it in). Do not conflate “the audit row didn’t persist” (audit_persisted:false) with “the read degraded” (an error response, no such field) — corroborate a degraded MCP read against the store’s own read-degrade metric instead.

The Hardware CI-record verbs inventory.device.ci and inventory.device.software add one case to the post-read-by-outcome group: when the device roster read failed, the row carries result=failure even though the record itself is still served (the device identity is shown as unavailable); key on the identity=roster unreadable detail token, not on result alone. The not-served arms (nothing else shown, by the two predicates in the inventory.device.ci row: the REST record’s 503 and the dashboard fragment’s “Device roster unreadable” page) carry state=roster unreadable with the same result=failure, so a SIEM rule should key on both tokens. The MCP tool get_hardware_ci writes the same state=roster unreadable token (with (MCP) in the detail prefix) and the same result=failure on its inventory.device.ci row, and list_hardware_ci writes result=failure hardware roster read failed on inventory.devices; each also writes a matching result=failure generic mcp.<tool> row. get_hardware_ci additionally writes state=software or tags unreadable (also result=failure) when the device is otherwise absent and only the software or tag read failed, so a SIEM rule should key on all of these tokens.

This split is a known, accepted state across surfaces, not a defect in any one of them; converging on one posture is deferred to the WS-B2 audit-relocation step (ADR-0031). See server/core/src/rest_audit.hpp’s file banner for the authoritative, code-level statement of this contract.

The /auto deploy verbs share this pattern: deployment.create emits resumed (re-attached to an in-flight deployment for the same run instead of creating a duplicate) and no_devices (empty go-cohort after scope re-intersection); deployment.delete emits noop (unknown / not-owner-visible id). None is a denied. deployment.advance is high-volume (one row per poll tick); SIEM rules counting operator-initiated actions should treat a same-deployment_id burst as one session, not many.

gRPC audit-failure signal. REST surfaces set the Sec-Audit-Failed: true response header (and an audit_emitted=false body field) when an audit row fails to persist. The gRPC Register/Subscribe paths carry the equivalent as trailing metadata x-yuzu-audit-failed: true (#1063; the key is a shared constant in grpc_audit_signal.hpp so it cannot drift between the direct-agent and gateway-proxy services). In both transports the underlying operation still completes — enrollment proceeds, a peer-mismatch Subscribe is still rejected — only the SOC 2 evidence chain is degraded for that one request. SRE alerting should cover both signals; SIEM pipelines ingesting gRPC telemetry should watch for the x-yuzu-audit-failed trailer on Register/Subscribe RPCs. The gateway-upstream peer-authorization guard deliberately does not set this trailer on its refusals (see session.gateway_peer_denied above): a refused call is decided before its audit row is written and its counter is complete either way.

REST API

Who may read the audit trail. Reading it needs the AuditLog:Read permission, on every surface: the REST routes below, the dashboard viewer, and the MCP query_audit_log / audit_engine_no_admin tools and yuzu://audit/recent resource.

• RBAC disabled (the shipped default): only an admin-role session may read the trail (#5771). Any other authenticated session gets 403, audited with a detail beginning topology floor: non-admin role denied AuditLog:Read (match on the prefix: an MCP-tier token’s row appends (mcp_tier=<tier>)). Engine principals and service-scoped tokens are refused as before.
• RBAC enabled: AuditLog:Read is an ordinary grant. The seeded Administrator, PlatformEngineer, Operator, ITServiceOwner and Viewer roles hold it; a custom role can be granted it. A holder sees every row.

See RBAC for the floor and for how to grant a non-admin the read.

GET /api/v1/audit

Query audit events with filtering and pagination. Maximum page size is 1000.

Query parameters:

The feature behind the echo.* actions was named TAR before 2026-10. Audit rows written before the rename keep their tar.* action spelling and are never rewritten; an action filter or action_prefixes entry that starts with echo. matches both spellings, while a tar.* filter matches only the old rows.

A negative since, and an until of 0 or below, are rejected (400 on /api/audit and GET /api/v1/audit, -32602 on MCP query_audit_log; to=0 is 400 on GET /api/v1/audit/auth-sample): the bounds are presence-tracked, and a literal -3600 or until=0 used to mean no bound and returned the whole log. Omit the parameter for no bound; since=0 (from=0 on the auth sample) is accepted and matches the same rows as omitting it. A strict-parse 400 on these audit and analytics reads (GET /api/audit, GET /api/v1/audit, GET /api/v1/audit/auth-sample, GET /api/analytics/recent) writes no audit row, and neither does one on the notification and patch lists, the legacy policy lists, the pre-flight run form, the OTA rollout change, the DEX blast-radius thresholds, the API-token form, or the DEX signal, DEX perf and network device lists, because each is answered before any audit write. These routes audit their rejection instead, with a result=denied row for a malformed or out-of-range number (see their rows): POST /api/settings/enrollment-tokens, POST /api/settings/enrollment-tokens/batch, POST /api/v1/offload-targets and POST /api/v1/guaranteed-state/rules (an out-of-range version). GET /api/v1/guaranteed-state/events parses limit and offset before it writes its dex.device.view row, so its 400 leaves no access row.

Example --- fetch the 20 most recent events:

curl -s -b cookies.txt \
  'http://localhost:8080/api/v1/audit?limit=20' | jq .

Response:

{
  "data": [
    {
      "id": 4101,
      "timestamp": 1710849600,
      "principal": "admin",
      "action": "tag.set",
      "result": "success",
      "target_type": "Tag",
      "target_id": "agent-001:environment",
      "detail": "Production"
    }
  ],
  "pagination": {
    "total": 20,
    "start": 0,
    "page_size": 100
  },
  "meta": {
    "api_version": "v1"
  }
}

total is the number of rows returned in this response, not the number of rows that matched. start echoes offset and page_size echoes limit; a page shorter than limit is the last page. The MCP query_audit_log tool also returns has_more, true when more matching rows exist past the page. Rows are ordered newest first and rows with an identical timestamp break by id (newest first). On an unchanging result set offset paging neither repeats nor skips rows; the log is live, so a new event or a retention delete between page fetches shifts later pages. Pin the window with a fixed until in the past and use id to de-duplicate. offset is capped at 10000 and never clamped (a larger value is a 400). To read deeper, set until to the oldest timestamp already seen (inclusive), restart at offset=0, and de-duplicate on id.

Example --- filter by principal and action:

curl -s -b cookies.txt \
  'http://localhost:8080/api/v1/audit?principal=admin&action=auth.login&limit=50' | jq .

Example --- find all failed login attempts:

curl -s -b cookies.txt \
  'http://localhost:8080/api/v1/audit?action=auth.login_failed' | jq .

GET /api/audit (legacy)

The legacy endpoint caps limit at 1000 and rejects offset above 10000, the same ceilings as /api/v1/audit. It answers 400 for a negative since, an until of 0 or below, an inverted window, and a string filter holding a NUL or invalid UTF-8 (the v1 route does the same for the string filters). New integrations should prefer /api/v1/audit, which also rejects a target_id given without target_type.

Query parameters:

The legacy response envelope differs from v1:

{
  "events": [ ... ],
  "count": 20,
  "total": 150
}

The legacy response includes additional fields per event not present in the v1 response: principal_role, source_ip (id is on both).

curl -s -b cookies.txt \
  'http://localhost:8080/api/audit?limit=1000&principal=admin' | jq .

Dashboard viewer

The audit trail is readable from the dashboard at Access > Audit and reviews (/access/audit). The page is open to every signed-in user, but each panel is gated by its own permission and a panel your account cannot read says so instead of rendering empty.

Audit log. Needs AuditLog:Read (admin-only while RBAC is disabled). It reads the same store, with the same gate, as GET /api/v1/audit and offers a subset of its filters (below), and shows only the fields that route returns: time, principal, action, target, result and detail, plus the event id in the row detail. The source address and the principal’s role, which the legacy GET /api/audit route also returns, are not shown. Filters:

• a time range of the last hour, 24 hours or 7 days (the page sets since; until is open);
• Principal and Action, both exact matches, not searches.

Rows are newest first, 50 to a page, and the page does not show a filtered total. Paging stops at row 10,000 (the same offset ceiling as the REST routes); the page says so and asks you to narrow the time range, it never silently shows a different page than the one asked for. The log is live, so an event written while you page can shift later pages. If the audit store does not answer, the panel is a red box, never an empty table: an unreadable log is not the same thing as no activity. Reading the log writes no audit row, exactly like the REST route.

Authentication sample. A random sample of sign-in, MFA, session and engine-principal events for evidence, drawn only when you press Draw sample (never on load). It is GET /api/v1/audit/auth-sample with the window left open; the sample size is 1 to 1000. The panel states how many candidates the sample was drawn from and whether the pool was capped to the newest events. Each draw is itself recorded as audit.auth_sample.exported; if that row cannot be written the panel says so.

Access reviews. The same page lists review campaigns, shows a campaign’s frozen grants (undecided first) with its summary, and shows the current grant population on request. It needs AccessReview:Read (the Administrator and Reviewer roles; admin-only while RBAC is disabled), a different permission from the audit log: holding AuditLog:Read does not grant it. A caller who also holds AccessReview:Attest can open a review, attest or flag a grant (with an optional justification) and close a campaign; each is a call to the existing /api/v1/access-reviews routes. Flag for revocation records evidence only; it never revokes the grant. Closing is one-way and may leave grants undecided. /access/audit?campaign=<id> opens a campaign directly. Viewing writes the same rows as the REST reads: access_review.list for the list, access_review.get for a campaign and access_review.exported (format=view) for the current grants, so opening a campaign leaves a row in the trail. If the access review store does not answer, the panel is a red box, never “no reviews”. The campaign list loads on every page view, so a refused browser session writes no denied-access row for it; opening a campaign (including by the ?campaign=<id> link) and reading the current grants are deliberate requests, and a refused caller writes the ordinary permission-denied row for each. A refused API or MCP token writes one on every fragment, and an engine-class session is refused on all three with an access_review.* row, result=denied, detail=engine_session_denied. Settings keeps an older Access Reviews panel; viewing it writes no access_review.* row, so use this page when the reading itself must be on the record.

Not available yet on this page: a campaign due date (campaigns have no due-date field) and a button that exports a campaign’s decisions (the Export decisions button is disabled). A campaign’s complete decisions are in the JSON from GET /api/v1/access-reviews/{id}; the current grants download as CSV from the Current grants panel.

Retention. Shows the configured retention period (--audit-retention-days, 365 days by default) and that cleanup runs hourly. It needs ServerConfig:Read, which is Administrator-only, so it is fetched on request rather than on every page view. It is read only.

Retention and cleanup

Audit events are retained for 365 days by default. A background thread runs every hour and deletes events whose ttl_expires_at has passed. That TTL is stamped once, when the row is written, as insert time + retention window — it is not derived from the event’s own timestamp, and nothing ever rewrites it. Deletion is permanent --- there is no soft-delete or archive step.

To preserve audit data beyond the retention window, export events periodically using the REST API or forward them to an external system (see Planned Features below).

The retention clock guard

Which alert sent you here? Find it in the table below — they report different conditions and the triage differs. The current rule set is docs/prometheus/yuzu-alerts.yml.

Retention is driven by the server’s wall clock. A cleanup pass that looks like it would destroy the audit trail declines instead of deleting: it logs a warning and adds one to yuzu_server_audit_clock_anomaly_skips_total. A decline deletes nothing. It is the protection working, not evidence lost. The rest of this section is about declines.

Retention sweeps fleet-wide from a single process. On the PostgreSQL substrate every replica runs the cleanup thread, but each pass first tries to take an advisory lease (pg_try_advisory_xact_lock('audit_store:reap')); only the lease winner sweeps, so exactly one process prunes per tick across the whole fleet. The 25,000-rows-per-pass cap therefore stays calibrated as a single drain rate — it can never become “N replicas × 25,000.” The clock guard’s dedup state now lives durably in the audit_retention_meta table (the previous pass’s serialized fact-set, alongside the last clock reading), so a condition reported by one replica is not re-reported by another, and the state survives a restart or a failover to a different replica. (Previously, on the single SQLite server, this dedup state was an in-process member — correct for one writer, but it does not port to a multi-replica store.)

Dedup-semantics change (multi-replica correctness). Because the dedup state is now shared across replicas, an identical-magnitude clock step that repeats with the same fact-set is reported once and then stands down, rather than re-incrementing yuzu_server_audit_clock_anomaly_skips_total on every pass as the single-process guard did. A genuinely different anomaly (for example a dead-CMOS boot followed by an NTP correction — a distinct fact-set) still reports; the catastrophic whole-window-wipe protection is fully preserved. Only a second identical clock step re-emitting the counter is lost. If you previously alerted on the cadence of that counter (a fresh increment every pass), alert on a sustained non-zero increase over several consecutive passes instead — see “Is retention actually stalled?” below.

Start with the log line. Every decline logs an AuditStore: warning naming which condition fired, and for an elapsed-time decline it prints the actual gap against the threshold. That answers most of the checklist below without guesswork, so read it before working down the list.

Was this expected?

1. Was --audit-retention-days (or YUZU_AUDIT_RETENTION_DAYS) reduced and the server restarted recently? Expected. Narrowing the enforced window narrows this guard’s survivor horizon, so a pass declines by design. It is a one-off; the backlog then drains at the capped rate. A runtime PUT /api/config/audit_retention_days does not do this: it only stores the value, and the store keeps the window it was built with.
2. Was the server down, suspended, or snapshot-paused for more than 7 days? Expected. Elapsed time cannot distinguish a long outage from a forward clock jump, and the guard deliberately treats both alike. The log line gives you the measured gap, so you do not have to reconstruct it.
3. Is the whole table already past its window? Expected, once. A pass that would expire every datable row declines before draining --- a restored backup older than the retention window is the common way to meet this.
4. Does the log line say the stored reading was unusable? The pass re-anchors on the current reading and the next one proceeds. The log line names both possible causes; a reading ahead of now is what a backward clock movement looks like, so do not rule out the clock here --- continue to rung 5.
5. Otherwise, check host time synchronisation --- chronyc tracking or w32tm /query /status --- and look for a restored VM snapshot, a dead CMOS battery, or a hand-set date. Fix the clock first: the drain resumes on its own once readings are stable.
6. If the clock reads fine but declines keep coming, look for two things setting it. A single correction declines a pass or two and then drains. Only a clock that keeps moving holds the guard declining --- a hypervisor sync fighting the guest’s NTP, two NTP sources disagreeing, a failing RTC being corrected over and over. Between corrections such a host looks healthy, so chronyc tracking alone will not show it; check for more than one time source and look at the correction history.

Is retention actually stalled? Read the counters, not this alert’s firing state. YuzuAuditRetentionClockAnomaly evaluates increase(...[1h]) against a pass period slightly longer than an hour, so it can resolve and re-fire on its own whether or not anything is wrong.

Retention is stalled when, sustained over several consecutive passes:

• yuzu_server_audit_clock_anomaly_skips_total rises on every pass, and
• yuzu_server_audit_retention_passes_total rises by the same amount over the same window (so every pass is declining, not merely some), and
• yuzu_server_audit_rows_deleted_total stays flat.

One or two consecutive declines is normal and is not this state. Do NOT look to yuzu_server_audit_retention_cap_reached_total to confirm a backlog: it only moves on a pass that actually deleted, so it is flat by construction in exactly the state you are testing for.

Whether that stall matters is a separate question, and there is no metric for it --- confirm a backlog exists by querying for events older than the retention window (see Protecting evidence below), or by watching the audit_store schema grow in PostgreSQL. A store with nothing to delete shows the same two counters.

If skips and yuzu_server_audit_rows_deleted_total are both flat, retention is not declining --- it is failing or not running at all; check yuzu_server_audit_cleanup_failed_total and yuzu_server_audit_retention_passes_total.

If skips rises by less than passes over the same window, some passes are declining and some are not. Read yuzu_server_audit_rows_deleted_total to see which: rising means the rest are draining normally, which is the ordinary post-reduction picture in rung 1 and is healthy; flat means the rest are failing, so check yuzu_server_audit_cleanup_failed_total. Compare increase() over a window covering several passes --- never the raw values, which are since-process-start and make skips smaller than passes on any healthy store.

Protecting evidence right now.

• To export a time range, use /api/audit, not /api/v1/audit. The v1 endpoint takes only limit (hard-capped at 1000), principal and action — it cannot express a range and cannot paginate, so it will silently hand you the first 1000 rows of a much larger set. (The periodic-export recipes further down this page use v1 deliberately --- they forward a rolling tail on a schedule, which is a different job from pulling a specific window once.) The legacy endpoint takes since, until, limit and offset, and is the only way to page a range out before the next cleanup interval:

  # Page out everything in a window, 1000 rows at a time.
  # FROM/TO are unix seconds. Stops on the first empty page.
  OFF=0
  while :; do
    if ! curl -sS -f -H "Authorization: Bearer $TOKEN" \
         "$SERVER/api/audit?since=$FROM&until=$TO&limit=1000&offset=$OFF" \
         -o "audit-$OFF.json"; then
      echo "EXPORT INCOMPLETE at offset $OFF" >&2   # token expiry, 503, reset
      exit 1
    fi
    grep -q '"count":0' "audit-$OFF.json" && { rm -f "audit-$OFF.json"; break; }
    OFF=$((OFF+1000))
  done

  Fail loudly rather than quietly: a mid-export 401 or 503 otherwise leaves a directory of files that looks like a finished run. The response’s total field cannot confirm completeness --- it is COUNT(*) over the whole table, not over your since/until window, so it will not match your export and is not a check. Count the pages you fetched instead.

• If a server ran with a backward-skewed clock, export before the next pass. Rows written while the clock was behind carry an already-past TTL, so once the clock is corrected they are the oldest rows in the table and the first the paced delete takes --- at info level, indistinguishable from routine ageing-out. That is the window an auditor asks about first, because it covers a host fault.

• Ship the AuditStore: warnings to your durable log store. The alert rules fire on increase() over a rolling window, so a one-time warning stops being visible in the counter once it ages out and the process restarts. The log is the evidence record; the metric is not.