Echo

The Echo plugin continuously captures system state snapshots and records changes as timestamped events in a local SQLite database on each endpoint. It enables retrospective investigation of “what happened on this machine” without requiring pre-configured logging or SIEM integration.

Engineers maintaining or extending Echo should also read docs/echo-implementer.md — it covers the on-disk format, persistence semantics across upgrade/uninstall/reinstall, the post-restart double-capture caveat, and device-impact expectations.

What Echo captures

Echo monitors these categories of system activity. Every capture source is on by default on every OS — each can be disabled per host with <source>_enabled=false (see Configuration). power and removable media (Wave 6) ship collecting in this release: power (Windows/Linux live subscription; macOS retrospective replay bounded by power_lookback_seconds) — an empty $Power_Live means no transitions occurred, not that collection has not started; removable media (Windows retrospective event-log replay bounded by removable_lookback_seconds; Linux/macOS live-only) — an empty $Removable_Live means no removable-media activity occurred, not that collection has not started. usage (Wave 7) also ships enabled by default alongside them: it derives per-executable run history directly from the always-on process source (no separate collector of its own), starting from the first tick after enablement or upgrade — see the usage_enabled row below for the access-limitations note. software, ARP, DNS, and mapped drives are on by default — software/DNS are Windows-only today (software is machine scope only — ADR-0015 covers ARP/DNS), while ARP also runs on Linux (/proc/net/arp) and macOS (constrained — a sysctl route-table dump, entry_type always reported unknown), and mapdrive (network-share mappings in both directions, §3.8) also runs on Linux and macOS (constrained on macOS — outbound-live only via getfsstat; the username column is always blank, but a credentialed SMB mount source is stored verbatim in remote_path; no inbound, no history). registry (Windows only) records changes to a curated set of registry keys — Run/RunOnce, Services, Image File Execution Options, AppInit_DLLs, Winlogon, LSA, Policies and Defender under HKLM, and the Run, RunOnce, Policies and Winlogon keys of every loaded user hive — with each value’s old and new data; the process that made a change is named by correlation where the agent can (a lead to verify, not proof, and only with an elevated agent — see Registry table under Key columns); its values are stored unmasked (see Registry table under Key columns). Further sources — per-app performance, module loads, per-connection network quality (netqual), and connectivity-transition history (netconn — network/Wi-Fi connect/disconnect + internet-capability changes, including a retrospective backfill of OS-retained history from before Echo was enabled; ADR-0020) — are covered under Configuration and the OS compatibility matrix below.

Every source except Performance and the cursor-model sources (power, removable, which track a durable log position instead) takes a snapshot of the current state each cycle, compares it to the previous snapshot, and records only the differences as events. This keeps the database compact while providing full visibility into system changes.

The Software source diffs the installed-software inventory to record install / uninstall / upgrade events over time — the historical “what was installed or removed on this box, and when” that the point-in-time installed_apps inventory cannot answer. On Windows it captures machine-wide installs only (the HKLM Uninstall keys, 64-bit and 32-bit) — machine scope only, no user identity / no PII. It runs on its own slower trigger (hourly by default — installs are infrequent) and is on by default: disable it per host with software_enabled=false (see software_enabled below). The inventory it gathers is asset-management and vulnerability-relevance data with no user identity, like Services and User sessions. It records names, versions, and publisher only — no command lines, no usage data.

The first run on a host seeds the baseline silently (it records no events) so an installed event always means “installed now”, not “was already present when the agent started watching”. Linux (dpkg/rpm) and macOS (pkgutil) collectors are planned; the $Software_* tables are queryable but stay empty on those platforms until then.

The Performance source is different: it is a fixed-cadence scalar sample, not an event diff. Each 30-second tick records one row of derived device metrics — CPU busy %, memory used % and commit-charge %, per-IO disk service time (µs) and read/write throughput, and non-loopback network rx/tx throughput. It is collected from raw kernel counters (no PDH, no WMI, no shell-out) and, like all Echo data, stays on the device — only aggregates leave the edge. On Windows and Linux it is fully supported (Windows: kernel counters; Linux: procfs); the macOS collector is planned. The first sample after the agent starts is a baseline (it establishes the counter reference and records no row); every subsequent tick records one sample.

Querying Echo data

From the Yuzu dashboard

Use the Echo Query instruction to search events by time range and type.

From the REST API

POST /api/v1/instructions/execute
{
  "plugin": "echo",
  "action": "query",
  "parameters": {
    "from": "1711000000",
    "to": "1711100000",
    "type": "process",
    "limit": "500"
  }
}

Query parameters

Output format

Each event is output as a pipe-delimited line:

timestamp|event_type|event_action|snapshot_id|detail_json

Example:

1711050123|process|started|1711050123001|{"pid":1234,"ppid":1,"name":"nginx","cmdline":"nginx -g daemon off;","user":"www-data"}
1711050123|network|connected|1711050123001|{"proto":"tcp","local_addr":"0.0.0.0","local_port":80,"remote_addr":"10.0.0.5","remote_port":54321,"state":"ESTABLISHED","pid":1234}
1711050423|service|state_changed|1711050423002|{"name":"sshd","display_name":"OpenSSH","status":"stopped","prev_status":"running","startup_type":"automatic","prev_startup_type":"automatic"}
1711050423|user|login|1711050423002|{"user":"admin","domain":"CORP","logon_type":"remote","session_id":"pts/0"}

The cmdline field above is shown populated for a Linux example. On Windows (ETW) and macOS, the process feeder is names-only, so cmdline is empty — see the OS compatibility matrix below. On macOS this holds on both paths: when the Endpoint Security stream is unavailable the agent falls back to the KERN_PROC_ALL sysctl poll, which also blanks cmdline (only Linux populates it).

JSON export

Use the export action for JSON output suitable for integration with Splunk, ClickHouse, or ELK:

POST /api/v1/instructions/execute
{
  "plugin": "echo",
  "action": "export",
  "parameters": { "type": "process", "limit": "100" }
}

Configuration

Use the configure action to adjust Echo behavior.

Validation rules:

• When both fast_interval and slow_interval are provided, fast_interval must be less than slow_interval.
• redaction_patterns and process_stabilization_exclusions must each be a JSON array of non-empty strings, with at most 256 elements of at most 256 characters each (both feed an O(patterns × command-line-length) scan per process per cycle).
• process_stabilization_exclusions matching is case-insensitive substring, not real glob. Leading and trailing * are stripped; ? and [abc] are treated as literals. The effective substring (after stripping *) must be at least 3 characters — a shorter one (e.g. "a", or "*a*", which strips to "a") would match almost every process and is rejected. Use a longer substring ("-helper", "chrome-helper"). Matching is on the process name only, so a binary renamed to contain an approved substring evades capture — exclusions are a noise-reduction convenience, not a threat-hunting control.
• network_capture_method is validated against the running host’s OS accept-list, not the union across all platforms — e.g. a Linux agent rejects the Windows-only iphlpapi. polling is always accepted (the platform default). Run the compatibility action to see the live accepted list for a given agent.
• Disabling a collector (<source>_enabled=false) atomically stops new captures — the disable runs under the collection lock, so an in-flight cycle either completes before the stop takes effect or observes the disabled flag and skips (no extra snapshot slips through). Existing rows remain queryable. On re-enable the source starts from a clean baseline: the snapshot-diff state is cleared on disable, so the first collection cycle after re-enable emits a started event for every entity currently running/open (an expected one-time rebaseline, not a real-time burst) and never ghost stopped events for entities that exited during the pause. If the agent database is momentarily busy and the baseline cannot be cleared, the disable is refused — the source stays enabled and configure returns an error — so a disabled source can never be left with a stale baseline; retry the disable.
  • The interval samplers (perf and procperf) report the delta between consecutive readings rather than started/stopped events, and keep that previous reading in memory. Disabling them resets that in-memory baseline under the same collection lock, and the sampler re-checks the disabled flag after taking the lock — so disabling them is equally atomic, and the first sample after a re-enable establishes a fresh baseline rather than reporting one row whose averaged rate covers the entire paused window. This matters most for procperf (per-application CPU/memory): no usage row ever spans the period it was disabled.

Example:

POST /api/v1/instructions/execute
{
  "plugin": "echo",
  "action": "configure",
  "parameters": {
    "retention_days": "14",
    "fast_interval": "30",
    "user_enabled": "false",
    "process_stabilization_exclusions": "[\"*-helper*\",\"*ide-language-server*\"]"
  }
}

OS compatibility matrix

Echo runs on Windows, Linux, and macOS, but each capture source has platform-specific constraints. Run the compatibility action to print the live matrix from the agent itself; the table below is a snapshot for documentation purposes.

macOS Endpoint Security development bundle

An opt-in development package can wrap the existing Echo process collector in a signed YuzuAgent.app. It is not a release/notarization lane. Packaging requires an explicit current macOS App ID profile with the ES entitlement set to Boolean true, a matching signer and provisioned Mac, plus an SDK that ships the Endpoint Security header and library stub (both the Command Line Tools and full Xcode SDKs do). The root LaunchDaemon always disables bare-executable OTA updates. External plugins remain subject to Apple library validation and Yuzu final-byte CMS policy. These artifact checks do not establish live ES: verify echo.status reports process_capture_method|endpoint_security, then observe a unique exec/exit pair and check both drop counters. Polling is an honest fallback, never ES proof.

Status values:

• supported — fully wired and exercised in CI.
• constrained — works but with a documented limitation.
• planned — not yet implemented; network_capture_method accepts the value so you can pre-stage configuration.
• unsupported — platform cannot supply the data at all (e.g., service on a kernel without a service manager).

To get the matrix at runtime (returns one row|... line per source/OS pair):

POST /api/v1/instructions/execute
{
  "plugin": "echo",
  "action": "compatibility"
}

Security: command-line redaction

Echo automatically redacts sensitive command-line arguments before storing process events. Any command line matching a redaction pattern has its cmdline field replaced with [REDACTED by Echo]. Redaction applies to command lines only: the registry source’s values are not redacted (see Registry table under Key columns).

Windows + macOS (stream) scope. On Windows the process source is the ETW Kernel-Process feeder; on macOS with the Endpoint Security stream active it is likewise names-only — both capture image names only, never a command line. The cmdline column is therefore empty for those process rows, and these redaction patterns have no effect on them (there is nothing to redact). They still apply to per-app perf rows (procperf, matched against the image name) on all platforms and to Linux process command lines (always poll-captured). On Linux, procperf redaction is best-effort against the kernel’s 15-byte comm — reliable only for a pattern whose sensitive substring is a ≤15-byte, prefix-aligned slice of the comm; a token past byte 15 is truncated away and cannot be matched, and a Windows-style core like outlook.exe does not match the extension-less comm outlook (see the procperf row in the OS compatibility matrix for the full rules and the over-suppression caveat). Patterns are matched in the stored, sanitized spelling: | and control bytes are stored as _ and \/newline as \\/\n, and the operator’s pattern is put through that same transform before matching, so *evil|app* correctly redacts a process that named itself evil|app. Redaction is a cmdline + per-app-perf control only: it never applies to the process source’s name column or to netqual owner-process names — an app redacted out of $ProcPerf_*/app_perf still appears by name in $Process_Live (and $NetQual_Live if enabled), on every OS. macOS is names-only on both paths (ES stream and sysctl poll), so these patterns have nothing to redact there either. If your threat model depends on never storing command-line secrets on Windows/macOS, both the streaming feeder and the macOS poll guarantee that structurally.

Default redaction patterns:

["*password*", "*secret*", "*token*", "*api_key*", "*credential*"]

Patterns use case-insensitive substring matching. Leading and trailing * are optional — they are stripped before matching, so the effective match is always a bare substring (*password*, password*, and password all match any command line containing “password” in any case). Interior ? and [abc] are treated as literals, not glob metacharacters.

The built-in default patterns always apply and cannot be disabled. configure lets you add patterns, but the baseline set (password, secret, token, api_key, credential) is enforced on every collect path regardless of what is stored. If redaction_patterns is set to an array whose elements are all invalid (over-long, non-string) or to [], the built-in defaults are silently restored as the active list rather than leaving redaction disabled — so a corrupt, tampered, or empty config can never cause secrets to be written in plaintext.

To customize:

POST /api/v1/instructions/execute
{
  "plugin": "echo",
  "action": "configure",
  "parameters": {
    "redaction_patterns": "[\"*password*\",\"*secret*\",\"*token*\",\"*private_key*\"]"
  }
}

Checking Echo status

The status action returns database health information:

storage_state|ok
db_health|ok|0
record_count|15234
oldest_timestamp|1710950000
newest_timestamp|1711050423
db_size_bytes|2097152
retention_days|7
config|process_enabled|true
config|process_paused_at|0
config|process_live_rows|18402
config|process_oldest_ts|1710950000
config|tcp_enabled|true
config|tcp_paused_at|0
config|tcp_live_rows|4193
config|tcp_oldest_ts|1710955000
config|service_enabled|true
config|service_paused_at|0
config|service_live_rows|812
config|service_oldest_ts|1710901000
config|user_enabled|true
config|user_paused_at|0
config|user_live_rows|97
config|user_oldest_ts|1710900100
config|module_enabled|true
config|module_paused_at|0
config|module_live_rows|0
config|module_oldest_ts|0
config|software_enabled|true
config|software_paused_at|0
config|software_live_rows|0
config|software_oldest_ts|0
config|arp_enabled|true
config|arp_paused_at|0
config|arp_live_rows|0
config|arp_oldest_ts|0
config|dns_enabled|true
config|dns_paused_at|0
config|dns_live_rows|0
config|dns_oldest_ts|0
retention_guard_declines_total|0
retention_guard_failures_total|0
config|network_capture_method|polling
config|network_capture_method_effective|polling
config|software_interval_seconds|3600
config|software_last_run_ts|1711050000
config|perf_capture_method|procfs
config|procperf_procs_seen|412
config|process_last_status|events_recorded
config|process_consecutive_failures|0
config|process_last_status_at|1711050400
config|tcp_last_status|events_recorded
config|tcp_consecutive_failures|0
config|tcp_last_status_at|1711050400

(The sample is abbreviated: <source>_last_status, <source>_consecutive_failures and <source>_last_status_at appear once per capture source, in registry order.)

Database health (db_health). db_health|ok|<epoch> reports the epoch of the last echo.db quarantine ever recorded on the device (0 = never); db_health|quarantined|<epoch> is reported only by the agent process whose startup quarantined a corrupt database (see “Corrupt-database quarantine” below). The fact is persisted in echo_config (db_health_last_quarantine_epoch / db_health_last_quarantine_file, basename only), so it survives restarts.

Per-source health. <source>_last_status is the terminal outcome token of the source’s most recent collection (never_ran until it has run in this agent process), <source>_consecutive_failures counts back-to-back failures, and <source>_last_status_at is the epoch of that outcome (0 if never). The failure tokens are capture_incomplete, counters_unavailable, cursor_lost, state_unreadable, insert_failed and state_save_failed. state_unreadable means the <source>_enabled flag (or, for software, the stored baseline) could not be read; the tick is skipped and counted as a failure. Every other token (including source_disabled, unsupported_platform and baseline resets) resets the streak. The ledger is process memory only: an agent restart resets every source to never_ran / 0, which is not a recovery. When one source hits a fatal early return in a tick, the sources after it do not run that tick, so read <source>_last_status_at to spot a source that is frozen by an upstream failure rather than healthy. perf_capture_method is the registry capture method for the running OS (procfs on Linux, ntcounters on Windows, none where there is no perf collector), and procperf_procs_seen is the process count in the last valid per-process counter snapshot.

Write failures. Snapshot-diff sources (process, tcp, service, user, software, arp, dns, mapdrive) commit a tick’s events and their baseline in one transaction, so a failed save can never leave events durable against a stale baseline. When a tick that had events fails, the collector emits error|<source> insert failed; when only a baseline-only save fails (no events in the diff, the process fallback seed) it emits error|<source> state_save_failed; the software cold-start seed reports on the software collector’s own line, echo|collect_software|0|state_save_failed. Either way nothing was persisted and the next tick re-derives the same delta. arp, dns and mapdrive are non-fatal: they emit no error| line and record the token in the ledger instead.

A block is emitted for every capture source. Every source reports <source>_enabled|true on a fresh agent — including module, software (both shown above), procperf, netqual, netconn, arp, dns, and mapdrive; a source reads false only after an operator disables it via configure (see the configuration table above). netqual_capture_method reports the per-connection quality mechanism actually in effect — inetdiag (Linux), estats (Windows once the elevation gate has latched active), estats_pending (Windows, netqual enabled but the first collect tick has not yet tested elevation — or netqual is off), or none (Windows non-elevated after the access-denied latch, macOS) — so “enabled but the agent can’t collect” is distinguishable from “off”, and a non-elevated agent never briefly advertises estats before flipping to none. module_live_rows stays 0 until a collector for the host’s OS ships; likewise dns is Windows-only today (planned on Linux/macOS), so on a Linux or macOS agent dns_enabled|true still reports dns_live_rows|0. arp is wired on every OS (Linux via /proc/net/arp; macOS constrained via the NET_RT_FLAGS sysctl route dump — entry_type always unknown there), so arp_enabled|true reports real rows there — run the compatibility action to distinguish a supported-but-empty source from a planned-but-unimplemented one. software_last_run_ts is the wall-clock of the last collect_software tick (0 if it has not run yet); it is the heartbeat that distinguishes “healthy but no software changed” (where software_live_rows stays low) from “the hourly trigger never fired”.

A <source>_enabled value can also read errored — automation that scrapes this output should match three values, not two:

config|process_enabled|errored

errored means the stored value is not the literal true/false the agent ever writes (corruption or tampering); the source is fail-closed until it is re-configured. See the tri-state description below.

retention_guard_declines_total counts retention passes this agent declined (see The retention clock guard for the four triggers and which of them are benign), and retention_guard_failures_total counts tables that could not be retained at all — whether the probes could not be read or the delete itself failed (see “The retention clock guard” below). Both are always emitted. Read them together: a zero declines total only means the clock is behaving if the failures total is also zero — a table whose probes fail every pass has silently stopped being retained, and would otherwise report as healthy. When either is non-zero, one extra line per affected warehouse table names the table and its own count:

retention_guard|process_hourly|3
retention_guard|tcp_hourly|3
retention_guard_failed|module_hourly|2
retention_guard_failed|__clock_state__|1
retention_guard_declines_total|6
retention_guard_failures_total|2

Only tables with a non-zero count appear, so a healthy agent emits just the two totals. __clock_state__ is not a table: it reports that the agent could not persist its own clock reading, which leaves the guard without a comparison point after the next restart. __implausible_now__ is also not a table: it reports that the clock reading handed to a pass (not the stored comparison point) was too far ahead to trust, so the whole pass declined before touching any table — see “The retention clock guard” below. The per-table counters are in-memory and reset when the agent restarts (the clock reading they compare against is persisted, so restarting does not blind the guard). Scrape this across the fleet to find endpoints whose clocks need attention — the agent has no /metrics endpoint, so this action is the fleet-readable signal. The retention-guard counters are not yet fleet-aggregated: there is no shipped instruction or heartbeat tag that rolls these up server-side; surfacing them via the agent heartbeat is a tracked follow-up. (echo.db corruption, by contrast, is fleet-visible: see “Corrupt-database quarantine” below.)

The four <source>_* blocks are emitted per capture source. <source>_enabled is one of three values: true (collector active), false (disabled via configure), or errored. errored means the stored value is not a recognised boolean — configure only ever writes true/false, so an errored value indicates the agent’s echo.db was tampered with or was corrupt and re-initialised (see “Corrupt-database quarantine” below). While an errored value persists the agent applies a fail-closed policy: the affected source stops collecting (it is treated as false, not enabled) and retention skips pruning that source’s rows, so any forensic data already captured is preserved. The source stays paused until you re-issue an explicit configure for it on that device to clear the value. <source>_paused_at is 0 when the source has never been disabled and the wall-clock UTC seconds when it was last transitioned enabled → disabled. The reverse transition resets it to 0 — and this includes recovery from errored: issuing configure <source>_enabled=true on a source whose stored value is errored clears paused_at to 0 in the same transition, so a recovered source never reports enabled=true alongside a stale paused timestamp. <source>_live_rows and <source>_oldest_ts are the count and minimum timestamp of the per-source *_live table at the moment of the status call. Agents older than v0.12.0 do not emit the per-source paused_at / live_rows / oldest_ts lines. In the retention-paused list the dashboard renders a “schema older than server” badge for such an agent’s disabled source (and sorts it as the oldest, at the top of the list) rather than hiding it behind a bare —; elsewhere a missing live_rows / oldest_ts still renders —.

Corrupt-database quarantine

When an agent’s echo.db fails its PRAGMA integrity_check at startup, the agent quarantines the corrupt file rather than trusting it: the database and its -wal/-shm/-journal sidecars are renamed aside to echo.db.corrupt-<epoch> (etc.) in the same directory, a fresh empty database is initialised in its place, and the agent logs echo.db.corruption_detected. echo.db.corruption_detected is an agent-local log line (spdlog, error level); the quarantine itself reaches the server as an echo.db.corruption_quarantined audit row via the heartbeat — see “Fleet visibility” below. Consequences an operator should know:

• All Echo history on that device is reset — the new database is empty; prior events live only in the .corrupt-<epoch> sidecar.
• Per-source enable/disable state is reset to defaults — a source previously paused for forensic preservation is collecting again. After the agent recovers, re-issue any required configure toggles, and status may briefly show errored for a source whose value could not be read.
• The quarantined file is not auto-deleted. Recover data from echo.db.corrupt-<epoch> before the new database’s retention overwrites the device’s storage budget — e.g. sqlite3 echo.db.corrupt-<epoch> ".recover" | sqlite3 recovered.db, or open it read-only with any SQLite tool — then remove the sidecar manually once recovered. Repeated corruption produces multiple timestamped quarantine files; none are pruned automatically, so an agent with a recurring storage fault can accumulate them — watch the data dir’s footprint.

Fleet visibility. A quarantine is reported off-device: the echo plugin publishes heartbeat.db_corruption_total (cumulative per agent, monotonic in the common case) and heartbeat.db_quarantine_last (<epoch>:<file>) in its plugin storage, which the agent forwards on every heartbeat as yuzu.plugin.echo.db_corruption_total / yuzu.plugin.echo.db_quarantine_last. The server exposes yuzu_fleet_echo_db_corruption_agents (see the metrics reference) and writes a echo.db.corruption_quarantined audit row per (agent, quarantine identity), deduplicated against the audit store and bounded to one row per agent per 10 minutes per server process (intermediate quarantines inside that window collapse to the newest identity; another HA node can add a second row for the same identity) — see the audit-log reference. A plugin that failed to initialise (including Echo when the corrupt file cannot be moved aside) is reported in the yuzu.plugins_failed heartbeat tag and yuzu_fleet_plugin_init_failed{plugin}, which distinguishes “failed to load” from “not installed”. The plugin keeps its count in a private storage record; if that record is present but unreadable the plugin logs ECHO: db_health.ledger unreadable; not overwriting and stops updating the two heartbeat keys rather than resetting the count — status still reports db_health, and the agent log carries the warning on every collect_slow tick. If instead the record write itself fails persistently (as opposed to a read of it), there is currently no fleet-visible signal beyond that same local warning — a device stuck this way is silently excluded from yuzu_fleet_echo_db_corruption_agents until the fault clears; tracked as a follow-up. The count is not monotonic in one further edge case: a transient fault on the read that decides whether a prior record exists (rather than the read above) is indistinguishable in-plugin from “no prior record,” so a faulted read followed by a successful write on a later tick restarts the count at 1 for that device rather than resuming it — the fleet gauge counts affected agents, not events, so this narrows to a small undercount rather than a lost signal; tracked as #5037.

Two things worth knowing about the timing of this signal. A mass event (many devices quarantining near-simultaneously, e.g. a bad storage-driver rollout) still updates the fleet gauge in real time, but individual audit rows serialize behind one another server-wide, so the audit trail can lag well behind the gauge during a burst — don’t read row latency as gauge inaccuracy. A device already quarantined before this fleet-visibility mechanism shipped has no durable epoch/basename recorded yet, so it produces no fleet signal until its next corruption event — there is no backfill of pre-existing quarantines.

If echo.db is corrupt and cannot be moved aside (read-only mount, locked file, permissions), the agent fails closed — it refuses to load Echo rather than silently trusting the corrupt database — and logs the reason. Other agent plugins continue running; only Echo is unavailable on that device until the underlying fault is cleared and the agent restarted.

network_capture_method is the configured value (which may be a value pre-staged ahead of a forthcoming kernel-event collector, or a cross-OS platform API); network_capture_method_effective is the mechanism actually collecting, which is always polling today — no kernel-event collector is wired yet, so every configured value currently collects via polling. The two are reported side by side so status can never misrepresent the active capture mechanism to a forensic analyst.

Echo dashboard page

The Yuzu dashboard includes a dedicated Echo page at /echo, reachable from the Echo entry in the sidebar’s Posture group (visible after authentication). The page is the central operator surface for Echo across the fleet. Phase 15.A delivers the retention-paused source list as the first frame; the process tree viewer and the capture sources frame sit below it. The capture sources frame locks the five core sources (process, tcp, service, user, perf), so pause or resume them with configure, and shows each as paused when the agent reports <source>_enabled=false and always on for any other reported value (an errored value still reads always on; the retention-paused list above flags it with its value-error badge). Ad-hoc Echo SQL is not on this page: it starts from Result sets (/result-sets), where From Echo query runs a SELECT on a scope and keeps the devices that answer, behind a confirm that shows the SQL.

Retention-paused source list

The first frame surfaces every device × source pair where the collector has been disabled (<source>_enabled=false). Rows are sorted paused-longest-first so devices accumulating non-aging data the longest float to the top of the list.

Columns: device hostname, source pill, paused since (UTC), paused for (coarse age), live rows count, oldest data age.

Partial result banner (#4644). The scan’s response read is capped at 10,000 rows and 50 MiB of response payload. When either cap cuts it, the page shows a “Partial result” banner above the table, the header counts (“responded”, “have all sources collecting normally”) are marked as describing only the responses that were read, and, if no paused rows are in the part that was read, the empty state says the page cannot tell and that Refresh will not change it (the cut repeats), instead of “still in progress” or “running normally”. Dropped responses read as agents that never answered, so the banner is NOT confirmation that every collector is running normally: narrow the scan to a smaller management group and scan again. A cut scan does not update the yuzu_echo_retention_paused_devices gauge (its partial row set would under-count), so the gauge keeps the last value from an uncut scan and can be stale. The same condition is result_truncated_by_cap: true in GET /api/v1/echo/retention-paused and the list_echo_retention_paused tool, which also set store_degraded: true for a cut scan so an older client sees an incomplete result.

Row states. Beyond a normal paused row (with a timestamp), the table surfaces two conditions and floats both to the top of the list so they aren’t missed:

• “schema older than server” badge — the agent reported the source disabled but sent no paused_at timestamp, i.e. it is a pre-v0.12.0 agent that lacks the field. The row sorts as the oldest entry. Action: upgrade the agent so it records the transition time.
• “value error” badge — the agent reported a <source>_enabled value other than true/false (e.g. errored, or garbage from a corrupt or tampered echo.db); the reported value is shown. Previously such a source was silently omitted, hiding a paused/broken collector. Action: re-configure the source (echo.configure <source>_enabled=true) or inspect the agent’s echo.db; if tampering is suspected, treat the device as potentially compromised.

Workflow:

1. Click Scan fleet to dispatch an echo.status command to the agents in your management-group scope. The scan-provenance header above the table reports how many agents were dispatched to, how many have responded so far, and how many have all sources collecting normally.
2. Review the table. A row with a high live_rows count or a long pause duration may indicate forensic data accumulating without being queried.
3. Click Re-enable on a row to dispatch echo.configure with <source>_enabled=true to that single device. The row drops optimistically; click Refresh to reconcile against a fresh scan.
4. Click Purge data on a row to permanently drop all of that source’s stored rows (<source>_{live,hourly,daily,monthly}) without re-enabling the collector — for “we’ve preserved what we need; reclaim the disk but keep collection paused.” Because this is irreversible you must type the device hostname to confirm. The source stays paused and the row remains (with a “Purge dispatched” note); click Scan fleet to see the counts drop to zero. If the source was re-enabled since your last scan the agent refuses the purge (source_not_paused) and nothing is deleted. If the agent cannot read the source’s enable flag it also refuses (config_unreadable: it could not confirm the source is paused) and nothing is deleted.

Purge is destructive and irreversible. It drops rows for the named source on the named device only — other sources and other devices are untouched. The typed-hostname confirmation is your last checkpoint; there is no undo.

Permissions:

• Viewing the page and the retention-paused list requires Infrastructure:Read.
• Scan fleet requires Execution:Execute (it dispatches a fleet-wide command).
• Re-enable requires Execution:Execute (it dispatches a configure command to a single device).
• Purge data requires Infrastructure:Delete — a higher tier than Re-enable, because it is destructive. It is management-group scope-checked like Re-enable (out-of-scope device_id collapses to the same 404). The “only purge a paused source” rule is enforced on the agent (it refuses with source_not_paused if the source is enabled, or config_unreadable if the enable flag cannot be read so it cannot confirm the source is paused), so a source re-enabled between your scan and the purge is never dropped. Requires an agent that supports the echo.purge_source action (v0.14.0+); an older agent silently no-ops the purge — verify with a fresh Scan fleet. For the registry source the purge also erases its per-key baselines in echo_state and its cursor.
• The per-device Re-enable and Purge data endpoints are management-group scope-checked — it rejects out-of-scope device_id values with the same 404 response as a not-connected agent (no enumeration oracle). The Scan dispatch and the rendered list, however, currently gate only on the global Execution:Execute / Infrastructure:Read permission: management-group confinement of these list/fan-out views is not yet effective (tracked as the ADR-0017 admit-then-filter gate) — under RBAC a confined operator is denied at the global gate rather than shown a narrowed list. When RBAC is disabled (the default), reads return the full enrolled fleet; when RBAC is enabled, list views require the global permission and per-device routes additionally enforce your management-group role assignments.

State persistence: Scan results are held in the server’s memory keyed by your username. Restarting the server clears the last-scan reference; click Scan fleet again after a restart. Persistence across restarts and multi-server coordination are planned for Phase 15.G operational hardening.

Audit trail: Every Scan emits an echo.status.scan audit event. Every Re-enable emits echo.source.reenable (with result=success and detail carrying device_id and source on success, or result=failure with detail carrying the real reason — scope_violation or agent_not_connected — on rejected attempts). Every Purge data emits echo.source.purge (same detail shape; result=denied if the Infrastructure:Delete permission is missing). Because purge dispatch is fire-and-forget, the number of rows deleted is recorded in the command’s response record, not the dispatch audit row. See docs/user-manual/audit-log.md for the full schema.

Process tree viewer

The third frame on the /echo page reconstructs a per-host process tree entirely from that host’s local Echo warehouse ($Process_Live + $TCP_Live) — no extra data is collected, and no other host is involved.

Workflow:

1. Pick a live host from the dropdown (only connected agents in your scope are listed). Selecting one reconstructs the default Last 10m window.
2. Choose a timescale — the preset chips On boot · On agent install · Last minute · Last 10m · Last hour · Last day, or pick or type a custom From/To (UTC) range and click Apply. Setting From == To gives a true point-in-time tree. A custom From or To that is filled in must be a valid UTC date and time (YYYY-MM-DDTHH:MM or YYYY-MM-DDTHH:MM:SS) or, in a hand-built request, epoch seconds (up to 18 digits). Anything else, including a date that does not exist such as 31 February, is refused with a note (“The custom window is not a valid time”) and nothing is dispatched to the device.
3. Read the tree. Each row shows a running/exited dot, PID, name, owning user, and — when the process has connections — an inline network summary of remote IP:port endpoints (public/internet egress is highlighted). Dozens of identical-name siblings (e.g. svchost.exe) collapse into one name ×N row you can expand.
4. Click any process to open the detail panel on the right: path, command line, user, start time, full connection list, and any anomaly evidence.
5. Filter with the toolbar: All / Running / Exited, an Anomalies only toggle, and a text box that matches name, PID, or remote IP. Filters combine and apply instantly (no reload).

What “anomalies” means here: the viewer flags suspicious parent→child spawns — a common-document or browser application (Word, Excel, Outlook, Chrome, Edge, …) launching a shell or LOLBin (powershell.exe, cmd.exe, mshta.exe, rundll32.exe, …). This is computed on the server from the Echo data; it is heuristic, name-based, and deliberately conservative.

Honest limitations:

• No seed. The tree is replayed from the retained $Process_Live events only. A process whose started event has aged out of the live-tier cap (or that started before the oldest retained row) may not appear; the banner states the observation window.
• On boot / On agent install are proxies. Echo stores no boot or install timestamp, so these anchors are derived from the retained events (install ≈ oldest retained row; boot ≈ the most recent root-process start).
• Windows is names-only. On Windows the process feeder is ETW (Kernel-Process), which captures image names only — so per-process path and command line are blank on Windows (they are populated on Linux/macOS). Loaded libraries/DLLs are not captured by Echo on any platform.

Permissions: viewing the frame requires Infrastructure:Read. Reconstructing a tree dispatches a read-only echo.sql to the device, so it additionally requires Execution:Execute and the device must be inside your management scope. The access is audited as echo.process_tree.read — once when the query is dispatched (recorded even if the device is offline) and again on a successful reconstruction (recording the time window, node/anomaly counts, the OS data-class, and whether connection data was shown). The device is the audit event’s target.

Forcing an immediate snapshot

The snapshot action triggers an immediate full collection of all enabled capture sources (including arp/dns), useful before a maintenance window or at the start of an investigation:

POST /api/v1/instructions/execute
{
  "plugin": "echo",
  "action": "snapshot"
}

Partial captures never fabricate events. If a subprocess-backed leg (systemctl/launchctl/smbstatus/journalctl/wevtutil) or a procfs/getfsstat read fails or is truncated mid-capture, that tick’s diff is skipped and the prior baseline is retained — no appeared/removed events are fabricated from a partial read. A rate-limited warning is logged and the source’s <source>_last_status reads capture_incomplete with <source>_consecutive_failures counting the streak in echo.status.

Performance impact

Echo is designed for minimal performance overhead:

• Fast collection (processes + network): typically completes in under 100ms
• Slow collection (services + users): typically completes in under 200ms
• Performance sampling: a handful of kernel-counter reads every 30s; one row written per sample. The $Perf_Live 7-day window holds ~20,000 rows (~1-2 MB) per endpoint at the default cadence.
• Database size: varies by system activity; a typical endpoint generates 1-5 MB per day (plus the ~1-2 MB perf window)
• CPU: negligible between collection cycles; brief spike during snapshot + diff
• Automatic purge: old events are removed on the rollup tick (every 900 s). Each tier’s window comes from the schema registry’s per-granularity default, not from the retention_days configure key — that key is stored and reported by status but is not what the warehouse tiers prune against. The per-tier defaults are published by GET /api/v1/settings/data-retention (echo.tiers) and shown on Settings > Data and retention (the live tier under the Exact column) with a “Look again after” floor per source.

The retention clock guard

Time-based retention deletes rows older than now - <tier retention>, where now is read from the endpoint’s own clock — the clock in a fleet most likely to be wrong. A dead CMOS battery, a long suspend, a cloned VM, or a boot before NTP converges all produce a reading that marks every row in a warehouse table expired at once, and an unguarded delete then takes the whole forensic window with it.

A retention pass first refuses to act at all if the clock reading handed to it is implausible (too far ahead to be trusted into arithmetic) — the whole pass declines before anything is persisted or any table is examined, and retention resumes once a sane reading arrives.

Per table, the pass then either declines (deletes nothing from that table, counts the decline) or proceeds, based on the same fact set each time — the combination of which of these four conditions hold for this table on this pass:

1. it would delete every datable row of that table;
2. more than a fixed 30 days elapsed since the previous pass;
3. the stored reading is ahead of the current clock;
4. there is no stored reading at all — the first pass after an agent upgrade or a restore.

A table declines whenever this pass’s fact set differs from what was last reported for it. An identical repeat is a suppressed, capped drain, so a warehouse that is legitimately all-expired still ages out — at the cost of one rollup tick (900 s) per distinct anomaly, not one per pass. A different anomaly arriving while one is still being worked off (e.g. the table starts wiping while a corrupt-reading decline is still active) reports again rather than deleting silently. Trigger 4 is the one exception: it is never recorded, so a table that keeps failing to persist its own comparison point declines every pass rather than being suppressed after the first — a missing comparison point is not an anomaly whose fact set can safely repeat unreported.

What to do: for 4, nothing. The reading is a single echo_config key shared by every source and table, so it is expected once per agent — but note EVERY enabled time-based table declines in that one pass, so the total jumps by the table count, not by 1. It recurs only if the agent cannot WRITE that key at all (read-only or full disk), which is the safe direction and is what retention_guard_failures_total is for. For 2 and 3, check the endpoint’s clock and its uptime; see the runbook for the audit twin of this triage.

The 30 days is absolute, not derived from the tier’s retention window: how far the clock moved has nothing to do with how long that tier keeps rows, and scaling it to the window put the threshold at a year on the monthly tier, where it could never fire. Elapsed time cannot tell a jump from a dark laptop, so the threshold sits past the point where an outage is itself remarkable — a laptop off over a long weekend reports nothing; one off for a month declines one tick. The reading is persisted in echo_config, so trigger 2 still fires on the first pass after an agent restart, including one that booted with a wrong RTC.

• Every accepted pass deletes at most 5,000 rows per table, oldest first (~480k/day/table at the 900 s cadence, far above any endpoint’s growth rate). A wipe the guard chose to allow ages out at a paced rate rather than in one statement.
• Rows timestamped implausibly far in the future — more than a day ahead, i.e. written while the clock was already skewed forward — are excluded from the “would this delete everything?” question. Without that, one such row would disarm the guard permanently.

Row-count retention is deliberately unguarded, but it is capped. The tiers that trim to a fixed row ceiling do so with no clock involved, so no clock reading can make them delete more than they always would — hence no guard and no decline. They ARE capped at the same 5,000 rows per table per pass, for a different reason: the whole retention batch runs under one held database lock, and an uncapped prune over a large excess would stall every collector on the endpoint. A big backlog now drains over a few rollup ticks instead.

A paused or errored source is skipped before the guard is consulted, so it neither deletes nor records a decline — an operator who paused a source for forensics never sees a clock-anomaly signal from it.

What this does and does not promise. The cap is the half that always applies: it bounds every delete unconditionally. The detectors are best-effort, and the outcome test in particular is defeated by any row written after the clock moved — which do_rollup reliably produces, since run_aggregation runs first and mints rows into the very tables retention then reads. That is why the elapsed-time check is persisted. Taken together the guard converts an instantaneous wipe of the device’s forensic window into a paced one plus a counter; it does not guarantee every clock anomaly is detected, and a persistently wrong clock will still drain the window over several ticks.

The operator surface is retention_guard_declines_total and retention_guard_failures_total (plus the per-table retention_guard|<table>|<n> and retention_guard_failed|<table>|<n> lines) in the status action, described above. A non-zero declines total has four possible causes, listed in The retention clock guard. One of them is benign and expected after an agent upgrade; the rest mean the endpoint’s clock moved, it was dark longer than the threshold, or the stored reading was ahead of the clock. Elapsed time cannot separate those, so check the host’s time synchronisation and its uptime history. A non-zero failures total means retention has stopped for that table for a reason that is not the clock: either its probes could not be read, or its delete failed. What happens to the REST of the pass depends on the error. One that aborts the transaction itself (a disk-full, an I/O error) rolls the whole pass back, and every table queued in it is reported — including row-count tables, which are otherwise outside the guard. An error that leaves the transaction intact — one table with a corrupt index, say — fails only that table, and the healthy tables in the same pass still have their deletions committed. That distinction exists because stopping on every error meant one permanently-broken table halted retention for the whole endpoint, for ever.

If that rollback ITSELF fails and the database is left inside the transaction, the agent closes the Echo database. Every later write on a connection stuck in an uncommittable transaction would be reported durable and then lost at restart, so it fails all of them closed instead. echo status then returns non-zero and emits an error| line followed by storage_state|offline, and nothing else — every config| line is withheld. Collection and retention are both stopped on that endpoint until the agent restarts. Historical data normally stays readable: the close affects only the read-write connection, so echo sql can still query what was already collected. The error line says which of those two cases you are in — the read-only connection is itself optional at open time, and on the rare endpoint where it never opened, echo sql cannot read the history either, so the line says that instead of promising a read path that is not there. This needs a disk-level fault to reach, but read storage_state before trusting any other line in that output.

In the dashboard, the capture-sources frame surfaces that error line verbatim rather than a generic failure, so the reason and the recovery advice reach the operator on the frame they opened to ask why the device’s data is missing.

Known dead band. Because the threshold is a fixed 30 days, a forward clock error smaller than that trips neither detector on a table whose own window is shorter: the step check is below its floor, and the outcome test is separately defeated by the rollup minting fresh rows before retention runs. In that band the table drains at the capped rate with no decline and no counter. The cap still bounds the damage; the detection does not fire. This is the deliberate cost of not reporting every switched-off laptop as a clock anomaly. Retention resumes automatically, paced, once the condition clears.

Upgrade note (every capture source is on by default; one-time reset of stored opt-outs). At that release all 16 capture sources then registered shipped enabled on every OS (every source added since ships enabled too). Eight sources were previously off by default: procperf, netqual, module, software, arp, dns, netconn and mapdrive.

Before you upgrade, if you paused any of the eight on purpose (for example to preserve evidence): run echo.status on that agent and record its config|<source>_enabled|false lines (and the matching config|<source>_paused_at|<epoch> lines), and export what you need with echo.sql, which reaches every table and every tier (echo.query and echo.export read only the live tables, and of these eight only software, arp, dns and mapdrive have one). The reset re-enables the source, and retention skips only disabled sources, so the exemption ends with it: from the first rollup pass after the upgrade (rollups run every 15 minutes) rows older than their tier cutoff start being pruned, and that cannot be undone. After the upgrade, re-apply <source>_enabled=false with echo.configure for each source you recorded to pause it again: that stops collection and exempts the rows that remain from pruning, but rows already pruned do not come back.

What the upgrade does. On the first start after upgrade each agent migrates its echo.db to schema version 8 once: any of those eight sources whose stored <source>_enabled value is false — including a value an operator set explicitly — is reset to true and its <source>_paused_at marker is cleared, in a single transaction. A value of errored (any stored value other than exactly true or false), and the stored values of the eight sources that were already on (process, tcp, service, user, perf, power, removable, usage), are left untouched. The reset runs only once: an opt-out you write after the migration has succeeded (<source>_enabled=false via configure) sticks across restarts. A failed migration is retried on the next start, and a disable written before it succeeds is reset by the retry.

If the migration fails. The success line in the agent log is EchoDatabase: migrated to schema version 8; a failure logs EchoDatabase: v8 migration failed at followed by the failing step, rolls the whole migration back, and retries it on the next start. Until a start succeeds the old stored values stay in force, and a <source>_enabled=false written for one of the eight in that window is reset by the retry: look for the success line before relying on a disable.

What to expect. echo.status reads <source>_enabled|true for all eight. New collection starts without operator action, on each OS where the source has a collector (see the OS compatibility matrix), for procperf (which feeds the daily app-performance sync), netqual, module (Windows ETW), software, arp, dns, netconn and mapdrive (netconn and mapdrive also run retrospective backfills bounded by netconn_lookback_seconds and the registry/event-log history). A source whose <source>_enabled row cannot be read at all is treated as disabled for that tick (fail-closed) and its stored rows are preserved.

Upgrade note (device perf sampling; per-app sampling is opt-in). (Superseded: every capture source now ships on by default — see the v8 note above.) On upgrade to this release, every Windows agent continues device performance sampling (perf_enabled defaults to true, 30-second cadence — unchanged from the prior release). Linux agents BEGIN device performance sampling on upgrade to this release: the Linux perf collector (procfs) is new, and because perf_enabled is default-ON, $Perf_Live rows start accruing on Linux hosts at the same 30-second cadence with no operator action — a real, operator-visible data-collection change on Linux fleets (set perf_enabled=false per host to opt out; the data is device-level with no user identity, the same class as the Windows rows). Per-application sampling (procperf_enabled, now implemented on Windows and Linux) is a new, distinct telemetry category — per-app CPU/working-set by image name — and ships off by default: it is not collected until an operator opts in, because it is usage-class data subject to works-council/DPA review (see docs/enterprise-readiness-soc2-first-customer.md). To enable it, set procperf_enabled=true via the configure action (fleet-wide or per-device). The same applies to netqual_enabled (per-connection TCP quality, Linux + Windows) and module_enabled (image-load capture) — both ship off by default. On upgrade to this release echo.status now correctly reports these opt-in sources as <source>_enabled|false on an agent that has never set them (a previous release misreported them as true even though nothing was being collected); if you parse echo.status to inventory active sources, expect module_enabled, procperf_enabled, and netqual_enabled to read false unless you have explicitly opted in. No data collection changes on upgrade from these opt-in sources (the one collection change this release does make is Linux device perf starting automatically — see the start of this note). Warehouse tables added by a new release are now created on every database open (previously a pre-existing echo.db missed tables introduced after it was first created), so no manual table-creation step is needed on upgrade.

arp_enabled / dns_enabled are new this release (ADR-0015) and also ship off by default. Expect both to read false in echo.status on upgrade; the new $ARP_* / $DNS_* warehouse tables are created automatically and stay empty until you opt in. ARP now collects on Windows, Linux, and macOS as of this release (previously Windows-only) — if you already set arp_enabled=true fleet-wide (e.g. while only Windows agents could collect), upgrading Linux/macOS agents starts them collecting neighbour IP/MAC records (plus interface on Linux; macOS has no interface field) at the next fast_interval tick, with no separate notice. Set arp_enabled=false before upgrading any endpoint where that new collection is not yet approved, then re-enable once it is. DNS remains Windows-only, so there is no DNS collection change on upgrade. GUI note: after enabling a source (Capture-sources frame or configure), the first rows appear at the next fast_interval tick (default 60 s) — an empty panel immediately after enabling is expected, not a fault.

Upgrade note (new software source — OFF by default). (Superseded: every capture source now ships on by default — see the v8 note above.) This release adds the software install/uninstall source, which ships off by default (opt-in) — a cautious posture for a new capture source. On Windows it captures machine-wide inventory (HKLM Uninstall — asset-management / vulnerability-relevance data, machine scope only, with no user identity / no PII, like Services and User sessions). On upgrade, echo.status gains a config|software_* block per agent (Windows: live values; Linux/macOS: zero rows, collector planned) plus the global software_interval_seconds and software_last_run_ts lines. Automation that parses echo.status by field count or terminal-field detection must be updated to tolerate these additional lines. The source is off by default, so opt in per host with software_enabled=true. See docs/enterprise-readiness-soc2-first-customer.md for the data-handling classification.

Upgrade note (Windows process capture → ETW). BREAKING for cmdline consumers. On upgrade, the Windows process source switches from the 60-second snapshot-diff poll to an event-driven ETW Kernel-Process feeder. Three operator-visible changes:

• cmdline is now empty for Windows process rows (the feeder is names-only). Any dashboard, SIEM export, or Guardian rule that relied on the Windows process command line will see an empty field after upgrade. Linux is unchanged (the poll still captures command lines there). macOS is now names-only on both paths — the Endpoint Security stream and the sysctl poll — so cmdline is empty there too (the poll blanks the proc_pidpath image it would otherwise store). This is intentional (works-council / data-minimization posture) and not reversible by configuration — see the redaction section.
• Live capture is active from the next agent start — gap-free during the live session, so short-lived processes the poll missed now appear, and $Process_Live holds more rows (cap raised to 100000). (One narrow seam: at the boot→live handoff, events in the window between the agent sampling its boot/live boundary and the live provider becoming active can fall in neither the backfill nor the live stream — tracked as a follow-up.) If the ETW session cannot start, the agent logs the reason and falls back to the toolhelp32 poll automatically; if the session later dies it self-heals to the poll. On Windows the death signal is immediate (the ETW ProcessTrace returns). On macOS the Endpoint Security client exposes no liveness API, so the agent treats a prolonged TOTAL silence (no exec/exit for ~1h) as presumed-dead and re-arms the poll then — the threshold is deliberately long because a NOTIFY-only client cannot distinguish a dead stream from a legitimately quiet host, and a false trip drops a healthy stream to the inferior poll. The active path is reported by the status action as process_capture_method (etw or polling on Windows; endpoint_security or polling on macOS). Once it has fallen back to the poll, stream capture is not re-established until the agent restarts (so process_capture_method=polling on a host that should be streaming indicates a prior session failure — restart the agent to retry). Two drop counters: process_stream_dropped is the userspace ring-overflow count (the drain tick fell behind; renamed from process_etw_dropped this release, now cross-platform), and process_stream_kernel_dropped is the kernel/provider-side drop count (Endpoint Security seq_num gaps on macOS; 0 on Windows ETW, which exposes no per-message sequence here).
• macOS parity gap (fork-without-exec). The Endpoint Security stream subscribes NOTIFY_EXEC/NOTIFY_EXIT only, so a child that forks but never execs is invisible until it exits (Windows ETW fires on every process create). Capturing it via NOTIFY_FORK is deferred (#1455) — adding it naively would double-count the overwhelmingly common fork→exec case.
• Boot-window backfill requires the boot AutoLogger, configured by the production InnoSetup installer (scoped to the advanced component that ships echo.dll) and by the developer install path (install-agent-user.ps1). A reboot is required for the AutoLogger to take effect: on a fresh install, or when upgrading from a release that did not configure it, boot-window backfill (the narrow window of processes that start and exit before the agent’s session opens) is absent until after the next reboot. A minimal install that omits the advanced component does not configure the AutoLogger. Boot-window events are names-only with no user.

To disable Windows process capture entirely (ETW and poll), set process_enabled=false via configure; there is no separate “disable ETW but keep polling” switch today. While disabled, the live ETW session keeps running but its buffered events are drained-and-discarded each cycle, so no process activity from the paused window is persisted when you re-enable. As with every snapshot-diff source, re-enabling process_enabled clears the stored baseline, so the first cycle after re-enable does not emit ghost stopped events for processes that exited during the pause (it re-bases on the live process set).

• WAL mode: SQLite Write-Ahead Logging ensures reads never block writes

Warehouse Query System

Echo includes a typed data warehouse that replaces the legacy flat tar_events table with structured, tiered tables optimized for different query time horizons.

Warehouse tables

Table names use $-prefixed identifiers (e.g., $Process_Live) which the agent translates to real SQLite table names at execution time. Each capture source has multiple granularity tiers:

The $Module_* tables are registered and queryable. On Windows the module source (M2, ETW) fills them with image loads; on Linux and macOS they return zero rows until the OS-specific collector ships (M4/M5 macOS Endpoint Security, M6 Linux auditd). See echo-module-loads.md for the ladder.

The $Software_* tables are populated on Windows when the source is enabled (the default; disable with software_enabled=false); on Linux/macOS they are registered and queryable but empty until those collectors ship.

• Live tables hold the most recent raw events with a row cap (oldest rows are evicted): 5000 rows for TCP / Service / User, and 100000 rows for $Process_Live — raised because the Windows ETW feeder is event-driven (every start/stop, including short-lived processes) and fills a 5000-row window far faster than the old 60-second poll did, so a larger raw window keeps a meaningful history before rows roll up into the hourly/daily/monthly count tiers. Exception: $Perf_Live and $ProcPerf_Live are time-based (7 days), not row-capped — a fixed-cadence sampler keeps a time window, so raising perf_interval_seconds must not shrink the history it covers.
• Hourly tables aggregate counts and summaries per hour, retained for 24 hours (perf/procperf hourly: 31 days).
• Daily tables aggregate per day, retained for 31 days.
• Monthly tables aggregate per month, retained for 12 months.
• Service only has live and hourly tiers. User only has live and daily tiers. Perf and ProcPerf have live and hourly tiers. Software has live, daily, and monthly tiers (no hourly — installs are infrequent).

Key columns by table type

Process tables: ts, name, pid, ppid, cmdline, user, action (started/stopped). Aggregated tiers add start_count, stop_count. cmdline is empty on Windows (ETW) and on macOS (names-only feeders — see the OS compatibility matrix and the command-line redaction section); it is populated on Linux only. macOS is names-only on both paths (the Endpoint Security stream and the sysctl-poll fallback).

TCP tables: ts, process_name, pid, remote_addr, remote_port, local_port, proto, state. Aggregated tiers add connect_count.

Service tables: ts, name, status, prev_status, action (started/stopped/state_changed). Hourly tier adds change_count.

User tables: ts, user, domain, logon_type, action (login/logout). Daily tier adds login_count.

Perf tables: ts, cpu_pct, mem_used_pct, commit_pct, disk_read_bps, disk_write_bps, disk_read_lat_us, disk_write_lat_us, net_rx_bps, net_tx_bps (all numeric; rates are bytes/sec, latencies are µs per I/O). $Perf_Hourly carries per-hour samples, cpu_avg/cpu_max, mem_avg/mem_max, commit_avg, and avg/max throughput and latency columns. Collection is trigger-driven (echo.collect_perf, every perf_interval_seconds), so the audit trail for perf is the configure action that enables/paces it, not a per-sample dispatch record.

collect_perf status tokens. sample_recorded, baseline (first tick after start), source_disabled, unsupported_platform (no perf collector for this OS: macOS today), counters_unavailable (the OS has a perf collector but the counter read failed; the same token is used for procperf) and state_unreadable (the <source>_enabled flag could not be read; the tick is skipped and counted as a failure).

ProcPerf tables: ts, name (image name only — never a command line), instances, cpu_pct, ws_bytes. Each tick records the top 10 applications by CPU plus the top 10 by working set (union, ≤ 20 rows), aggregated across same-name processes (instances = how many). cpu_pct is the app’s share of total machine capacity — one saturated core on an 8-core box reads 12.5, matching $Perf_Live and Task Manager. $ProcPerf_Hourly aggregates per (hour, name): samples, instances_max, cpu_avg/cpu_max, ws_avg_bytes/ws_max_bytes. Apps matching a redaction pattern are never recorded. Example — yesterday’s CPU-hungriest apps on a device:

SELECT name, MAX(cpu_max) AS peak, AVG(cpu_avg) AS typical
FROM $ProcPerf_Hourly GROUP BY name ORDER BY peak DESC LIMIT 10

Software tables: $Software_Live carries ts, snapshot_id, action (installed/removed/upgraded), name, version, prev_version (populated only for upgraded), publisher, install_date — machine scope only, no user identity / no PII. The daily/monthly tiers carry day_ts (or month_ts), name, install_count, remove_count, upgrade_count. Example — everything installed or removed on a device in the last 30 days:

SELECT ts, action, name, version, prev_version, publisher
FROM $Software_Live
WHERE action IN ('installed','removed','upgraded')
ORDER BY ts DESC

NetQual tables (ADR-0020): $NetQual_Live carries ts, snapshot_id, proto (tcp/tcp6), remote_bucket (the only destination field — a coarse class: loopback/private/public/unknown; the raw address is never stored), process_name (image name only), rtt_us, rtt_var_us (jitter), lost (current-conditions loss gauge — the sort key), retrans, segs_out, ca_state (0=Open … 4=Loss). $NetQual_Boot is the per-boot retrospective baseline: ts (agent start), snapshot_id, boot_ts, window_s (the pre-Echo window length this boot), retrans_segs, segs_out, estab_resets (since-boot TCP totals), and if_in_errors/if_in_discards/if_out_errors/if_out_discards/if_in_octets/if_out_octets (non-loopback interface totals). Since-boot totals are coarse context, never a current-loss verdict.

NetConn table (ADR-0020): $NetConn_Live carries ts (the event’s own timestamp — backfilled rows predate Echo), snapshot_id, action (connected/disconnected/wifi_connected/wifi_connect_failed/wifi_disconnected/capability_changed), channel (networkprofile/ncsi/wlan), category (public/private/domain, or empty), capability (none/local/internet, or empty), iface_kind (wifi or empty), reason_code (WLAN disconnect/fail reason or NCSI change reason). Every text column is a closed enum token — no SSID, BSSID, profile name, GUID, or MAC is ever stored. Example — connectivity/Wi-Fi drops on a device, including before Echo was enabled:

SELECT ts, action, channel, capability, reason_code
FROM $NetConn_Live
WHERE action IN ('disconnected','wifi_disconnected','wifi_connect_failed','capability_changed')
ORDER BY ts DESC

MapDrive tables (capability-map §3.8): $MapDrive_Live carries ts, snapshot_id, action (appeared/removed for live snapshot-diff, historical for the init backfill), direction (outbound/inbound), local_mount, remote_path, remote_host, username, provider, and origin (live/historical). The origin column separates historically-inferred mappings (seeded once from the registry / fstab / event logs, carrying the artifact’s timestamp or 0) from live-observed ones; a currently-mapped persistent drive legitimately appears as both a historical and a live appeared row. $MapDrive_Hourly carries hour_ts, direction, local_mount, remote_path, remote_host, appear_count, remove_count. Example — every previously and currently mapped drive, both directions:

SELECT ts, direction, local_mount, remote_path, remote_host, username, origin, action
FROM $MapDrive_Live
ORDER BY ts DESC

Registry table (Windows): $Registry_Live carries ts (unix seconds), snapshot_id, hive (HKLM or HKU), key_path (below the hive; HKU paths lead with the user SID), value_name (empty for key_removed and gaps), action (added, changed, removed, key_removed, capture_gap), the NEW value as data_type/data/data_size/data_truncated, the OLD value as prev_data_type/prev_data/prev_data_size/prev_data_truncated, actor_pid, actor_image, source, observed_at (unix ms of the read that saw the change), detail and record_key. actor_pid and actor_image name the process correlated to the change (see The actor is correlated, not verified below) and are 0 and empty when none is named. detail holds the attribution marker (attr:...) on a change row and the explanation on a capture_gap row. A column that does not apply is empty or 0; an empty data_type is what tells an absent value (a removal) from a value that is present and empty. data follows registry-plugin rendering: strings as UTF-8, REG_MULTI_SZ joined with newlines, REG_DWORD/REG_QWORD in decimal, anything else as lowercase hex, capped at 4096 bytes of UTF-8 text (data_truncated = 1 when cut). source is notify when the key watch fired and reconcile when the full re-read (every 12 slow ticks, hourly at the default 300 s) found the change. A deletion is recorded as one removed row per value followed by one key_removed row for the key.

• Values are stored unmasked. Any value under a watched key, changed or not, is readable through echo.sql: the per-key baselines Echo keeps in echo_state (registry:<root>, at most 512 bytes of text per value) hold every current value from the first read on — a Winlogon autologon DefaultPassword is an example — and $Registry_Live holds the old and new data of every change. A change that is still waiting for its writer to settle is held in the source’s cursor row (echo_cursor) with its old and new data, which is readable through echo.sql until the next slow tick at most. A change held when the source is disabled stays in that row until the source is re-enabled (it is then written attr:lost-restart with its original observed_at) or until echo.purge_source runs on the paused registry source. All of it is erased together by echo.purge_source on the paused registry source.
• The actor is correlated, not verified. The key watch finds what changed. A write-only Kernel-Registry ETW session (value writes, value deletions and key deletions; never key opens or creates) reports which process wrote a value, but its events carry the value name, type and size and not the key. Echo therefore matches each change to the writes of the same value name, type and size inside the time window in which the change must have happened. When exactly one process made such a write, actor_pid and actor_image name it and detail is attr:corr (attr:corr;events:<n> when that process wrote the value n times in the window). That is a lead to verify against $Process_Live, not proof: a change made without a value-write event (a hive load, a restore, a transacted commit outside the window) can be correlated to an unrelated process that wrote the same name, type and size in that window. So can the removed row of a value in a deleted key: deleting a key produces one key-deletion event and none for its values, so each such row can be correlated to an unrelated process that deleted a value of the same name in the window (the key_removed row itself never names a process: it is attr:none when the session is up and no loss applies, and attr:off, attr:degraded and attr:lost-restart take precedence). So can a change whose own write event reached Echo more than about 5 seconds after the window closed without any counted loss (a starved consumer, a suspend and resume). Any local process can force attr:ambiguous on a change by writing a value of the same name, type and size anywhere in the window, because events carry no key name; the contract then names nobody. A standard user can likewise push their own change to attr:degraded by writing values fast enough inside one tick to overflow the event buffer past the events of that change’s window (a gap:loss row records it; the change row is still written). actor_image is the name of the process that held the pid when the event was delivered, up to about a second after the write: it is empty when that process had already exited or a newer process held the pid, and under pid reuse it can name a different process for up to five minutes. Every other outcome is marked rather than guessed, and actor_pid is 0 with an empty actor_image whenever no process is named; 0 means “not named”, never “the system made the change”. detail says why: attr:ambiguous:<n> (n processes wrote a matching value in the window; none is named), attr:none (the session is up, no loss applies and no matching write was seen; a key_removed row is attr:none on those terms whatever was written, because a key deletion carries no value name to match), attr:degraded (write events were lost during the window, see gap:loss), attr:off (the ETW session is not running: it needs an elevated agent, and without one Echo keeps the key watch alone), attr:lost-restart (the change was seen before an agent restart, or before the source was disabled and re-enabled, which discards the events that would have named its writer). When several apply, attr:lost-restart wins over attr:off, which wins over attr:degraded, which wins over the correlation outcome (attr:corr, attr:ambiguous, attr:none).
• What is not recorded. A value written and set back between two reads leaves no net change and no row. A change under a key outside the curated list is invisible. A user hive that is not loaded (logged-off user) is not read. A root whose first read hits the read budget (registry_roots_capped) is baselined from the part that was read, so the next complete read reports the remainder as added once. A value larger than 4 MiB is recorded by type and size only (data empty, data_truncated = 1, data_size the full size); a change that keeps its size is not detected. A root read for the first time is baselined silently: a user hive loaded at logon, or a user who logged off and back on, has its Run, RunOnce, Policies and Winlogon roots baselined at their first read, so anything written to them between the hive loading and that read (up to one slow_interval) is part of the baseline and is not a row. A watched key that is itself a symbolic link is recorded as the link (its link target value), and what the link points at is not followed, so a user who replaces their own Run key with a link to another key of their hive hides later writes under the target. A child key this agent is denied read access to (for example a key a user created under their own Run or Policies key with a restrictive ACL) hides only itself, up to 32 such children in one root: the rest of its root is read as usual, the values stored for it earlier are kept, and deletions beneath it are not derived until it can be read again; the 33rd makes the read partial, so no deletion is derived in that root until a read sees fewer. A child that becomes denied after the root was last read is found at the root’s next read, which is the full re-read (every 12 slow ticks, hourly at the default 300 s) at the latest, because the watch reports added, removed and written keys and values but not permission changes. capture_gap rows (gap:restart, gap:disabled, gap:cursorlost, gap:baselinelost:<root>, gap:unreadable:<root>, gap:loss:<n>) mark the stretches where capture was interrupted, a baseline was lost, part of a root could not be read, or write events were lost: a gap:unreadable:<root> row is written when a root cannot be read (the root itself, or any key beneath it other than an access-denied child), and when a child key under a root is newly denied (the set of denied child keys gains a member; its detail names the first few); only an access-denied child is isolated, and every other read failure beneath a root makes the whole root unreadable. Nothing is written while the situation holds steady, shrinks or clears. A gap:loss:<n> row is written when n write events were lost since the previous tick (the event buffer overflowed, the kernel session dropped events, an event could not be decoded, or the session could not be queried); every change whose time window overlaps the loss is then written attr:degraded instead of naming a process.
• Timing. A change is read at the slow tick that sees it. Its row is written in that same tick once its writer is settled, which means the write events for its time window have all arrived (they can trail the write by a few seconds). A change whose window closed less than about 5 seconds before the tick is held and written at the next tick instead, and so is every change found without a key-watch notify (a root with no live watch, or the hourly full re-read), because its window runs to the end of the read. The exception is a change whose marker cannot depend on a later event (attr:off, attr:lost-restart): it is written in the tick that sees it, so after an agent restart the whole re-read is written at once. A row therefore lands within two slow_intervals of the change (600 seconds by default), usually within one. observed_at is when the read saw it, not when it was written to the registry.
• Health. echo.status reports registry_capture_method (notify+etw while the ETW session runs, notify when it does not, none before the first tick), registry_watch_roots, registry_watch_roots_unarmed, registry_roots_unreadable, registry_roots_capped, registry_etw_events (write events the session delivered since the agent started), registry_etw_dropped (events evicted from the 65536-event buffer), registry_etw_lost (events the kernel session or the decoder lost, plus failed session queries), and registry_attr_attributed, registry_attr_ambiguous and registry_attr_none (rows written with a named process, with several candidates, and without a named process for any reason other than ambiguity: attr:none, attr:off, attr:degraded and attr:lost-restart, since the agent started). A root that is unreadable or capped keeps its last good baseline and is retried; it is never reported as removed, and a root that cannot be read at all is also reported once as a gap:unreadable:<root> row. A root is capped when its read hit the read budget or when a key under it was written to while it was being read (a partial read derives no deletions; the root is read again at the next tick). A root that was read completely once keeps the event buffer from releasing events newer than that read, so an idle root holds it back by up to an hour (every root is re-read hourly) and one that then stays unreadable or capped holds it back for as long as it does. The buffer holds 65536 events: a root last read an hour ago fills it at about 18 registry writes per second across the host, and one last read ten minutes ago at about 110 per second. Past that the oldest events are evicted, registry_etw_dropped rises on every tick, gap:loss rows recur, and the changes the hourly re-read finds are written attr:degraded. If registry_etw_dropped keeps rising, check registry_roots_unreadable and registry_roots_capped first: a root stuck in either state is the one holding the buffer back furthest. registry_watch_roots_unarmed counts roots with no live watch: a root whose key does not exist (for example most of the per-user keys of the S-1-5-18, S-1-5-19 and S-1-5-20 service accounts) and any root past the 62-watch limit, which are read on every full re-read instead. It is normally non-zero on some hosts; the signal that the watcher itself has stopped is registry_watch_roots_unarmed equal to registry_watch_roots. The eleven keys are written only when a value changes, so on a quiet host they look frozen. To tell a quiet source from a stalled one, read registry_last_collect_ts (epoch seconds of the last committed tick; SELECT value FROM echo_config WHERE key = 'registry_last_collect_ts' through echo.sql), which should be no older than a few slow_intervals, and the collect output, which carries echo|collect_registry|0|capture_incomplete on a tick that could not complete (also reported as registry_last_status in echo.status). The ETW session is named YuzuEchoRegistryEtw-<agent pid> and outlives an agent that crashed or was killed, holding one of the host’s limited trace-session slots until it is stopped or the host restarts; only an agent that gets the same pid again stops it by itself. logman query -ets lists the YuzuTar* sessions and logman stop <name> -ets removes a leaked one. An agent that ran the key-watch-only build without elevation stays that way after the upgrade: registry_capture_method stays notify and every row says attr:off until the agent runs elevated.

Querying with SQL

Use the echo.sql action to execute SELECT queries against warehouse tables:

POST /api/v1/instructions/execute
{
  "plugin": "echo",
  "action": "sql",
  "parameters": {
    "sql": "SELECT name, pid, cmdline FROM $Process_Live ORDER BY ts DESC LIMIT 50"
  }
}

Pre-built queries are available in content/definitions/echo_warehouse.yaml, covering common use cases like recent processes, TCP connections by process, listening ports, hourly summaries, service state changes, user sessions, and process trees.

Rollup aggregation

The warehouse automatically runs a rollup aggregation cycle every 15 minutes. Each cycle:

1. Aggregates live-tier data into hourly summaries
2. Aggregates hourly data into daily summaries
3. Aggregates daily data into monthly summaries
4. Enforces retention limits on each tier

To force an immediate rollup, use the echo.rollup action.

Safety controls

SQL queries are validated at multiple levels:

• Server-side: Only SELECT statements are permitted. A keyword blocklist rejects INSERT, UPDATE, DELETE, DROP, ALTER, CREATE, ATTACH, DETACH, PRAGMA, VACUUM, and REINDEX.
• Agent-side: Queries run on a dedicated read-only SQLite connection guarded by a sqlite3 authorizer. Only SELECT against registry-known warehouse tables is permitted (reference them by the $-prefixed names: $Process_Live, $TCP_Hourly, etc.); PRAGMA, ATTACH, schema-table reads (sqlite_master), writes, and recursive CTEs are denied at prepare time. $-prefixed names are translated to physical names only outside string literals and comments. Only a single statement is allowed, and queries exceeding 4KB are rejected. A blocked query returns query rejected: operation or table not permitted — for schema discovery use the $-prefixed warehouse table names rather than PRAGMA/sqlite_master.

Data storage

Echo events are stored in {data_dir}/echo.db (SQLite), where data_dir is the agent’s configured data directory. The database uses:

• WAL journal mode for concurrent performance
• busy_timeout=5000 for thread safety
• secure_delete=ON to zero deleted data
• On Linux and macOS, echo.db and its -wal/-shm/-journal files are set to owner-only (0600) each time the database is opened. On Windows the agent creates or re-secures the same four files with a protected DACL before SQLite opens them (SYSTEM, Administrators and the agent service account only, no inheritance from C:\ProgramData\Yuzu), so local standard users cannot read them; an upgraded database is re-secured on its first open. A file in that set owned by any other account, or carrying more than one hard link, is treated as planted: a sidecar is removed and re-created (only its own name in the state directory is removed), and a foreign-owned or hard-linked echo.db stops the open, with the file, its owner and the remedy in the agent log. The owner test needs a volume with ACLs: where the owner cannot be read (FAT, exFAT) it is no evidence and refuses nothing. For a foreign-owned echo.db, do not adopt it: move echo.db and its -wal, -shm and -journal files aside or remove them, then restart the agent (the history in them is lost). The same applies when the agent now runs as a different account from the one that created the files and that former account is neither SYSTEM nor an administrator: the files stay owned by it, and icacls <file> /reset does not change an owner. Adopt them only after comparing the owner SID the log names with the former agent account’s SID (a local standard user’s S-1-5-21-…-1xxx is the planted case) and finding that same owner on all four files; otherwise move the set aside. To adopt them, an administrator takes ownership of the four files and grants the new account full access (this sequence has not yet been verified on a Windows host). When LocalSystem or an administrator account created the files, no owner is named, because those owners pass the owner test: the log shows could not open <file> to secure it (error 5) followed by failed to open echo.db, and the remedy is to move the four files aside or have an administrator grant the new account full access (also not yet verified on a Windows host). For a hard-linked file, remove the other link, which fsutil hardlink list <file> names. A file whose permissions deny the agent the right to change them gets the same two checks and keeps its permissions if it passes. A handle opened before the upgrade keeps its access until it is closed. Every other file in the state directory keeps the directory’s inherited ACL (see docs/agent-privilege-model.md)
• Indexes on timestamp, (event_type, timestamp), and snapshot_id

Upgrading from the pre-rename database

Echo was named TAR before 2026-10, and its database was {data_dir}/tar.db. On its first start after the upgrade the agent adopts that file before any plugin loads: it moves tar.db-journal, tar.db-wal and tar.db-shm to the echo.db names first and tar.db last, so an interrupted adoption simply resumes on the next start. The Echo plugin then renames the file’s tar_state, tar_config and tar_cursor tables to echo_* (schema version 9) in one transaction. Nothing is copied and the file keeps its owner-only permissions, so no history or operator setting is lost, including a source an operator paused.

• If echo.db and tar.db both exist (an agent downgraded after the rename and then upgraded again), they are two independent databases. The agent uses echo.db, leaves tar.db and its sidecar files untouched and logs one warning that names them; rows the older agent wrote to tar.db in the meantime are not merged.
• If tar.db exists and cannot be adopted (a read-only data directory, for example), the agent logs the error and the Echo plugin refuses to start rather than creating a fresh, empty echo.db beside it; a fresh file would re-enable every source an operator had paused.
• Old tar.db.corrupt-* quarantine files are not touched and can be deleted once they are no longer needed.
• The Echo plugin’s stored health history (its key-value namespace) is re-keyed from tar to echo at the same time.