Observability and SIEM integration

Quadrant: How-to. Audience: SRE and security engineers wiring humanymous Gate into your monitoring and SIEM.

This guide shows you what observability surface the reference implementation of humanymous Gate exposes, how to pull edge decisions out of it with curl, how to scrape process metrics and wire liveness/readiness probes, and how to bridge decision data to a SIEM until native log shipping is in place.

Important: This repository is a reference implementation, not a production-hardened build. Gate now exposes a Prometheus metrics endpoint and load-balancer/orchestrator health probes (documented below); structured-log SIEM shipping in a standard schema is still a production delta, so the poll-and-forward pattern below remains the supported interim path for decision data.

What exists today

The reference Gate gives you five observability surfaces. Three are on the authenticated admin listener (-admin-addr, default 127.0.0.1:8445 (loopback)); the metrics endpoint is on the admin listener too; the liveness/readiness probes are the exception — they answer unauthenticated on the public edge control plane so an orchestrator can reach them.

1. The audit stream — GET /__hmn/admin/audit, structured JSON records of every edge decision, with server-side filters and a paging cursor. This is your primary machine-readable feed, and the source of per-verdict rates (see below).
2. The Ledger Overview KPIs — a human-facing live feed of allow/challenge/deny decisions with rollup counters, at https://localhost:8445/__hmn/admin/console. See the Ledger tour.
3. Chain-integrity status — the Integrity view and GET /__hmn/admin/integrity, which run live SelfVerify (hash chain + HMAC + Ed25519 STHs + witness when configured). Empty chains report empty-chain, not a green pass.
4. Prometheus metrics — GET /__hmn/admin/metrics on the admin plane, process/chain/ban gauges and counters in text-exposition format (see Scrape process metrics).
5. Liveness and readiness probes — GET /__hmn/healthz and GET /__hmn/readyz, unauthenticated on the public edge, for orchestrator probes and load-balancer drain ordering (see Wire liveness and readiness probes).

Where per-verdict rates come from

The metrics endpoint deliberately does not emit per-verdict rates (allow/challenge/deny counts). Derive those from the audit stream (GET /__hmn/admin/audit), which your SIEM already ingests — the audit records carry verdict, risk_score, route_class, and triggered_rules, so rates and breakdowns are a query over data you already collect. /__hmn/admin/metrics covers the process, chain-growth, and ban-state gauges that a query over the audit stream cannot give you.

Note on /health: The /health route is not a Gate health check. It is an origin application path that ships mapped to the off preset — Gate does not inject or enforce on it. It is a documented bypass of detection, not a liveness signal for Gate itself. Use /__hmn/healthz for Gate’s own liveness; point orchestrator probes there, not at /health.

Authenticate to the admin API

The admin API base is /__hmn/admin/ on the admin listener. Every request needs a bearer token; the comparison is constant-time, and a missing or invalid token returns 404 (deny-by-default — the admin plane does not advertise itself). Every authenticated access is meta-audited (an admin.access record) before anything is served.

Tokens are configured through the HMN_ADMIN_TOKENS environment variable ("auditor:tok,operator:tok,approver:tok,dpo:tok"); if unset, Gate mints random tokens per boot and prints them at startup. For observability you only need read access, so use the Auditor token — Auditor is read-only (canRead) and cannot request or approve any change. See RBAC and separation of duties for the full role matrix.

Set your token once:

export HMN_ADMIN_TOKEN="<your-auditor-token>"

Pull edge decisions from the audit stream

GET /__hmn/admin/audit returns edge-decision records as structured JSON. It supports these server-side filters, which you can combine:

• verdict= — allow, challenge, deny, or none
• host= — origin host
• route= — matched route_class (html|api|upgrade|control|static), not a URL path
• rule= — a hard-rule identifier (for example a specific HR)
• minRisk= — minimum risk score (0–100)
• subject= — a pseudonymous subject, matched against session_pseudonym or actor.id_pseudonym (see Export one subject’s records)
• before= — the paging cursor

Fetch the most recent deny decisions:

curl --silent \
  --header "Authorization: Bearer ${HMN_ADMIN_TOKEN}" \
  "https://localhost:8445/__hmn/admin/audit?verdict=deny"

Note: The reference dev certificate is self-signed in memory. Against a dev instance you will need curl --insecure (or -k) to skip certificate validation. Do not carry that flag into any trusted environment.

Page through history with the cursor

The response carries a nextBefore cursor. To page backward through older records, pass that value as the next request’s before= parameter; repeat until the cursor is empty (no more records).

curl --silent \
  --header "Authorization: Bearer ${HMN_ADMIN_TOKEN}" \
  "https://localhost:8445/__hmn/admin/audit?verdict=deny&before=<nextBefore-from-previous-page>"

The response envelope is {"records":[…],"count":<n>,"nextBefore":<seq>} — records newest-first; page the next (older) batch with ?before=<nextBefore>. limit defaults to 50, max 500.

Each record’s canonical fields for a SIEM mapping: event_id, seq, node_id, ts (RFC3339 nanoseconds, UTC), event_type, event_version, actor{kind,id_pseudonym}, tenant_id, session_pseudonym, host, route_class (html|api|upgrade|control|static), verdict (allow|challenge|deny|none), risk_score, triggered_rules (["HR-…"]), top_signals[{id,verdict,conf}], enforcement_action, enforcement_mode (monitor|shadow|enforce), fail_mode (none|fail_open|fail_closed|degraded), fail_reason, latency_us, upstream{status,error_class}, tls{ja4_pseudonym,h2fp_pseudonym,sni_pseudonym}, config_version, key_id, data_class, the chain fields prev_hash/record_hash/hmac, and incident (the opaque handle, added at read time). Note the subject is a pseudonym (actor.id_pseudonym / session_pseudonym), never a raw identifier. The full shapes live in the CLI, config & policy reference.

Narrow to what you care about

Combine filters to build focused queries an on-call engineer or a scheduled job can run. For example, high-risk challenges on HTML page requests whose host contains checkout (route= filters by route_class, and host= is a substring match — neither takes a URL path):

curl --silent \
  --header "Authorization: Bearer ${HMN_ADMIN_TOKEN}" \
  "https://localhost:8445/__hmn/admin/audit?verdict=challenge&host=checkout&route=html&minRisk=50"

Records are pseudonymous: raw identifiers (IP, TLS/HTTP fingerprints, UA, device fingerprint) are stored only as per-subject-key-derived pseudonyms, never raw. Your SIEM will ingest pseudonyms, not source IPs. Re-identification is a separate, dual-controlled action and is out of scope for a monitoring feed.

Export one subject’s records (GDPR Art. 15)

The ?subject= filter scopes the stream to a single pseudonymous subject, for a subject-access request (GDPR Art. 15 / PIPA). Pass the deterministic pseudonym the Console shows for a subject; Gate matches it against both session_pseudonym and actor.id_pseudonym, so a subject who appears in either position is returned.

curl --silent \
  --header "Authorization: Bearer ${HMN_ADMIN_TOKEN}" \
  "https://localhost:8445/__hmn/admin/audit?subject=<subject-pseudonym>"

This produces a subject-scoped export you can hand to a data-subject-access workflow. It combines with the other filters and the before= cursor exactly as above (add &limit=500 and page the cursor to drain a subject’s full history). The filter takes a pseudonym, not a raw identifier — resolving a raw identity to its pseudonym (or the reverse) is a custodial offline act against the keystore, out of scope for this read-only feed (Gate has no re-identification API).

Check chain integrity

GET /__hmn/admin/integrity runs the audit-log verification live and reports whether the tamper-evident chain is intact — the same logic the Console’s Integrity view surfaces. Poll it on a schedule and alert on any failure class.

curl --silent \
  --header "Authorization: Bearer ${HMN_ADMIN_TOKEN}" \
  "https://localhost:8445/__hmn/admin/integrity"

The verification can report these mismatch classes, any of which should page your security on-call: hash-break, hmac-invalid, seq-gap, linkage-break, checkpoint-mismatch, and node-missing (a suppression alert).

GET /__hmn/admin/integrity returns {"node","ok":<bool>,"class":"<mismatch-class>","records":<n>,"checkpoints":<n>,"witnessed":<bool>,"lastSTH":{"treeSize","root"}}; on failure it adds divergentSeq, detail, and (if the witness attestation fails) witnessFailAt. Alert on ok:false or witnessed:false. The mismatch classes are hash-break, hmac-invalid, seq-gap, linkage-break, checkpoint-mismatch, and node-missing — see Verify the audit log.

Scrape process metrics (/__hmn/admin/metrics)

GET /__hmn/admin/metrics exposes Gate’s process, chain-growth, and ban-state telemetry in Prometheus text-exposition format (Content-Type: text/plain; version=0.0.4). It is served on the admin plane and is auth-gated (any role — an Auditor token is enough), so Prometheus must scrape it with a bearer token, and behind mTLS if you set -admin-mtls-ca.

curl --silent \
  --header "Authorization: Bearer ${HMN_ADMIN_TOKEN}" \
  "https://localhost:8445/__hmn/admin/metrics"

The endpoint emits these series:

What to alert on

The audit-chain series are the highest-signal structural alerts on this endpoint — they need no tuning and no invented thresholds. Alert on hmn_gate_audit_integrity_ok == 0, hmn_gate_audit_witnessed == 0, and any increase in hmn_gate_audit_projection_dropped_total (increase(hmn_gate_audit_projection_dropped_total[10m]) > 0). The first two are the metrics twin of the GET /__hmn/admin/integrity ok:false / witnessed:false conditions — same signal, now scrapeable without a poll. The drop counter surfaces a Tier-1/2 sink shedding records, which is otherwise only a once-a-minute WARN log. Also alert on hmn_gate_killswitch == 1 (enforcement demoted fleet-wide) and on up == 0 (a metrics blackout hides all of the above). Ready-to-load rules live in deployments/observability/alerts.yml.

Note: Per-verdict rates are intentionally absent here — derive allow/challenge/deny rates from the audit stream (see Where per-verdict rates come from). Configure your Prometheus scrape job with the Auditor bearer token (authorization / bearer_token_file) and, in dev, insecure_skip_verify: true for the self-signed admin cert. A ready-to-merge scrape job is in deployments/observability/prometheus-scrape.yml; see the observability README.

Wire liveness and readiness probes

Gate answers two probe endpoints on the public edge control plane, both unauthenticated (an orchestrator or load balancer must reach them without a token) and answered by Gate itself, not proxied to origin:

Point your orchestrator’s livenessProbe at /__hmn/healthz and its readinessProbe at /__hmn/readyz.

Drain ordering. On SIGINT/SIGTERM, Gate flips the readiness gate first — so /__hmn/readyz returns 503 draining while /__hmn/healthz keeps returning 200. A load balancer sees the readiness failure and removes the node before the in-flight drain begins, so no new traffic lands on a node that is shutting down. The drain is bounded by -shutdown-grace (default 25s); set it ≥ your orchestrator’s termination grace so the drain completes before the pod is killed. Do not gate liveness on readiness: a draining node is still alive, and a livenessProbe pointed at /__hmn/readyz would kill it mid-drain.

curl --silent "https://localhost:8444/__hmn/healthz"   # 200 ok
curl --silent -o /dev/null -w '%{http_code}\n' \
  "https://localhost:8444/__hmn/readyz"                # 200 ready, 503 once draining

Note: These probes are on the edge listener (-addr, default :8444), not the admin listener — that is deliberate, so a load balancer fronting the public edge can reach them without admin credentials.

Bridge to a SIEM in the meantime

Native SIEM shipping is a production concern the reference does not implement (see below). Until then, the supported pattern is poll-and-forward:

1. Run a small collector job on an interval.
2. Call GET /__hmn/admin/audit with an Auditor token, filtering to what you retain (or leaving filters off to capture all decisions).
3. Follow the nextBefore cursor to drain new records since your last run, persisting the cursor between runs so you never miss or double-ship.
4. Map each record’s fields to your SIEM schema and forward.

sequenceDiagram
  participant J as "Collector job (interval)"
  participant A as "Admin listener :8445"
  participant S as "SIEM"
  loop each run
    J->>A: GET /__hmn/admin/audit?before=cursor (Auditor token)
    A-->>J: records[], count, nextBefore
    J->>J: map fields to SIEM schema
    J->>S: forward records
    J->>J: persist nextBefore cursor for next run
  end
  J->>A: GET /__hmn/admin/integrity (scheduled)
  A-->>J: ok, class, witnessed, lastSTH
  J->>S: alert on ok:false or witnessed:false

This gives you allow/challenge/deny decisions, risk scores, matched routes, and triggered hard rules in your SIEM without waiting for native shipping. Because the feed is pull-based and the records are pseudonymous, your collector needs only network reach to the admin listener and a read-only token — keep that token scoped to Auditor and rotate it like any other credential.

Note: You are polling, not streaming. Your SIEM’s freshness is bounded by your poll interval. Size the interval against your audit volume so a single page-drain keeps up between runs.

Explicitly production-only (prod-delta)

The following are not in the reference build. Do not document or configure them as working; plan for them as production deltas:

• Native structured-log SIEM shipping in a standard schema (for example OCSF or CEF) — decision data still ships via the poll-and-forward pattern above.
• Server-sent-events (SSE) live-push for the Console and downstream consumers — the reference Console refreshes and polls rather than receiving pushes.

For the full list and the reasoning behind each, see Production vs. reference.

Related

• CLI, config, and policy reference — the -admin-addr flag, HMN_ADMIN_TOKENS, and other startup levers.
• Ledger tour — the Overview KPIs and Integrity view described here, in the UI.
• Production vs. reference — the complete prod-delta list.