On-call quick reference card
Diátaxis quadrant: Reference. Audience: security-operations on-call staff during and between incidents.
Fast lookup for a shift on humanymous Gate (the reverse-proxy enforcement layer, “Gate” after first mention). Scan it; do not read it. This is the reference build, not a production-hardened one. When you need the full procedure, jump to Incident runbooks and Kill switch & bans.
Verdict legend — what each does at the edge
| Verdict | Edge action | Origin contacted? |
|---|---|---|
| ALLOW | pass |
Yes |
| CHALLENGE | challenge_pow — minimal reference challenge response; deployed recovery flow required |
No |
| DENY | block |
No |
| none (Unknown, no evidence yet) | fail-closed on strict routes or unsafe methods (POST/PUT/PATCH/DELETE) → challenge; fail-open only for safe GET/HEAD on non-strict routes | Depends on the fail rule |
Built-in score bands are 0–29 ALLOW, 30–69 CHALLENGE, and 70–100 DENY. Runtime Settings can change the thresholds. Enforcement rules raise the score-based result. A valid proof-of-work result can clear only a score-based challenge, and the reference Gate does not provide the complete visitor flow by itself.
Top enforcement rules to recognize
Do not treat a DENY spike as proof by itself. Confirm the predicate, collection plane, topology, and representative records. High-confidence combined rules are stronger evidence than heuristics, but no public document promises a zero false-positive rate.
| Rule | Fires on | Action | Read a spike as |
|---|---|---|---|
| hard automation artifact rule | Automation artifact (Selenium / Puppeteer / Playwright) | DENY | Attack |
| browser-control leak plus automation evidence rule | Chrome DevTools Protocol leak + automation hint | DENY | Attack |
| cross-session correlation rule | One fingerprint across many subnets (residential-proxy rotation) | DENY | Attack |
| automated-browser interaction signature rule | AI browser-agent (teleport click + LLM cadence) | DENY | Attack |
| Hypertext Transfer Protocol version 2 denial-of-service protection rule | Rapid Reset or continuation-frame flooding | DENY | Protocol abuse; verify frame evidence |
| decision-probing sweep protection | Decision-probing sweep: many sessions from one fingerprint | DENY | Attack |
| no interaction observed rule | No interaction over the observation window | CHALLENGE | Heuristic — can catch some real humans. Investigate before treating as attack volume. |
Full table and signal-ID drill-down: enforcement rules & verdicts.
The three levers — and who commits
| Lever | Endpoint | Committer |
|---|---|---|
| Unblock (lift temp ban) | POST /__hmn/admin/bans/lift |
Single Operator |
Temp ban (1h/6h/24h) |
POST /__hmn/admin/bans |
Single Operator |
| Permanent / CIDR ban | POST /__hmn/admin/bans → POST /__hmn/admin/approvals/<id> |
Distinct Approver (dual-control) |
| Kill switch (node-local) | POST /__hmn/admin/killswitch → POST /__hmn/admin/approvals/<id> |
Distinct Approver (dual-control) |
| Erasure (crypto-shred) | (control plane) | Distinct data protection officer (dual-control; generic Approver cannot) |
The requester never self-commits a dual-control action, in either direction (placing or lifting). Actor identity is server-derived from the bearer token; body actor fields are ignored.
Warning: The kill switch is node-local and not route-scoped. Detection still scores and logs, and manual bans still enforce. Coordinate every node separately in a fleet.
Ban ladder & key choice
Auto-bans climb on repeat strikes, with strike decay downward:
1h → 6h → 24h → permanent
Source=auto = ladder placed it; Source=manual = you placed it. Active set: GET /__hmn/admin/bans.
fp:<fingerprint>— use for residential-proxy rotation (cross-session correlation rule). The IP moves; the fingerprint is the stable handle.ip:<addr>/cidr:<range>— use for a fixed hostile source (many fingerprints, one IP/range). Dual-control for CIDR.- Avoid
ip:/cidr:against shared egress — CGNAT and corporate egress put real humans behind the same address. Preferfp:-scoped action or let rate metering absorb it.
20-second: attack vs false-positive
- Which verdict spiked? DENY → lean attack. CHALLENGE → could be humans.
- Which rule? A high-confidence DENY rule (hard automation artifact rule/9/19/20/21/30) firing → attack; act with bans. no interaction observed rule (CHALLENGE, heuristic) climbing → suspect humans first.
- Are real users complaining? User reports of blocks/challenges on a legit path = false-positive signal. No reports + rising DENY = attack.
- Shared handle? Spike keyed to one
fp:across subnets = cross-session correlation rule rotation (ban thefp:). Spike behind one NAT/carrierip:= likely shared-egress false positive (do not IP-ban). - Verdict: Attack → escalate bans, keep enforcement on. False-positive storm locking out customers → route-table demote + restart, or the node-local kill switch. Unsure → run the triage in Incident runbooks before touching the switch.
Not the behavioral model. If the Core runs the optional behavioral model, its signal (
l4.ml.behavioral) is weight-0 / audit-only — it is never the cause of a block. Never point a false-positive investigation at it, and if its canary auto-rollback fires (ml-canary: AUTO-ROLLBACKin the Core log, orcanaryrolled-back at/api/mlcorrect) the model simply reverted to heuristics: verdicts are unaffected — do not page it as an outage. Confirm the revert, then redeploy the previous Core image tag to restore a known-good artifact.
Endpoints you’ll actually hit
| Action | Endpoint |
|---|---|
| Read decision feed / filter | GET /__hmn/admin/audit (?verdict=&host=&route=&rule=&minRisk=&before=) |
| Verify chain integrity | GET /__hmn/admin/integrity |
| Place a ban | POST /__hmn/admin/bans |
| Lift a ban | POST /__hmn/admin/bans/lift |
| Request kill switch | POST /__hmn/admin/killswitch |
| Commit a dual-control request | POST /__hmn/admin/approvals/<id> (find id: GET /__hmn/admin/approvals) |
Console: https://localhost:8445/__hmn/admin/console (separate admin listener; 404 on the public edge). Bearer auth on every call — a missing/invalid token returns 404, which is deny-by-default, not a bug.
The table above is the Gate admin plane. Separately, if the Core runs the behavioral model, GET /api/mlcorrect (Core ops-token, read-only, no personal data) reports the model’s health — active artifact version/digest, calibration, drift, shadow, and canary state. See Operate the behavioral model.
For a quick Core health rollup, GET /api/counters (same Core ops-token) now carries two on-call tells alongside version/uptime: ml.canaryState — if it reads tripped, the behavioral model auto-disabled itself (a self-disable is otherwise only in the log; verdicts are unaffected since the signal is weight-0); and fleet.redisFallbackTotal — a monotonic count of fleet-correlation observations that degraded to per-node (Redis coordinator down or a broken script). Alert on that total’s rate: a rising redisFallbackTotal means cross-node correlation is silently no longer fleet-wide (again verdict-neutral — fail-open by design — but worth fixing the coordinator).
Note: Bodies —
POST /bans{"Key","Reason","Incident","DurationSec"};/killswitch{"On":true};/bans/liftuses a?key=<ban-key>query parameter, not a body. Full shapes: CLI, config & policy reference.
KPI & alert thresholds — a framework, not numbers
The Overview view surfaces KPIs computed from the audit stream (GET /__hmn/admin/audit), and the Integrity view re-verifies the chain. The reference build hardcodes no alarm thresholds. Set your own from a monitor-mode baseline; do not copy numbers from another deployment.
Watch three signals:
- CHALLENGE / DENY mix — the ratio of verdicts over a window. A shift toward DENY leans attack; a shift toward CHALLENGE (especially no interaction observed rule-driven) leans false-positive.
- DENY-rate deltas — change against your own baseline matters more than any absolute rate. Alarm on the delta, not a fixed count.
- False-positive signal from user reports — blocks/challenges reported on legitimate paths. This is the counter-signal to a rising DENY rate and the trigger to suspect a misfire.
Note: Healthy vs alarm bands are operator-set. Establish them from your own monitor-mode baseline and tune per route.
TODO(verify): specific numeric healthy/alarm ranges for CHALLENGE/DENY mix and DENY-rate delta — none exist in the reference and none may be invented; each operator must derive them from their own baseline.
The Overview surfaces five KPI tiles, each a count over the recent window: Allow, Challenge, Deny, Ban actions (auto + manual), and Records (total events sealed into this node’s chain).
Related
- Incident runbooks — symptom-indexed procedures and the attack-or-false-positive triage.
- Kill switch & bans — blast radius, dual-control, apply / escalate / lift.
- enforcement rules & verdicts — the full rule set, verdict bands and signal-ID drill-down.