Monitoring and recovery

Partly implemented. GET /v1/health exists and reports the protocol signals below as machine-readable checks. There is still no collector, no dashboard and no pager integration in this repository — those are yours to point at the endpoint. What is described as design is labelled as such.

#What would be worth watching

#GET /v1/health

Distinct from /v1/status, which reports whether dependencies are reachable. Every dependency can be green while no keeper has mirrored a known divergence for an hour, and that is what this exists to surface.

Each check carries a status, a detail a human can act on, and an observed object of raw numbers so a monitor can apply its own thresholds rather than inheriting the ones chosen here.

The HTTP status carries the verdict, so a monitor need not parse the body: 200 while working or degraded, 503 once a check is failing.

#Why unknown is a status

A check that could not run is not a passing check. Reporting it as one is how an outage stays invisible while every light says fine, so an unreadable check says unknown and says why. unknown outranks ok in the overall verdict and never outranks a real failure, and it is never 503 — paging on "I could not see" trains an operator to ignore the endpoint.

Running with no database is a supported configuration, so indexer-lag reports unknown there rather than pretending either health or breakage.

#Why silence is not failure

The obvious keeper check is time since the last sync, and it is the wrong one. A keeper with nothing to do is silent, and a vault whose source has not moved needs no mirroring — alerting on quiet would page someone nightly for a healthy system.

What distinguishes a stuck keeper from an idle one is unresolved divergence: a vault that knows its source has moved and has not caught up. Ages are measured against the chain's own clock rather than the server's, because the two diverge exactly when something is wrong.

#What is deliberately not an alarm

A pause is an operator acting deliberately, and is reported as ok with the count. Making the control look like an incident would discourage using it.

A vault refusing to state a NAV it cannot compute is degraded, not failing — the risk control working, not the system breaking. It still needs attention, because in that state the vault will not accept deposits, while withdrawal in kind keeps working because it consults no price feed.

/v1/status already reports chain reachability, identity provider mode, database presence and head block, and distinguishes required from optional dependencies. That is the seam a monitor would read.

#What can be done in an incident

The honest list is short, and it is short by design.

Available. Pause deposits. Pause execution. Pause an individual asset in the registry. Each is scoped and none of them touches custody.

Not available, and deliberately absent. There is no function to move a depositor's assets, rewrite a snapshot, force a redemption, block a withdrawal, or upgrade a vault's behaviour. An incident cannot be resolved by seizing funds, because the capability does not exist.

Security. Pausing execution stops mirroring. It does not stop withdrawal: redeemInKind has no pause check and consults no price feed, so depositors can always leave with their pro-rata basket while an incident is being worked.

#Recovery posture

Because there is no upgrade path, recovery from a contract-level defect means deploying a new vault version and letting depositors move — not patching a live one. That is a deliberate cost: the same property that makes recovery slow is what makes the deployed code trustworthy without trusting an upgrade key.