Incident runbook

Planned

The procedure: what to read, what to pause, and why recovery means a new vault.

Not yet rehearsed. docs/PRODUCTION_GATES.md Gate 6 requires this procedure to be practised at least once, including a pause drill, before mainnet. Writing it does not satisfy that. A runbook nobody has executed is a document, not a capability.

This is the operational procedure. What each control actually does, and why recovery has the shape it has, is in Security model and Known limitations.

#Before anything else

Depositors can always leave. redeemInKind has no pause check and consults no price feed. Whatever else is true during an incident, the exit that returns a pro-rata share of every asset the vault holds keeps working. Nothing in this runbook can trap a depositor, and nothing in it should be delayed out of fear that it might.

That is worth saying first because it changes what pausing costs. Pausing stops new risk; it does not strand anyone.

#1. Establish what is actually happening

bash

curl -s https://<api>/v1/health | jq

Each check and what it means for action:

Check	Reading it
`divergence`	`failing` means a vault knows its source moved and has not caught up for over an hour. Usually keeper liveness, not the protocol.
`nav-availability`	`degraded` means a vault cannot price itself and is refusing deposits. That is the risk control working.
`skip-reasons`	One reason dominating usually means one upstream failure, not many independent decisions.
`indexer-lag`	Affects history only. Never a reason to pause anything.
`pauses`	What is already paused. Check this before pausing more.

unknown is not ok. A check that could not run has told you nothing, and the absence of an alarm from it is not evidence.

Then confirm the chain agrees, because the API is a reader and can be wrong about a chain that is fine:

bash

pnpm verify:chain -- --mainnet     # 19 cross-checks, read-only
pnpm verify:roles -- --mainnet     # who holds every role right now

#2. Decide whether to pause, and what

Pausing is scoped. Reach for the narrowest control that addresses the problem.

Symptom	Control	Role	Effect
One asset is behaving badly	`AssetRiskRegistry.pauseAsset(token, reason)`	`PAUSER_ROLE`	That asset stops trading everywhere. Others continue.
One vault is the problem	`MirrorVault.setExecutionPaused(true, reason)`	`PAUSER_ROLE`	That vault stops mirroring. Withdrawal unaffected.
One vault should take no new money	`MirrorVault.setDepositsPaused(true, reason)`	`PAUSER_ROLE`	Deposits refuse. Existing depositors unaffected.
Execution is unsafe protocol-wide	`ExecutionRouter.setExecutionPaused(true, reason)`	`PAUSER_ROLE`	All mirroring stops. Withdrawal unaffected.

Every one of these takes a reason string and emits it. Write something a person reading the event in six months can act on — not "pause".

Pausing is one-way from a hot key. pauseAsset needs PAUSER_ROLE; unpauseAsset needs RISK_ROLE. That asymmetry is deliberate: stopping should be cheap and available under pressure, and restarting should require the colder key that approved the asset originally. Do not plan a response that assumes you can quickly undo a pause.

#3. What you cannot do, by design

There is no function to move a depositor's assets, rewrite a snapshot, force a redemption, block a withdrawal, or upgrade a vault's behaviour. None of these capabilities exist to be abused, which also means none of them exist to be used in an emergency.

So an incident cannot be resolved by seizing funds and making people whole afterwards. If value is lost inside a vault, it is lost, and the response is disclosure rather than recovery.

#4. Recovery

Because there is no upgrade path, recovering from a contract-level defect means deploying a new vault version and letting depositors move. It does not mean patching a live one.

That is slow, and the slowness is the same property that makes the deployed code trustworthy without trusting an upgrade key. Plan for it rather than around it:

Pause execution on the affected vaults, with a reason naming the defect.
Say publicly what is wrong, what is paused, and that in-kind exit still works.
Deploy the corrected version.
Let depositors exit the old vault and enter the new one at their own pace.

Do not un-pause a defective vault to "let people out". They can already get out.

#5. Afterwards

The reason strings are the incident record. They are on chain and permanent.
reorg_events in the indexer records reorganisations rather than swallowing them; an unexplained gap in history is a bug report.
Update Known limitations if the incident revealed something the documentation did not already say.

#The drill Gate 6 asks for

Rehearse on testnet, and rehearse the whole thing:

Pause one asset with pauseAsset, and confirm mirroring stops for it while other assets continue.
Confirm a depositor can still exit in kind from an affected vault while it is paused. This is the step that matters most and the one most likely to be skipped.
Unpause with RISK_ROLE, and confirm PAUSER_ROLE alone cannot.
Record how long each step took and who could perform it.

Step 2 is the drill. The rest is checking the controls are wired; step 2 is checking the promise this protocol makes to depositors is real under the conditions where it counts.