Set up notary hot failover

Starting in version 1.43, the platform's anti-rewind mode decides how strictly the notary applies anti-rewind protection, and whether failover is possible. There are three modes: STRICT, BALANCED, and DISABLED. BALANCED and DISABLED both enable failover.

With failover, Ripple Custody runs a standby notary that takes over automatically when the serving notary is silent for the bridge's exclusivity period. The period is 15 minutes by default, and you can set it to any value of 10 minutes or more. A notary outage then adds latency to in-flight transactions instead of stopping intent processing until you recover the notary.

This page describes the modes, how to deploy the standby notary, and how to monitor and respond to a takeover.

The notary compares the collection revisions that the bridge sends from the database with the revisions that its anti-rewind file (ARF) records. The anti-rewind mode decides what the notary does when they don't match:

• STRICT is the behavior of versions before 1.43. A standby notary starts with an empty ARF, so STRICT can't accept it.
• BALANCED accepts a collection that the notary has no record of, which lets a standby notary start from an empty ARF. It still rejects a collection whose revision conflicts with its record.
• DISABLED skips the anti-rewind check. The notary still updates its ARF.

The notary takes its mode from the first of these sources that applies:

1. The mode that you set at Genesis or with a v0_SetSystemProperty intent. The notary records it in its ARF, and it takes precedence over the other sources. To change it, submit a new intent.
2. A mode that you declare on the notary host. If the notary already runs with HMZ_NOTARY_ALLOW_EMPTY_COLS=true, it runs in BALANCED mode, so platforms that use this setting keep their current behavior. For the replacement, antiRewind.seedMode, see Notary configuration.
3. STRICT, by default. Platforms that completed Genesis before version 1.43 run in STRICT mode, and behave as before, until you change the mode.

New platforms declare the mode in the Genesis request. To use failover on a new platform, set the top-level antiRewindMode field to BALANCED:

{
    "rootDomainSetup": { ... },
    "cryptoSetup": { ... },
    "antiRewindMode": "BALANCED"
}

For the full Genesis request body, see Genesis payload reference.

• Two notaries run. One is serving, and the other is on standby.
• If the serving notary is silent for the bridge's exclusivity period, the standby notary takes its place. The period is 15 minutes by default, and you can change it with HMZ_NOTARY_BRIDGE_EXCLUSIVITY_PERIOD. This takeover is final: the platform can't fail back to the old notary.
• Each notary keeps its own ARF. The standby notary starts with an empty ARF and accepts the database as it is at takeover. If someone rolled back the database just before a takeover, the new notary doesn't detect it. The rewind after failover alert covers this case.

A takeover needs two notaries and a platform mode of BALANCED or DISABLED. You can set BALANCED with a single notary, but no takeover can happen until you deploy the standby notary.

1. Upgrade the notaries, and then the bridge, to version 1.43.

2. Set the anti-rewind mode to BALANCED. Submit a v0_SetSystemProperty intent, following the steps in Manage intents and approvals, with a payload similar to the following example:

   "payload": {
       "value": {
           "type": "AntiRewindModeProperty",
           "value": {
               "mode": "BALANCED"
           }
       },
       "type": "v0_SetSystemProperty"
   }

   The first time you set the mode, omit revision: the intent creates the property. To change the mode after that, add a revision field with the property's current revision, as when you rotate the state review authority key. An intent with a revision fails if the property doesn't exist, and an intent without one fails if it does.

3. Confirm the mode. Check that the intent completes and that GET /v1/properties returns ANTI_REWIND_MODE with the mode BALANCED.

4. Deploy the standby notary, ideally in another availability zone. The standby is a second instance of the same notary, with the same notary keys and HSM access. Give it its own persistent storage for its ARF and identity file:

   • ARF: The standby starts with an empty ARF, and records collections as it serves them.
   • Identity file: The notary generates its identity file at startup if the file is missing.

   Configure it as a standby in your notary deployment. If you deploy the notary with the approval-notary Helm chart, set standby.enabled to true. See Notary configuration.

5. Check that failover is ready:

   • Both notaries are running.
   • The notary.notary_instance table has exactly one row with the status SERVING. Use only read-only queries on this table.
   • The hmz_bridge_notary_silence_ratio metric is 0.

If your platform has a state review authority, don't declare the mode on the notary host with antiRewind.seedMode. Set it with the intent.

Create the following three alerts from the bridge and notary metrics.

The rewind after failover alert looks back 30 days, so keep at least 30 days of metrics history. If the new notary first uses a collection after that window, the alert can't detect a rewind on it.

By default, let the takeover happen, and find out why the notary is silent.

Shut down the standby notary to prevent the takeover if either of the following applies:

• You're doing planned work that takes longer than the exclusivity period.
• You have a reason to doubt the database, such as a restore, a database incident, or revision mismatch rejections just before the silence.

1. Keep the evidence: the bridge and notary logs, the notary.notary_instance table, and 30 days of hmz_notary_collection_revision. Don't reset either notary yet.

2. Compare the revisions of each collection. The bridge's takeover log line gives the identities of both notaries. Filter hmz_notary_collection_revision on notary_instance_id to find the last revision on the old notary and the first revision on the new notary. Don't compare with the current database, which has changed since the takeover.

   Old revision minus new revision	Meaning
   0 or less	Normal failover
   1	One request is missing. This is normal if the request was in progress when the notary stopped and the notary never confirmed it. If the notary confirmed the request, it's a rewind.
   2 or more	Rewind

3. After a normal failover, clean up the old notary's storage by deleting both its ARF and its identity file. It restarts as the new standby notary. The platform keeps using the notary that took over.

1. Shut down the bridge. The notary signs nothing, and no takeover can happen. Intents wait in the queue.
2. Don't change anything else. Don't reset a notary, wipe storage, or edit the notary.notary_instance table. The old notary's ARF is the evidence.
3. Contact Ripple Support with the evidence.

A rewind alert that resolves on its own doesn't mean the platform is safe. New activity eventually passes the old maximum revision.

A notary doesn't recover a corrupt ARF on its own, in any mode. Either:

• Recover the ARF with the signed manifest procedure.
• In BALANCED or DISABLED mode, if you deployed a standby notary, shut down the notary with the corrupt ARF. The standby notary takes over after the exclusivity period. Then follow the takeover steps.

• Data integrity and audit trail
• Recover your anti-rewind file
• Resilience planning