# Set up notary hot failover

Starting in version 1.43, the platform's anti-rewind mode decides how strictly the notary applies anti-rewind protection, and whether failover is possible. There are three modes: `STRICT`, `BALANCED`, and `DISABLED`. `BALANCED` and `DISABLED` both enable failover.

With failover, Ripple Custody runs a standby notary that takes over automatically when the serving notary is silent for the bridge's exclusivity period. The period is 15 minutes by default, and you can set it to any value of 10 minutes or more. A notary outage then adds latency to in-flight transactions instead of stopping intent processing until you recover the notary.

This page describes the modes, how to deploy the standby notary, and how to monitor and respond to a takeover.

## Anti-rewind modes

The notary compares the collection revisions that the bridge sends from the database with the revisions that its [anti-rewind file (ARF)](/pt-br/products/custody/overview/security/data-integrity-and-audit-trail#anti-rewind-file-arf) records. The anti-rewind mode decides what the notary does when they don't match:

| Mode | Collection missing from the ARF | Collection conflicts with the ARF | Takeover by a standby notary |
|  --- | --- | --- | --- |
| `STRICT` | Rejects it | Rejects it | Not possible |
| `BALANCED` | Accepts it, and starts to track the collection | Rejects it | Possible |
| `DISABLED` | Accepts it | Accepts it | Possible |


- **`STRICT`** is the behavior of versions before 1.43. A standby notary starts with an empty ARF, so `STRICT` can't accept it.
- **`BALANCED`** accepts a collection that the notary has no record of, which lets a standby notary start from an empty ARF. It still rejects a collection whose revision conflicts with its record.
- **`DISABLED`** skips the anti-rewind check. The notary still updates its ARF.


DISABLED removes rewind protection
In `DISABLED` mode, the notary doesn't detect a database that someone restored to an earlier state. Use `DISABLED` only if other controls already protect the database against rollback.

The notary takes its mode from the first of these sources that applies:

1. The mode that you set at Genesis or with a `v0_SetSystemProperty` intent. The notary records it in its ARF, and it takes precedence over the other sources. To change it, submit a new intent.
2. A mode that you declare on the notary host. If the notary already runs with `HMZ_NOTARY_ALLOW_EMPTY_COLS=true`, it runs in `BALANCED` mode, so platforms that use this setting keep their current behavior. For the replacement, `antiRewind.seedMode`, see [Notary configuration](/pt-br/products/custody/deployment/reference/kms-notary#hot-failover-fields).
3. `STRICT`, by default. Platforms that completed Genesis before version 1.43 run in `STRICT` mode, and behave as before, until you change the mode.


New platforms declare the mode in the Genesis request. To use failover on a new platform, set the top-level `antiRewindMode` field to `BALANCED`:

```json
{
    "rootDomainSetup": { ... },
    "cryptoSetup": { ... },
    "antiRewindMode": "BALANCED"
}
```

For the full Genesis request body, see [Genesis payload reference](/pt-br/products/custody/governance/genesis/payload-reference#anti-rewind-mode).

Moving to STRICT
After you move to `STRICT`, the notary rejects every collection that its ARF has no record of. There's no simple way to check the ARF's coverage before you switch. The `hmz_notary_collection_revision` metric shows which collections the ARF holds, but reviewing it is difficult.

## How failover works

- Two notaries run. One is **serving**, and the other is on **standby**.
- If the serving notary is silent for the bridge's exclusivity period, the standby notary takes its place. The period is 15 minutes by default, and you can change it with `HMZ_NOTARY_BRIDGE_EXCLUSIVITY_PERIOD`. This **takeover** is final: the platform can't fail back to the old notary.
- Each notary keeps its own ARF. The standby notary starts with an empty ARF and accepts the database as it is at takeover. If someone rolled back the database just before a takeover, the new notary doesn't detect it. The [rewind after failover alert](#monitor-failover) covers this case.


A takeover needs two notaries and a platform mode of `BALANCED` or `DISABLED`. You can set `BALANCED` with a single notary, but no takeover can happen until you deploy the standby notary.

## Set up failover

1. **Upgrade the notaries, and then the bridge,** to version 1.43.
2. **Set the anti-rewind mode to `BALANCED`.** Submit a `v0_SetSystemProperty` intent, following the steps in [Manage intents and approvals](/pt-br/products/custody/governance/intents/manage-intents-and-approvals), with a payload similar to the following example:

```json
"payload": {
    "value": {
        "type": "AntiRewindModeProperty",
        "value": {
            "mode": "BALANCED"
        }
    },
    "type": "v0_SetSystemProperty"
}
```
The first time you set the mode, omit `revision`: the intent creates the property. To change the mode after that, add a `revision` field with the property's current revision, as when you [rotate the state review authority key](/pt-br/products/custody/operations-and-maintenance/backup-and-restore/register-a-key#rotate-the-key). An intent with a `revision` fails if the property doesn't exist, and an intent without one fails if it does.
3. **Confirm the mode.** Check that the intent completes and that [`GET /v1/properties`](/pt-br/products/custody/reference/api/openapi/systemproperties/getsystemproperties) returns `ANTI_REWIND_MODE` with the mode `BALANCED`.
4. **Deploy the standby notary,** ideally in another availability zone. The standby is a second instance of the same notary, with the same notary keys and HSM access. Give it its own persistent storage for its ARF and identity file:
  - **ARF:** The standby starts with an empty ARF, and records collections as it serves them.
  - **Identity file:** The notary generates its identity file at startup if the file is missing.
Configure it as a standby in your notary deployment. If you deploy the notary with the `approval-notary` Helm chart, set `standby.enabled` to `true`. See [Notary configuration](/pt-br/products/custody/deployment/reference/kms-notary#hot-failover-fields).
5. **Check that failover is ready:**
  - Both notaries are running.
  - The `notary.notary_instance` table has exactly one row with the status `SERVING`. Use only read-only queries on this table.
  - The `hmz_bridge_notary_silence_ratio` metric is `0`.


Set the exclusivity period to 10 minutes or more
The bridge's exclusivity period, `HMZ_NOTARY_BRIDGE_EXCLUSIVITY_PERIOD`, is 15 minutes by default. You can set it to any value of 10 minutes or more. Below 10 minutes, the bridge doesn't start. Choose a period longer than any planned restart or node reboot.

If your platform has a state review authority, don't declare the mode on the notary host with `antiRewind.seedMode`. Set it with the intent.

## Monitor failover

Create the following three alerts from the bridge and notary metrics.

| Alert | Fires when |
|  --- | --- |
| **Notary silent** | The serving notary has been silent for half the exclusivity period, 7.5 of 15 minutes by default. Unless the notary comes back, the standby notary takes over when the period ends. |
| **Takeover** | The standby notary took over in the last 5 minutes. Keep the alert active for 15 minutes. |
| **Rewind after failover** | For at least 1 minute, the serving notary holds a collection at a lower revision than the highest revision any notary published for it in the last 30 days. Ignore demoted notaries that are still running. |


| Metric | Source | Description |
|  --- | --- | --- |
| `hmz_bridge_notary_silence_ratio` | Bridge | How far the serving notary's silence has progressed through the exclusivity period, from `0` to `1`. |
| `hmz_bridge_notary_takeovers` | Bridge | Count of takeovers. |
| `hmz_bridge_notary_serving` | Bridge | The notary that the bridge serves, tagged with `notary_instance_id`. |
| `hmz_notary_collection_revision` | Notary | The revision of each collection, tagged with the notary's `notary_instance_id`. |


The rewind after failover alert looks back 30 days, so keep at least 30 days of metrics history. If the new notary first uses a collection after that window, the alert can't detect a rewind on it.

## Respond to an alert

### Notary silent

By default, let the takeover happen, and find out why the notary is silent.

Shut down the standby notary to prevent the takeover if either of the following applies:

- You're doing planned work that takes longer than the exclusivity period.
- You have a reason to doubt the database, such as a restore, a database incident, or revision mismatch rejections just before the silence.


### Takeover

1. **Keep the evidence:** the bridge and notary logs, the `notary.notary_instance` table, and 30 days of `hmz_notary_collection_revision`. Don't reset either notary yet.
2. **Compare the revisions of each collection.** The bridge's takeover log line gives the identities of both notaries. Filter `hmz_notary_collection_revision` on `notary_instance_id` to find the last revision on the old notary and the first revision on the new notary. Don't compare with the current database, which has changed since the takeover.
| Old revision minus new revision | Meaning |
|  --- | --- |
| 0 or less | Normal failover |
| 1 | One request is missing. This is normal if the request was in progress when the notary stopped and the notary never confirmed it. If the notary confirmed the request, it's a rewind. |
| 2 or more | Rewind |
3. **After a normal failover,** clean up the old notary's storage by deleting both its ARF and its identity file. It restarts as the new standby notary. The platform keeps using the notary that took over.


### Rewind, or if you're unsure

1. **Shut down the bridge.** The notary signs nothing, and no takeover can happen. Intents wait in the queue.
2. **Don't change anything else.** Don't reset a notary, wipe storage, or edit the `notary.notary_instance` table. The old notary's ARF is the evidence.
3. **Contact Ripple Support** with the evidence.


A rewind alert that resolves on its own doesn't mean the platform is safe. New activity eventually passes the old maximum revision.

## If the ARF is corrupt

A notary doesn't recover a corrupt ARF on its own, in any mode. Either:

- [Recover the ARF](/pt-br/products/custody/operations-and-maintenance/backup-and-restore/recover-anti-rewind-file) with the signed manifest procedure.
- In `BALANCED` or `DISABLED` mode, if you deployed a standby notary, shut down the notary with the corrupt ARF. The standby notary takes over after the exclusivity period. Then follow the [takeover](#takeover) steps.


## Related topics

- [Data integrity and audit trail](/pt-br/products/custody/overview/security/data-integrity-and-audit-trail)
- [Recover your anti-rewind file](/pt-br/products/custody/operations-and-maintenance/backup-and-restore/recover-anti-rewind-file)
- [Resilience planning](/pt-br/products/custody/deployment/planning/resilience)