# Resilience planning

This page defines the high availability (HA) and disaster recovery (DR) requirements for Ripple Custody components and outlines your implementation responsibilities.

Platform resilience depends on both Ripple's application design and your infrastructure implementation. All core platform components are containerized and deployed into your Kubernetes/OpenShift environment, relying on the container orchestration platform for scaling and resilience.

## Core concepts

| Concept | Definition |
|  --- | --- |
| **High availability (HA)** | Automated, real-time protection against single-component failures (pod crash, server reboot) using redundancy. |
| **Disaster recovery (DR)** | Manual, procedural recovery from catastrophic failures (data center loss, core component failure) using backups. |


For conceptual background, see:

- [Security](/products/custody/v1.36/overview/security-model) - Zero-trust model and component isolation.
- [Data integrity](/products/custody/v1.36/overview/security/data-integrity-and-audit-trail) - Merkle tree and anti-rewind file (ARF) protection.


## Component categories

| Category | Definition |
|  --- | --- |
| **Client-provided** | You deploy and manage your own instance(s) using your vendor of choice. |
| **Client-deployed** | Ripple provides the binary; you deploy following best practices for production support. |
| **Client-managed** | You are fully responsible for the component's lifecycle, HA, and DR. |
| **Core dependency** | Required for a functional Ripple Custody cluster. |
| **Secure component** | Core dependency requiring encryption of data in-flight, at-rest, and in-use (e.g., enclave technology). |
| **Recommended** | Not required but highly recommended based on typical deployments. |
| **Optional** | Not required; enable if the specific functionality is needed. |


## Component summary

| Component | Description | Category | Instances | State | HA | DR |
|  --- | --- | --- | --- | --- | --- | --- |
| PostgreSQL Database | Primary data store holding policies, accounts, ledger states, and wrapped key material. HSM-wrapped keys are stored here. | Client-provided | 3-node cluster | Stateful | [Details](#postgresql-database) | [Details](#postgresql-database) |
| AMQP Message Broker | Asynchronous messaging, queuing API requests, and distributing events. Typically RabbitMQ. | Client-provided | 3-node cluster | Stateful | [Details](#amqp-message-broker) | [Details](#amqp-message-broker) |
| Notary | Evaluates governance policies, validates trusted state with the Merkle tree and ARF, and signs trusted-state attestations. | Client-deployed (secure) | 1 (Singleton) | Stateful | [Details](#notary) | [Details](#notary) |
| Notary Bridge | Proxy between workflow services and Notary; moves approval and attestation messages across the protected boundary. | Client-deployed | 1 (Singleton) | Stateless | Singleton | Helm redeploy |
| Vault | Verifies notary-attested operation queries, builds ledger-specific transactions, performs protected signing through your KMS, and returns signed transactions to the platform for broadcast. Deploy via Helm, Nitro Enclaves, HPVS, GCP, Azure, or Red Hat Linux. | Client-deployed (secure) | 2+ replicas | Stateless | [Details](#vault) | [Details](#vault) |
| Vault Bridge | Web UI for fully air-gapped signing. Provides an interface to drag-and-drop transaction files for signing and outputs signed transactions. | Client-deployed | 1 per vault | Stateless | Runs with vault | Helm redeploy |
| API Gateway | Single entry point for all API requests; validates JWTs, routes traffic. Public API for clients, signed internal API for components. HTTP only. | Client-deployed | 3+ replicas | Stateless | Multi-replica | Helm redeploy |
| GraphQL | Data layer for Web UI v2; provides GraphQL API for improved query performance. Same role as API Gateway for v2 clients. | Client-deployed | 3+ replicas | Stateless | Multi-replica | Helm redeploy |
| Ledger Accounting | Manages system accounting, balances, transactions, and compliance processes (e.g., quarantine management). | Client-deployed | 1 (Singleton) | Stateless | Singleton | Helm redeploy |
| Indexers (Legacy & UIS) | Chain-specific services monitoring blockchain nodes; fetch blocks, parse data, identify relevant transactions. UIS (v1.26+) uses component-based architecture with bulkhead pattern. | Client-deployed | 1 per ledger | Stateful (DB) | K8s restart | Helm redeploy |
| Event Processing | Consumes AMQP messages; processes events and async tasks (e.g., updating transaction state after indexing). May require singleton configuration depending on queue setup to avoid duplicate processing. | Client-deployed | 1-2 replicas | Stateless | [Details](#event-processing) | Helm redeploy |
| Authentication Server | Legacy authentication orchestrator; handles mobile signing authentication. Generates/validates JWTs for API Gateway. | Client-deployed | 2+ replicas | Stateless | Multi-replica | Helm redeploy |
| Keycloak (SSO) | General-purpose SSO provider; open-source IAM with SSO, user federation, and identity brokering. | Client-deployed | 3-5 replicas | Stateless | Multi-replica | Helm redeploy |
| Web UI | Back-office interface for admins/operators. Connects via HTTP REST. v2 uses GraphQL for performance. | Client-deployed | 2+ replicas | Stateless | Multi-replica | Helm redeploy |
| Notification Server | Pushes notifications to Web UI and apps via WebSocket. Only relevant for legacy Authentication Server deployments. | Client-deployed | 3-5 replicas | Stateless | Multi-replica | Helm redeploy |
| Core Extensions | Handles SMTP emails, user invitations, and related notifications. Works alongside Notification Server. | Client-deployed | 2+ replicas | Stateless | Multi-replica | Helm redeploy |
| KMS (HSM/MPC) | Client-managed hardware/software protecting notary keys, vault keys, blockchain account keys, and configured system-signing keys. HSM or MPC cluster for protected signing operations. | Client-managed | Varies | Stateful | [Details](#kms-hsm-or-mpc) | [Details](#kms-hsm-or-mpc) |
| Blockchain Nodes | Full/archive nodes for each supported chain. Indexers connect directly to read on-chain data. | Client-managed | Redundant | Stateful | [Details](#blockchain-nodes) | [Details](#blockchain-nodes) |
| Compliance Adapter | Optional connection to third-party screening APIs (Chainalysis, Elliptic) for transaction screening and compliance. | Client-deployed | 1-2 replicas | Stateless | Multi-replica | Helm redeploy |


## Component details

### PostgreSQL Database

**HA requirements:**

- Implement a production-grade 3-node cluster with synchronous replication
- Use automated failover (Patroni, AWS RDS Multi-AZ, or equivalent)


**DR requirements:**

- Standard database backups with defined RPO/RTO


**Critical:** Database backups must be time-synced with the anti-rewind file (ARF) backup. A mismatch will be detected as a replay attack and permanently halt the platform.

**Critical:** The database contains HSM-wrapped key material. Seed generation uses the HSM's secure random number generator, and the wrapped keys are stored in the database. Dropping the production database without a backup results in **permanent, unrecoverable seed loss**.

### AMQP Message Broker

**HA requirements:**

- Deploy a multi-node broker cluster (3-node RabbitMQ recommended)
- Use persistent, replicated queues (Quorum Queues)
- A fully managed instance is highly recommended


**DR requirements:**

- In-flight messages are ephemeral and not backed up
- In a DR scenario, rebuild the broker as a new, empty cluster


### Notary

**HA requirements:**

- Must run as a singleton (Merkle tree operations cannot be parallelized)
- Some clients run multiple replicas in HA mode, but this disables the ARF file and reduces security


**DR requirements:**

- Rebuild ARF using a signed manifest verified by the State Review Authority public key
- Requires PostgreSQL database to be restored first
- See [Recover ARF](/products/custody/v1.36/operations-and-maintenance/backup-and-restore/recover-anti-rewind-file) and [Register State Review Authority Key](/products/custody/v1.36/operations-and-maintenance/backup-and-restore/register-a-key)


### Vault

**HA requirements:**

- Deploy 2+ replicas for high availability
- Vault is stateless and does not use the database for runtime operations
- Ensure the backing HSM has an HA cluster


**DR requirements:**

- Vault is stateless; reconnect to the same HSM cluster after redeployment
- Redeploy via Helm (or Vault Deployer for Nitro) and reconnect to your KMS
- See platform-specific guidance: [Luna HSM](/products/custody/v1.36/deployment/integrate-kms/on-premise-hsm/thales-luna), [AWS CloudHSM](/products/custody/v1.36/deployment/integrate-kms/cloud-hsm/aws-cloudhsm), [IBM LinuxONE](/products/custody/v1.36/deployment/integrate-kms/on-premise-hsm/ibm-linuxone), [MPC](/products/custody/v1.36/deployment/integrate-kms/mpc/overview)


**Multi-AZ vault + HSM architecture:**

The following diagram shows a production-grade multi-AZ deployment with vault replicas distributed across availability zones and mesh connectivity to an HSM HA group:


```mermaid
flowchart TB
    subgraph app["Application Layer"]
        core["Core Platform"]
    end

    subgraph cloud["Cloud Provider Environment"]
        lb["Load Balancer / Proxy"]

        subgraph az1["AZ 1"]
            v1["Vault Replica<br/>Region Alpha"]
        end

        subgraph az2["AZ 2"]
            v2["Vault Replica<br/>Region Beta"]
        end

        subgraph az3["AZ 3"]
            v3["Vault Replica<br/>Region Gamma"]
        end
    end

    subgraph hsm["HSM HA Group"]
        h1["HSM Site Alpha"]
        h2["HSM Site Beta"]
        h3["HSM Site Gamma"]
    end

    v1 --> lb
    v2 --> lb
    v3 --> lb
    lb -->|"Fetch transactions"| core

    v1 -->|"Primary"| h1
    v1 -.->|"Standby"| h2
    v1 -.->|"Standby"| h3

    v2 -.->|"Standby"| h1
    v2 -->|"Primary"| h2
    v2 -.->|"Standby"| h3

    v3 -.->|"Standby"| h1
    v3 -.->|"Standby"| h2
    v3 -->|"Primary"| h3
```

**Key architecture points:**

- Each vault replica has a **primary** HSM connection and **standby** connections to all other HSMs
**Requirements**:
- At least 2 Luna HSM appliances (one partition per HSM for replication)
- Matching partition configurations on each HSM


### KMS (HSM or MPC)

**Recommended instances:**

- HSM: 2+ nodes in HA configuration
- MPC: 3-of-4 for most operations (4-of-4 for account creation)


**HA requirements:**

- **HSM:** Use vendor-specific HA groups/clustering
- **MPC:** Natively resilient as long as the m-of-n threshold is met


**DR requirements:**

- You are responsible for vendor-specific on/off-site secure key backup procedures (offline HSM master keys, MPC shard backups, etc.)
- See platform-specific guides: [Luna HSM](/products/custody/v1.36/deployment/integrate-kms/on-premise-hsm/thales-luna), [AWS CloudHSM](/products/custody/v1.36/deployment/integrate-kms/cloud-hsm/aws-cloudhsm), [IBM LinuxONE](/products/custody/v1.36/deployment/integrate-kms/on-premise-hsm/ibm-linuxone), [MPC](/products/custody/v1.36/deployment/integrate-kms/mpc/overview)
- For MPC disaster recovery (deploying to a secondary AWS region), see [MPC disaster recovery](/products/custody/v1.36/deployment/integrate-kms/mpc/disaster-recovery)


### Blockchain nodes

**HA requirements:**

- Use a load balancer to provide a single, highly-available endpoint
- Consider multiple backing vendors for redundancy


**DR requirements:**

- Vendor dependent
- See [Blockchain Node Planning](/products/custody/v1.36/deployment/planning/blockchain-nodes) for detailed guidance


### Event processing

**HA requirements:**

- May require singleton configuration depending on queue setup
- Running 2 replicas reading the same messages could create duplicate processing depending on queue configuration


**DR requirements:**

- Stateless; redeploy via Helm


## Checklist: Before you proceed

### Decisions made

- [ ] HA strategy selected for PostgreSQL (e.g., 3-node Patroni, RDS Multi-AZ)
- [ ] HA strategy selected for RabbitMQ (e.g., 3-node cluster with quorum queues)
- [ ] DR strategy confirmed (backup locations, recovery time objectives)
- [ ] State Review Authority key custodian identified for ARF recovery


### Artifacts ready

- [ ] Component responsibility matrix documented internally
- [ ] HA/DR runbooks drafted for each client-managed component
- [ ] ARF backup procedure time-synced with database backups


### Configuration reference

See [Back up, restore, and recover](/products/custody/v1.36/operations-and-maintenance/backup-and-restore) for disaster recovery steps.

## Next step

- [Key management planning](/products/custody/v1.36/deployment/planning/key-management) - Choose your Key management system (HSM or MPC).