Skip to content

This guide covers production-ready deployment patterns for Ripple Custody with any KMS platform (HSM, Cloud KMS, or MPC). These patterns ensure high availability, observability, and operational resilience.

Applies to: All KMS integrations — both HSM and MPC deployments.

Namespace segmentation

For production deployments, use namespace segmentation to isolate different components and enable multi-tenancy.

NamespacePurposeComponents
custody-coreCore infrastructureNotary, Core services
custody-vaultsVault workloadsVault instances
custody-networksNetwork adaptersBlockchain indexers, network bridges
custody-monitoringObservabilityPrometheus, Grafana, alerting

Benefits

  • Security isolation: Network policies restrict cross-namespace traffic.
  • Resource quotas: Limit resource consumption per namespace.
  • RBAC Boundaries: Fine-grained access control per component
  • Operational clarity: Clear separation of concerns.
  • Compliance: Meet regulatory requirements for multi-tenant isolation

Implementation

# Create namespaces
kubectl create namespace custody-core
kubectl create namespace custody-vaults
kubectl create namespace custody-networks
kubectl create namespace custody-monitoring

# Apply network policies (restrict cross-namespace traffic)
kubectl apply -f network-policies/

Helm deployment:

helm upgrade --install harmonize ./harmonize-custody \
  --namespace custody-core \
  --set components.notary.namespace=custody-core \
  --set harmonize.vaults.*.namespace=custody-vaults \
  -f production.yaml

High availability

Kubernetes components

Deploy multiple replicas of critical components. Note that the notary must run as a singleton (Merkle tree operations cannot be parallelized), while the vault supports multiple replicas for high availability:

# production.yaml
harmonize:
  vaults:
    "00000000-0000-0000-0000-000000000000":
      enabled: true
      replicas: 3  # Multiple vault replicas for HA

      # Pod anti-affinity (spread across nodes)
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchLabels:
                  app: harmonize-vault
              topologyKey: kubernetes.io/hostname

      resources:
        limits:
          cpu: 2000m
          memory: 512Mi
        requests:
          cpu: 500m
          memory: 256Mi

KMS high availability (HA)

KMS PlatformHA Strategy
Thales Luna HSMHA Group with multiple HSMs, automatic failover
BlockSafe HSMHSM clustering across racks/locations
Securosys Primus HSMHA Group with multiple HSMs, automatic failover
AWS CloudHSMMulti-AZ cluster deployment
IBM HSMGREP11 API with IBM HPVS, CEX6S clustering for secure components only
MPCDistributed nodes (3-of-4 threshold signing)

Refer to your specific KMS integration guide for detailed HA configuration.

Environment topologies: PROD vs DEV

Use different HSM topologies for production and non-production environments to balance security with cost efficiency.

Production environments require a full HA cluster with partition replication across multiple HSMs:

PROD Environment - HA Cluster

Thales Luna HA Group

Replication

Replication

Partition

HSM 1

Partition

HSM 2

Partition

HSM 3

Client Applications

Notary

Vault Replica

Vault Replica

PROD Environment - HA Cluster

Thales Luna HA Group

Replication

Replication

Partition

HSM 1

Partition

HSM 2

Partition

HSM 3

Client Applications

Notary

Vault Replica

Vault Replica

Key points:

  • 3+ HSMs in an HA group for redundancy
  • Partition replication across all HSMs
  • Multiple vault replicas for signing capacity
  • Automatic failover if any HSM becomes unavailable

Monitoring and alerting

Key metrics

CategoryMetrics to Monitor
KMS/HSM HealthConnection status, latency, error rate
Signing OperationsRequests/second, success rate, latency
Pod HealthCPU, memory, restarts, readiness
ApplicationTransaction throughput, queue depth

Prometheus ServiceMonitor

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: harmonize-notary
  namespace: custody-monitoring
spec:
  selector:
    matchLabels:
      app: harmonize-notary
  namespaceSelector:
    matchNames:
      - custody-core
  endpoints:
  - port: metrics
    interval: 30s
    path: /metrics
AlertConditionSeverity
KMSConnectionFailedHSM/KMS connection down > 1 minCritical
SigningLatencyHighp99 latency > 500ms for 5 minWarning
NotaryPodDownNotary pod unavailable > 1 minCritical
VaultReplicasLowVault replicas < 2Warning
HighErrorRateError rate > 1% for 5 minWarning

Certificate rotation

For mTLS-enabled deployments (Luna, BlockSafe):

Rotation procedure

  1. Generate new certificates using your HSM vendor tools
  2. Create new Kubernetes secret with updated certificates
  3. Perform rolling update via Helm to pick up new credentials
  4. Verify connectivity to HSM with new certificates
  5. Revoke old certificates after confirming new ones work

Zero-Downtime Rotation: With multiple vault replicas, rolling updates ensure continuous availability during certificate rotation. Note that the notary runs as a singleton, so plan a brief maintenance window for notary certificate rotation.

Backup and disaster recovery

What to backup

ComponentBackup MethodFrequency
HSM KeysVendor-specific backup (Luna: partition backup, AWS CloudHSM: cluster backup)Daily
Kubernetes SecretsEncrypted backup to secure storageDaily
Helm ValuesVersion control (Git)On change
ConfigurationInfrastructure as Code (Terraform/CDK)On change

Recovery testing

Perform quarterly disaster recovery testing:

  1. HSM Recovery: Restore keys to test partition/cluster
  2. Kubernetes recovery: Redeploy from IaC to test cluster.
  3. End-to-End Test: Verify signing operations work after recovery
  4. Document results: Update runbooks based on findings.

Advanced secret management

Recover a cold vault