# Production best practices This guide covers production-ready deployment patterns for Ripple Custody with any KMS platform (HSM, Cloud KMS, or MPC). These patterns ensure high availability, observability, and operational resilience. **Applies to**: All KMS integrations — both HSM and MPC deployments. ## Namespace segmentation For production deployments, use **namespace segmentation** to isolate different components and enable multi-tenancy. ### Recommended namespace structure | Namespace | Purpose | Components | | --- | --- | --- | | `custody-core` | Core infrastructure | Notary, Core services | | `custody-vaults` | Vault workloads | Vault instances | | `custody-networks` | Network adapters | Blockchain indexers, network bridges | | `custody-monitoring` | Observability | Prometheus, Grafana, alerting | ### Benefits - **Security isolation**: Network policies restrict cross-namespace traffic. - **Resource quotas**: Limit resource consumption per namespace. - **RBAC Boundaries**: Fine-grained access control per component - **Operational clarity**: Clear separation of concerns. - **Compliance**: Meet regulatory requirements for multi-tenant isolation ### Implementation ```bash # Create namespaces kubectl create namespace custody-core kubectl create namespace custody-vaults kubectl create namespace custody-networks kubectl create namespace custody-monitoring # Apply network policies (restrict cross-namespace traffic) kubectl apply -f network-policies/ ``` **Helm deployment**: ```bash helm upgrade --install harmonize ./harmonize-custody \ --namespace custody-core \ --set components.notary.namespace=custody-core \ --set harmonize.vaults.*.namespace=custody-vaults \ -f production.yaml ``` ## High availability ### Kubernetes components Deploy multiple replicas of critical components. Note that the notary must run as a singleton (Merkle tree operations cannot be parallelized), while the vault supports multiple replicas for high availability: ```yaml # production.yaml harmonize: vaults: "00000000-0000-0000-0000-000000000000": enabled: true replicas: 3 # Multiple vault replicas for HA # Pod anti-affinity (spread across nodes) affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchLabels: app: harmonize-vault topologyKey: kubernetes.io/hostname resources: limits: cpu: 2000m memory: 512Mi requests: cpu: 500m memory: 256Mi ``` ### KMS high availability (HA) | KMS Platform | HA Strategy | | --- | --- | | **Thales Luna HSM** | HA Group with multiple HSMs, automatic failover | | **BlockSafe HSM** | HSM clustering across racks/locations | | **Securosys Primus HSM** | HA Group with multiple HSMs, automatic failover | | **AWS CloudHSM** | Multi-AZ cluster deployment | | **IBM HSM** | GREP11 API with IBM HPVS, CEX6S clustering for secure components only | | **MPC** | Distributed nodes (3-of-4 threshold signing) | Refer to your specific KMS integration guide for detailed HA configuration. ### Environment topologies: PROD vs DEV Use different HSM topologies for production and non-production environments to balance security with cost efficiency. Production (HA cluster) **Production** environments require a full HA cluster with partition replication across multiple HSMs: ```mermaid flowchart TB subgraph prod["PROD Environment - HA Cluster"] subgraph apps["Client Applications"] direction LR notary["Notary"] vault1["Vault Replica"] vault2["Vault Replica"] end subgraph luna["Thales Luna HA Group"] direction LR hsm1["HSM 1"] hsm2["HSM 2"] hsm3["HSM 3"] subgraph p1["Partition"] part1[" "] end subgraph p2["Partition"] part2[" "] end subgraph p3["Partition"] part3[" "] end hsm1 --- p1 hsm2 --- p2 hsm3 --- p3 p1 <-->|"Replication"| p2 p2 <-->|"Replication"| p3 end end notary --> luna vault1 --> luna vault2 --> luna ``` **Key points:** - 3+ HSMs in an HA group for redundancy - Partition replication across all HSMs - Multiple vault replicas for signing capacity - Automatic failover if any HSM becomes unavailable DEV/UAT (single HSM) **Non-production** environments can use a single HSM with multiple partitions to reduce costs: ```mermaid flowchart TB subgraph dev["DEV Environment - Single HSM"] subgraph apps1["DEV Applications"] notary1["Notary"] vault1["Vault"] end subgraph apps2["UAT Applications"] notary2["Notary"] vault2["Vault"] end subgraph luna["Thales Luna HSM"] partDev["Partition: DEV"] partUat["Partition: UAT"] end end notary1 --> partDev vault1 --> partDev notary2 --> partUat vault2 --> partUat ``` **Key points:** - Single HSM with separate partitions per environment - DEV and UAT isolated via partition boundaries - Cost-effective for non-production workloads - No HA - acceptable for development/testing ## Monitoring and alerting ### Key metrics | Category | Metrics to Monitor | | --- | --- | | **KMS/HSM Health** | Connection status, latency, error rate | | **Signing Operations** | Requests/second, success rate, latency | | **Pod Health** | CPU, memory, restarts, readiness | | **Application** | Transaction throughput, queue depth | ### Prometheus ServiceMonitor ```yaml apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: harmonize-notary namespace: custody-monitoring spec: selector: matchLabels: app: harmonize-notary namespaceSelector: matchNames: - custody-core endpoints: - port: metrics interval: 30s path: /metrics ``` ### Recommended alerts | Alert | Condition | Severity | | --- | --- | --- | | `KMSConnectionFailed` | HSM/KMS connection down > 1 min | Critical | | `SigningLatencyHigh` | p99 latency > 500ms for 5 min | Warning | | `NotaryPodDown` | Notary pod unavailable > 1 min | Critical | | `VaultReplicasLow` | Vault replicas < 2 | Warning | | `HighErrorRate` | Error rate > 1% for 5 min | Warning | ## Certificate rotation For mTLS-enabled deployments (Luna, BlockSafe): ### Rotation procedure 1. **Generate new certificates** using your HSM vendor tools 2. **Create new Kubernetes secret** with updated certificates 3. **Perform rolling update** via Helm to pick up new credentials 4. **Verify connectivity** to HSM with new certificates 5. **Revoke old certificates** after confirming new ones work **Zero-Downtime Rotation**: With multiple vault replicas, rolling updates ensure continuous availability during certificate rotation. Note that the notary runs as a singleton, so plan a brief maintenance window for notary certificate rotation. ## Backup and disaster recovery ### What to backup | Component | Backup Method | Frequency | | --- | --- | --- | | **HSM Keys** | Vendor-specific backup (Luna: partition backup, AWS CloudHSM: cluster backup) | Daily | | **Kubernetes Secrets** | Encrypted backup to secure storage | Daily | | **Helm Values** | Version control (Git) | On change | | **Configuration** | Infrastructure as Code (Terraform/CDK) | On change | ### Recovery testing Perform quarterly disaster recovery testing: 1. **HSM Recovery**: Restore keys to test partition/cluster 2. **Kubernetes recovery**: Redeploy from IaC to test cluster. 3. **End-to-End Test**: Verify signing operations work after recovery 4. **Document results**: Update runbooks based on findings. ## Related guides - [KMS integration overview](/products/custody/deployment/integrate-kms/overview) - All supported KMS platforms - [Advanced secret management](/products/custody/deployment/integrate-kms/advanced-secret-management) - ESO, Vault Agent, AWS Secrets Manager - [Domain injection](/products/custody/governance/domains/inject-a-domain) - Advanced key isolation patterns - [Thales Luna HSM](/products/custody/deployment/integrate-kms/on-premise-hsm/thales-luna) - [BlockSafe HSM](/products/custody/deployment/integrate-kms/on-premise-hsm/blocksafe) - [AWS CloudHSM](/products/custody/deployment/integrate-kms/cloud-hsm/aws-cloudhsm) - [IBM LinuxONE](/products/custody/deployment/integrate-kms/on-premise-hsm/ibm-linuxone) - [MPC overview](/products/custody/deployment/integrate-kms/mpc/overview) - [MPC backups](/products/custody/deployment/integrate-kms/mpc/backups)