# Advanced secret management

This guide covers advanced secret management patterns for Ripple Custody, including **External Secrets Operator (ESO)**, **HashiCorp Vault Agent Injector**, and **AWS Secrets Manager** integration.

**Target audience**: This guide is intended for platform engineers and DevOps teams managing production Ripple Custody deployments with enterprise secret management requirements.

## Table of contents

1. [Overview](#overview)
2. [Why advanced secret management](#what-is-advanced-secret-management)
3. [External Secrets Operator (ESO)](#external-secrets-operator-eso)
4. [HashiCorp Vault Agent Injector](#hashicorp-vault-agent-injector)
5. [AWS Secrets Manager](#aws-secrets-manager)
6. [Comparison: ESO vs Vault Injector vs AWS Secrets Manager](#comparison-eso-vs-vault-injector-vs-aws-secrets-manager)
7. [Security best practices](#security-best-practices)
8. [Troubleshooting](#troubleshooting)
9. [Platform-specific examples](#platform-specific-examples)


## Overview

### What is advanced secret management?

Advanced secret management involves using external secret stores (like HashiCorp Vault or AWS Secrets Manager) to centrally manage sensitive credentials, with automatic synchronization to Kubernetes Secrets or direct injection into application pods.

### Benefits

**Centralized secret management**:

- Single source of truth for all secrets
- Centralized access control and audit logging
- Consistent secret rotation policies


**Automatic rotation**:

- Secrets are automatically updated in Kubernetes
- No manual intervention required
- Reduced risk of stale credentials


**Enhanced security**:

- Secrets never stored in Git or Helm values
- Fine-grained access control via IAM/RBAC
- Encryption at rest and in transit


**Compliance**:

- Audit trail for all secret access
- Meets regulatory requirements (SOC 2, PCI-DSS, etc.)
- Separation of duties (platform team vs application team)


## External Secrets Operator (ESO)

### Overview

The **External Secrets Operator (ESO)** synchronizes secrets from external secret stores (HashiCorp Vault, AWS Secrets Manager, Azure Key Vault) to Kubernetes Secrets.

**How it works**:


```mermaid
flowchart LR
    vault["HashiCorp Vault<br/>secret/app/luna-creds"]
    es["ExternalSecret"]
    ks["Kubernetes Secret"]
    pod["Application Pod"]

    vault -->|ESO polls| es
    es -->|creates| ks
    ks -->|volumeMount| pod
```

### Prerequisites

**1. Install External Secrets Operator**:


```bash
# Add Helm repository
helm repo add external-secrets https://charts.external-secrets.io
helm repo update

# Install ESO
helm install external-secrets \
  external-secrets/external-secrets \
  --namespace external-secrets-system \
  --create-namespace \
  --set installCRDs=true

# Verify installation
kubectl get pods -n external-secrets-system
```

**2. Configure HashiCorp Vault**:


```bash
# Enable Kubernetes auth method
vault auth enable kubernetes

# Configure Kubernetes auth
vault write auth/kubernetes/config \
  kubernetes_host="https://kubernetes.default.svc:443"

# Create policy for ESO
vault policy write eso-reader - <<EOF
path "secret/data/app/*" {
  capabilities = ["read"]
}
EOF

# Create role for ESO
vault write auth/kubernetes/role/eso-reader \
  bound_service_account_names=external-secrets \
  bound_service_account_namespaces=custody-core \
  policies=eso-reader \
  ttl=1h
```

**3. Store Secrets in Vault**:


```bash
# Example: Store Luna HSM credentials
vault kv put secret/app/luna-creds \
  ca.pem=@ca.pem \
  client.pem=@client.pem \
  client.key=@client.key \
  pin="your-hsm-pin"

# Verify
vault kv get secret/app/luna-creds
```

### Step 1: Create SecretStore

The **SecretStore** defines the connection to HashiCorp Vault.

**Create `secretstore.yaml`**:


```yaml
# secretstore.yaml
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: vault-backend
  namespace: custody-core
spec:
  provider:
    vault:
      # Vault server address
      server: "https://vault.example.com"

      # KV-v2 secrets engine mount path
      path: "secret"
      version: "v2"

      # Authentication method
      auth:
        kubernetes:
          # Kubernetes auth mount path in Vault
          mountPath: "kubernetes"

          # Vault role to use
          role: "eso-reader"

          # Service account for authentication
          serviceAccountRef:
            name: external-secrets
```

**Apply**:


```bash
kubectl apply -f secretstore.yaml

# Verify
kubectl get secretstore -n custody-core
```

### Step 2: Create ExternalSecret

The **ExternalSecret** defines which secrets to sync and how to map them.

**Create `externalsecret-luna.yaml`**:


```yaml
# externalsecret-luna.yaml
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: external-kms-luna-secret
  namespace: custody-core
spec:
  # Refresh interval (how often to check Vault for updates)
  refreshInterval: 1h

  # Reference to SecretStore
  secretStoreRef:
    name: vault-backend
    kind: SecretStore

  # Target Kubernetes Secret
  target:
    name: kms-luna-credentials
    creationPolicy: Owner

  # Data mapping: Vault key -> Kubernetes Secret key
  data:
    # ca.pem
    - secretKey: ca.pem
      remoteRef:
        key: app/luna-creds
        property: ca.pem

    # client.pem
    - secretKey: client.pem
      remoteRef:
        key: app/luna-creds
        property: client.pem

    # client.key
    - secretKey: client.key
      remoteRef:
        key: app/luna-creds
        property: client.key

    # PIN
    - secretKey: KMS_LUNA_PIN
      remoteRef:
        key: app/luna-creds
        property: pin
```

**Apply**:


```bash
kubectl apply -f externalsecret-luna.yaml

# Verify ExternalSecret
kubectl get externalsecret -n custody-core

# Verify Kubernetes Secret was created
kubectl get secret kms-luna-credentials -n custody-core

# View secret keys (not values)
kubectl describe secret kms-luna-credentials -n custody-core
```

### Step 3: Reference secret in Helm values

Update your Helm values to reference the ESO-managed secret.

**Example `values.yaml`**:


```yaml
# values.yaml
kms:
  kind: "luna"

  # Reference existing secret created by ESO
  existingSecret: "kms-luna-credentials"

  # Optional: Specify custom key names if different from defaults
  existingSecretKeys:
    pin: "KMS_LUNA_PIN"
    caPem: "ca.pem"
    clientPem: "client.pem"
    clientKey: "client.key"
```

**Deploy**:


```bash
helm upgrade --install harmonize ./harmonize-custody \
  --namespace custody-core \
  -f values.yaml
```

### Step 4: Verify secret rotation

ESO automatically updates the Kubernetes Secret when the Vault secret changes.

**Test rotation**:


```bash
# Update secret in Vault
vault kv put secret/app/luna-creds \
  ca.pem=@ca-new.pem \
  client.pem=@client-new.pem \
  client.key=@client-new.key \
  pin="new-hsm-pin"

# Wait for refreshInterval (default: 1h) or force sync
kubectl annotate externalsecret external-kms-luna-secret \
  force-sync=$(date +%s) \
  -n custody-core

# Verify secret was updated
kubectl get secret kms-luna-credentials -n custody-core -o yaml

# Restart pods to pick up new secret
kubectl rollout restart deployment/harmonize-notary -n custody-core
```

## HashiCorp Vault Agent Injector

### Overview

The **Vault Agent Injector** uses a sidecar pattern to inject secrets directly into application pods without creating Kubernetes Secrets.

**How it works**:


```mermaid
flowchart LR
    agent["vault-agent sidecar"]
    vol["/vault/secrets/"]
    kms["kms-luna container"]

    agent -->|writes secrets| vol
    vol -->|volumeMount| kms
```

### Prerequisites

**1. Install Vault Agent Injector**:


```bash
# Add Helm repository
helm repo add hashicorp https://helm.releases.hashicorp.com
helm repo update

# Install Vault with injector
helm install vault hashicorp/vault \
  --namespace vault-system \
  --create-namespace \
  --set "injector.enabled=true" \
  --set "server.dev.enabled=true"

# Verify installation
kubectl get pods -n vault-system
```

**2. Configure Vault** (same as ESO prerequisites)

### Step 1: Annotate pod for injection

Add Vault annotations to your Helm values to enable injection.

**Example `values.yaml`**:


```yaml
# values.yaml
kms:
  kind: "luna"

  # Vault Agent Injector annotations
  podAnnotations:
    vault.hashicorp.com/agent-inject: "true"
    vault.hashicorp.com/role: "eso-reader"
    vault.hashicorp.com/agent-inject-secret-ca.pem: "secret/data/app/luna-creds"
    vault.hashicorp.com/agent-inject-template-ca.pem: |
      {{- with secret "secret/data/app/luna-creds" -}}
      {{ .Data.data.ca.pem }}
      {{- end -}}

    vault.hashicorp.com/agent-inject-secret-client.pem: "secret/data/app/luna-creds"
    vault.hashicorp.com/agent-inject-template-client.pem: |
      {{- with secret "secret/data/app/luna-creds" -}}
      {{ .Data.data.client.pem }}
      {{- end -}}

    vault.hashicorp.com/agent-inject-secret-client.key: "secret/data/app/luna-creds"
    vault.hashicorp.com/agent-inject-template-client.key: |
      {{- with secret "secret/data/app/luna-creds" -}}
      {{ .Data.data.client.key }}
      {{- end -}}

    vault.hashicorp.com/agent-inject-secret-pin: "secret/data/app/luna-creds"
    vault.hashicorp.com/agent-inject-template-pin: |
      {{- with secret "secret/data/app/luna-creds" -}}
      {{ .Data.data.pin }}
      {{- end -}}

  # Environment variables pointing to injected files
  extraEnvVars:
    - name: KMS_LUNA_CA_PEM_PATH
      value: "/vault/secrets/ca.pem"
    - name: KMS_LUNA_CLIENT_PEM_PATH
      value: "/vault/secrets/client.pem"
    - name: KMS_LUNA_CLIENT_KEY_PATH
      value: "/vault/secrets/client.key"
    - name: KMS_LUNA_PIN
      valueFrom:
        secretKeyRef:
          name: vault-injected-pin
          key: pin
```

### Step 2: Deploy and verify


```bash
# Deploy with Vault Agent Injector
helm upgrade --install harmonize ./harmonize-custody \
  --namespace custody-core \
  -f values.yaml

# Verify vault-agent sidecar was injected
kubectl get pods -n custody-core

# Expected output:
# NAME                                READY   STATUS    RESTARTS   AGE
# harmonize-notary-<hash>             2/2     Running   0          1m
#                                     ^^^
#                                     2 containers: kms-luna + vault-agent

# Verify secrets were injected
kubectl exec -it -n custody-core deployment/harmonize-notary -c kms-luna -- \
  ls -la /vault/secrets/

# Expected output:
# -rw-r--r-- 1 vault vault 1234 Jun 26 10:00 ca.pem
# -rw-r--r-- 1 vault vault 1234 Jun 26 10:00 client.pem
# -rw-r--r-- 1 vault vault 1234 Jun 26 10:00 client.key
# -rw-r--r-- 1 vault vault   12 Jun 26 10:00 pin
```

## AWS Secrets Manager

### Overview

**AWS Secrets Manager** is used by the Vault Deployer UI to store deployment configuration and HSM credentials for AWS Nitro enclave deployments.

**Storage path structure**:


```
/ripple/custody/<tenant-id>/<instance-id>/<construct-name>/<construct-id>/
```

Where:

- `<tenant-id>`: Your Ripple Custody tenant identifier
- `<instance-id>`: Your Ripple Custody instance identifier
- `<construct-name>`: Always `vault` for vault deployments
- `<construct-id>`: The vault ID (UUID)


### Secrets stored

**1. `config` key**: Deployment configuration


```json
{
  "deployment": {
    "vault": {
      "release-name": "vault-01",
      "construct-id": "00000000-0000-0000-0000-000000000001"
    }
  },
  "custody": {
    "environment": "https://api.your-instance.custody.example.com/"
  },
  "infrastructure": {
    "aws": {
      "region": "us-east-1",
      "ec2-iam-role": "vault-role",
      "vpc": {
        "create-new": false,
        "id": "vpc-abc123",
        "subnet-id": "subnet-def456"
      }
    }
  }
}
```

**2. `hsm` key**: HSM credentials


```json
{
  "kms-awscloudhsm": "cluster-a1b2c3d4e5f",
  "kms-awscloudhsm-cacert": "-----BEGIN CERTIFICATE-----\nMI...==\n-----END CERTIFICATE-----\n",
  "kms-password": "CryptoUserPassword",
  "kms-soft-wrapping-key": "0123456789abcdef...",
  "kms-username": "vault-crypto-user",
  "kms-wrap-key": "0000000000000001",
  "kms-awscloudhsm-clientcert": "-----BEGIN CERTIFICATE-----\n...",
  "kms-awscloudhsm-clientkey": "-----BEGIN PRIVATE KEY-----\n..."
}
```

### Security best practices

**1. Limit Access**:


```bash
# Create IAM policy for limited access
cat > secrets-manager-policy.json <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "secretsmanager:GetSecretValue",
        "secretsmanager:DescribeSecret"
      ],
      "Resource": "arn:aws:secretsmanager:us-east-1:123456789012:secret:/ripple/custody/*"
    }
  ]
}
EOF

# Attach policy to role
aws iam put-role-policy \
  --role-name vault-deployer-role \
  --policy-name SecretsManagerAccess \
  --policy-document file://secrets-manager-policy.json
```

**2. Enable Audit Logging**:


```bash
# Enable CloudTrail for Secrets Manager
aws cloudtrail create-trail \
  --name secrets-manager-audit \
  --s3-bucket-name my-audit-bucket

# Start logging
aws cloudtrail start-logging \
  --name secrets-manager-audit
```

**3. Recovery Window**:

AWS Secrets Manager provides a **7-day recovery window** for deleted secrets.


```bash
# Delete secret (with recovery window)
aws secretsmanager delete-secret \
  --secret-id /ripple/custody/tenant/instance/vault/vault-id/config \
  --recovery-window-in-days 7

# Restore secret within 7 days
aws secretsmanager restore-secret \
  --secret-id /ripple/custody/tenant/instance/vault/vault-id/config
```

### Accessing secrets


```bash
# Get config secret
aws secretsmanager get-secret-value \
  --secret-id /ripple/custody/<tenant-id>/<instance-id>/vault/<vault-id>/config \
  --query SecretString \
  --output text | jq .

# Get HSM credentials
aws secretsmanager get-secret-value \
  --secret-id /ripple/custody/<tenant-id>/<instance-id>/vault/<vault-id>/hsm \
  --query SecretString \
  --output text | jq .
```

**Important**: While AWS Secrets Manager is used during vault **deployment** automation, Ripple Custody vault itself **does not** use AWS Secrets Manager to store wallet private keys. Wallet keys are encrypted with AWS CloudHSM and stored in the Ripple Custody database.

## Comparison: ESO vs Vault Injector vs AWS Secrets Manager

| Feature | External Secrets Operator | Vault Agent Injector | AWS Secrets Manager |
|  --- | --- | --- | --- |
| **Secret Storage** | Kubernetes Secret | Files in shared volume | AWS Secrets Manager |
| **Rotation** | Automatic (refreshInterval) | Automatic (lease renewal) | Manual or Lambda-based |
| **Kubernetes Native** | Yes (CRDs) | Yes (annotations) | No (AWS-specific) |
| **Multi-Cloud** | Yes (Vault, AWS, Azure) | Yes (Vault only) | No (AWS only) |
| **Complexity** | Low | Medium | Low |
| **Performance** | Good (cached in K8s Secret) | Excellent (direct injection) | Good (cached in AWS) |
| **Security** | Good (K8s RBAC) | Excellent (no K8s Secret) | Good (IAM policies) |
| **Use Case** | General-purpose | High-security environments | AWS Nitro enclaves |
| **Ripple Custody Support** | Recommended | Supported | Vault Deployer UI only |


### Recommendations

**Use External Secrets Operator (ESO) when**:

- You need multi-cloud support
- You want Kubernetes-native secret management
- You need automatic rotation with minimal complexity
- **Recommended for most Ripple Custody deployments**


**Use Vault Agent Injector when**:

- You have strict security requirements (no K8s Secrets)
- You need fine-grained secret access control
- You want secrets to be ephemeral (not persisted in K8s)
- You're already using HashiCorp Vault extensively


**Use AWS Secrets Manager when**:

- You're deploying vaults in AWS Nitro enclaves
- You're using the Vault Deployer UI
- You need AWS-native secret management
- **Only for Vault Deployer UI deployments**


## Security best practices

### Principle of least privilege

**1. Vault Policies**:


```hcl
# eso-reader-policy.hcl
# Read-only access to specific paths
path "secret/data/app/luna-creds" {
  capabilities = ["read"]
}

# Deny all other paths
path "secret/*" {
  capabilities = ["deny"]
}
```

**2. Kubernetes RBAC**:


```yaml
# rbac.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: secret-reader
  namespace: custody-core
rules:
- apiGroups: [""]
  resources: ["secrets"]
  resourceNames: ["kms-luna-credentials"]
  verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: secret-reader-binding
  namespace: custody-core
subjects:
- kind: ServiceAccount
  name: harmonize-notary
  namespace: custody-core
roleRef:
  kind: Role
  name: secret-reader
  apiGroup: rbac.authorization.k8s.io
```

### Secret rotation

**1. Automated Rotation with ESO**:


```yaml
# externalsecret.yaml
spec:
  refreshInterval: 1h  # Check Vault every hour

  # Optional: Force immediate sync on annotation change
  metadata:
    annotations:
      force-sync: "2025-06-26T10:00:00Z"
```

**2. Rotation Procedure**:


```bash
# 1. Update secret in Vault
vault kv put secret/app/luna-creds \
  ca.pem=@ca-new.pem \
  client.pem=@client-new.pem \
  client.key=@client-new.key \
  pin="new-pin"

# 2. Force ESO sync (optional, or wait for refreshInterval)
kubectl annotate externalsecret external-kms-luna-secret \
  force-sync=$(date +%s) \
  -n custody-core

# 3. Restart pods to pick up new secret
kubectl rollout restart deployment/harmonize-notary -n custody-core

# 4. Verify new secret is in use
kubectl logs -n custody-core deployment/harmonize-notary | grep "Connected to HSM"
```

### Audit logging

**1. Vault Audit Logging**:


```bash
# Enable audit logging
vault audit enable file file_path=/vault/logs/audit.log

# Query audit logs
cat /vault/logs/audit.log | jq 'select(.request.path == "secret/data/app/luna-creds")'
```

**2. Kubernetes Audit Logging**:


```yaml
# audit-policy.yaml
apiVersion: audit.k8s.io/v1
kind: Policy
rules:
- level: RequestResponse
  resources:
  - group: ""
    resources: ["secrets"]
  namespaces: ["custody-core"]
```

### Encryption at rest

**1. Vault Transit Engine**:


```bash
# Enable transit engine
vault secrets enable transit

# Create encryption key
vault write -f transit/keys/custody-secrets

# Encrypt secret before storing
vault write transit/encrypt/custody-secrets \
  plaintext=$(base64 <<< "my-secret-value")
```

**2. Kubernetes Encryption at Rest**:


```yaml
# encryption-config.yaml
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      - aescbc:
          keys:
            - name: key1
              secret: <base64-encoded-32-byte-key>
      - identity: {}
```

### Network security

**1. Vault TLS**:


```bash
# Configure Vault with TLS
vault write pki/config/urls \
  issuing_certificates="https://vault.example.com:8200/v1/pki/ca" \
  crl_distribution_points="https://vault.example.com:8200/v1/pki/crl"
```

**2. Network Policies**:


```yaml
# network-policy.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-vault-access
  namespace: custody-core
spec:
  podSelector:
    matchLabels:
      app: harmonize-notary
  policyTypes:
  - Egress
  egress:
  - to:
    - podSelector:
        matchLabels:
          app: vault
    ports:
    - protocol: TCP
      port: 8200
```

## Troubleshooting

### Issue: ESO not syncing secrets

**Symptoms**:

- ExternalSecret shows `SecretSyncedError`
- Kubernetes Secret not created or not updated


**Possible causes**:

1. Vault authentication failed
2. Vault path incorrect
3. Vault policy doesn't allow read access


**Solution**:


```bash
# Check ExternalSecret status
kubectl describe externalsecret external-kms-luna-secret -n custody-core

# Check ESO logs
kubectl logs -n external-secrets-system deployment/external-secrets

# Test Vault authentication manually
kubectl exec -it -n custody-core deployment/harmonize-notary -- \
  vault login -method=kubernetes role=eso-reader

# Verify Vault path
vault kv get secret/app/luna-creds

# Verify Vault policy
vault policy read eso-reader
```

### Issue: Vault Injector secrets not injected

**Symptoms**:

- vault-agent sidecar not present
- `/vault/secrets/` directory empty


**Possible causes**:

1. Annotations not set correctly.
2. Vault role not configured.
3. Service account not authorized.


**Solution**:


```bash
# Check pod annotations
kubectl get pod <pod-name> -n custody-core -o yaml | grep vault.hashicorp.com

# Check vault-agent logs
kubectl logs <pod-name> -n custody-core -c vault-agent

# Verify Vault role
vault read auth/kubernetes/role/eso-reader

# Test authentication
kubectl exec -it <pod-name> -n custody-core -c vault-agent -- \
  cat /vault/secrets/ca.pem
```

### Issue: AWS Secrets Manager access denied

**Symptoms**:

- Vault Deployer UI shows "Access Denied"
- Cannot retrieve secrets


**Possible causes**:

1. IAM role doesn't have permissions.
2. Secret doesn't exist.
3. Secret is in different region.


**Solution**:


```bash
# Verify IAM role permissions
aws iam get-role-policy \
  --role-name vault-deployer-role \
  --policy-name SecretsManagerAccess

# Verify secret exists
aws secretsmanager describe-secret \
  --secret-id /ripple/custody/<tenant-id>/<instance-id>/vault/<vault-id>/config

# Check region
aws secretsmanager list-secrets --region us-east-1 | grep ripple

# Update IAM policy if needed
aws iam put-role-policy \
  --role-name vault-deployer-role \
  --policy-name SecretsManagerAccess \
  --policy-document file://secrets-manager-policy.json
```

## Platform-specific examples

### AWS CloudHSM with ESO

See [AWS CloudHSM integration guide](/products/custody/v1.36/deployment/integrate-kms/cloud-hsm/aws-cloudhsm) for complete example.

### Thales Luna HSM with ESO

See [Thales Luna HSM integration guide - advanced secret management](/products/custody/v1.36/deployment/integrate-kms/on-premise-hsm/thales-luna#advanced-secret-management) for complete example.

### BlockSafe HSM with ESO

See [BlockSafe HSM integration guide - advanced secret management](/products/custody/v1.36/deployment/integrate-kms/on-premise-hsm/blocksafe#advanced-secret-management) for complete example.

### MPC with ESO

See [MPC integration guide](/products/custody/v1.36/deployment/integrate-kms/mpc/overview) for complete example.

## Related resources

- [External Secrets Operator documentation](https://external-secrets.io/)
- [HashiCorp Vault documentation](https://www.vaultproject.io/docs)
- [AWS Secrets Manager documentation](https://docs.aws.amazon.com/secretsmanager/)
- [Kubernetes Secrets documentation](https://kubernetes.io/docs/concepts/configuration/secret/)
- [Ripple Custody KMS integration guides](/products/custody/v1.36/deployment/integrate-kms/overview)


## Support

For assistance with advanced secret management:

1. **ESO issues**: [External Secrets Operator GitHub](https://github.com/external-secrets/external-secrets)
2. **Vault issues**: [HashiCorp Vault support](https://support.hashicorp.com/)
3. **AWS issues**: [AWS support](https://aws.amazon.com/support/)
4. **Ripple Custody Issues**: Contact your Ripple liaison


**Required information for support requests**:

- Secret management tool (ESO, Vault Injector, AWS Secrets Manager)
- Error messages from logs
- ExternalSecret or pod annotations (sanitized)
- Vault policy and role configuration (sanitized)
- Kubernetes version and ESO version