Advanced secret management

This guide covers advanced secret management patterns for Ripple Custody, including External Secrets Operator (ESO), HashiCorp Vault Agent Injector, and AWS Secrets Manager integration.

Target audience: This guide is intended for platform engineers and DevOps teams managing production Ripple Custody deployments with enterprise secret management requirements.

Overview

What is advanced secret management?

Advanced secret management involves using external secret stores (like HashiCorp Vault or AWS Secrets Manager) to centrally manage sensitive credentials, with automatic synchronization to Kubernetes Secrets or direct injection into application pods.

Benefits

Centralized secret management:

Single source of truth for all secrets
Centralized access control and audit logging
Consistent secret rotation policies

Automatic rotation:

Secrets are automatically updated in Kubernetes
No manual intervention required
Reduced risk of stale credentials

Enhanced security:

Secrets never stored in Git or Helm values
Fine-grained access control via IAM/RBAC
Encryption at rest and in transit

Compliance:

Audit trail for all secret access
Meets regulatory requirements (SOC 2, PCI-DSS, etc.)
Separation of duties (platform team vs application team)

External Secrets Operator (ESO)

Overview

The External Secrets Operator (ESO) synchronizes secrets from external secret stores (HashiCorp Vault, AWS Secrets Manager, Azure Key Vault) to Kubernetes Secrets.

How it works:

Prerequisites

1. Install External Secrets Operator:

# Add Helm repository
helm repo add external-secrets https://charts.external-secrets.io
helm repo update

# Install ESO
helm install external-secrets \
  external-secrets/external-secrets \
  --namespace external-secrets-system \
  --create-namespace \
  --set installCRDs=true

# Verify installation
kubectl get pods -n external-secrets-system

2. Configure HashiCorp Vault:

# Enable Kubernetes auth method
vault auth enable kubernetes

# Configure Kubernetes auth
vault write auth/kubernetes/config \
  kubernetes_host="https://kubernetes.default.svc:443"

# Create policy for ESO
vault policy write eso-reader - <<EOF
path "secret/data/app/*" {
  capabilities = ["read"]
}
EOF

# Create role for ESO
vault write auth/kubernetes/role/eso-reader \
  bound_service_account_names=external-secrets \
  bound_service_account_namespaces=custody-core \
  policies=eso-reader \
  ttl=1h

3. Store Secrets in Vault:

# Example: Store Luna HSM credentials
vault kv put secret/app/luna-creds \
  ca.pem=@ca.pem \
  client.pem=@client.pem \
  client.key=@client.key \
  pin="your-hsm-pin"

# Verify
vault kv get secret/app/luna-creds

Step 1: Create SecretStore

The SecretStore defines the connection to HashiCorp Vault.

Create secretstore.yaml:

# secretstore.yaml
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: vault-backend
  namespace: custody-core
spec:
  provider:
    vault:
      # Vault server address
      server: "https://vault.example.com"

      # KV-v2 secrets engine mount path
      path: "secret"
      version: "v2"

      # Authentication method
      auth:
        kubernetes:
          # Kubernetes auth mount path in Vault
          mountPath: "kubernetes"

          # Vault role to use
          role: "eso-reader"

          # Service account for authentication
          serviceAccountRef:
            name: external-secrets

Apply:

kubectl apply -f secretstore.yaml

# Verify
kubectl get secretstore -n custody-core

Step 2: Create ExternalSecret

The ExternalSecret defines which secrets to sync and how to map them.

Create externalsecret-luna.yaml:

# externalsecret-luna.yaml
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: external-kms-luna-secret
  namespace: custody-core
spec:
  # Refresh interval (how often to check Vault for updates)
  refreshInterval: 1h

  # Reference to SecretStore
  secretStoreRef:
    name: vault-backend
    kind: SecretStore

  # Target Kubernetes Secret
  target:
    name: kms-luna-credentials
    creationPolicy: Owner

  # Data mapping: Vault key -> Kubernetes Secret key
  data:
    # ca.pem
    - secretKey: ca.pem
      remoteRef:
        key: app/luna-creds
        property: ca.pem

    # client.pem
    - secretKey: client.pem
      remoteRef:
        key: app/luna-creds
        property: client.pem

    # client.key
    - secretKey: client.key
      remoteRef:
        key: app/luna-creds
        property: client.key

    # PIN
    - secretKey: KMS_LUNA_PIN
      remoteRef:
        key: app/luna-creds
        property: pin

Apply:

kubectl apply -f externalsecret-luna.yaml

# Verify ExternalSecret
kubectl get externalsecret -n custody-core

# Verify Kubernetes Secret was created
kubectl get secret kms-luna-credentials -n custody-core

# View secret keys (not values)
kubectl describe secret kms-luna-credentials -n custody-core

Step 3: Reference secret in Helm values

Update your Helm values to reference the ESO-managed secret.

Example values.yaml:

# values.yaml
kms:
  kind: "luna"

  # Reference existing secret created by ESO
  existingSecret: "kms-luna-credentials"

  # Optional: Specify custom key names if different from defaults
  existingSecretKeys:
    pin: "KMS_LUNA_PIN"
    caPem: "ca.pem"
    clientPem: "client.pem"
    clientKey: "client.key"

Deploy:

helm upgrade --install harmonize ./harmonize-custody \
  --namespace custody-core \
  -f values.yaml

Step 4: Verify secret rotation

ESO automatically updates the Kubernetes Secret when the Vault secret changes.

Test rotation:

# Update secret in Vault
vault kv put secret/app/luna-creds \
  ca.pem=@ca-new.pem \
  client.pem=@client-new.pem \
  client.key=@client-new.key \
  pin="new-hsm-pin"

# Wait for refreshInterval (default: 1h) or force sync
kubectl annotate externalsecret external-kms-luna-secret \
  force-sync=$(date +%s) \
  -n custody-core

# Verify secret was updated
kubectl get secret kms-luna-credentials -n custody-core -o yaml

# Restart pods to pick up new secret
kubectl rollout restart deployment/harmonize-notary -n custody-core

HashiCorp Vault Agent Injector

Overview

The Vault Agent Injector uses a sidecar pattern to inject secrets directly into application pods without creating Kubernetes Secrets.

How it works:

Prerequisites

1. Install Vault Agent Injector:

# Add Helm repository
helm repo add hashicorp https://helm.releases.hashicorp.com
helm repo update

# Install Vault with injector
helm install vault hashicorp/vault \
  --namespace vault-system \
  --create-namespace \
  --set "injector.enabled=true" \
  --set "server.dev.enabled=true"

# Verify installation
kubectl get pods -n vault-system

2. Configure Vault (same as ESO prerequisites)

Step 1: Annotate pod for injection

Add Vault annotations to your Helm values to enable injection.

Example values.yaml:

# values.yaml
kms:
  kind: "luna"

  # Vault Agent Injector annotations
  podAnnotations:
    vault.hashicorp.com/agent-inject: "true"
    vault.hashicorp.com/role: "eso-reader"
    vault.hashicorp.com/agent-inject-secret-ca.pem: "secret/data/app/luna-creds"
    vault.hashicorp.com/agent-inject-template-ca.pem: |
      {{- with secret "secret/data/app/luna-creds" -}}
      {{ .Data.data.ca.pem }}
      {{- end -}}

    vault.hashicorp.com/agent-inject-secret-client.pem: "secret/data/app/luna-creds"
    vault.hashicorp.com/agent-inject-template-client.pem: |
      {{- with secret "secret/data/app/luna-creds" -}}
      {{ .Data.data.client.pem }}
      {{- end -}}

    vault.hashicorp.com/agent-inject-secret-client.key: "secret/data/app/luna-creds"
    vault.hashicorp.com/agent-inject-template-client.key: |
      {{- with secret "secret/data/app/luna-creds" -}}
      {{ .Data.data.client.key }}
      {{- end -}}

    vault.hashicorp.com/agent-inject-secret-pin: "secret/data/app/luna-creds"
    vault.hashicorp.com/agent-inject-template-pin: |
      {{- with secret "secret/data/app/luna-creds" -}}
      {{ .Data.data.pin }}
      {{- end -}}

  # Environment variables pointing to injected files
  extraEnvVars:
    - name: KMS_LUNA_CA_PEM_PATH
      value: "/vault/secrets/ca.pem"
    - name: KMS_LUNA_CLIENT_PEM_PATH
      value: "/vault/secrets/client.pem"
    - name: KMS_LUNA_CLIENT_KEY_PATH
      value: "/vault/secrets/client.key"
    - name: KMS_LUNA_PIN
      valueFrom:
        secretKeyRef:
          name: vault-injected-pin
          key: pin

Step 2: Deploy and verify

# Deploy with Vault Agent Injector
helm upgrade --install harmonize ./harmonize-custody \
  --namespace custody-core \
  -f values.yaml

# Verify vault-agent sidecar was injected
kubectl get pods -n custody-core

# Expected output:
# NAME                                READY   STATUS    RESTARTS   AGE
# harmonize-notary-<hash>             2/2     Running   0          1m
#                                     ^^^
#                                     2 containers: kms-luna + vault-agent

# Verify secrets were injected
kubectl exec -it -n custody-core deployment/harmonize-notary -c kms-luna -- \
  ls -la /vault/secrets/

# Expected output:
# -rw-r--r-- 1 vault vault 1234 Jun 26 10:00 ca.pem
# -rw-r--r-- 1 vault vault 1234 Jun 26 10:00 client.pem
# -rw-r--r-- 1 vault vault 1234 Jun 26 10:00 client.key
# -rw-r--r-- 1 vault vault   12 Jun 26 10:00 pin

AWS Secrets Manager

Overview

AWS Secrets Manager is used by the Vault Deployer UI to store deployment configuration and HSM credentials for AWS Nitro enclave deployments.

Storage path structure:

/ripple/custody/<tenant-id>/<instance-id>/<construct-name>/<construct-id>/

Where:

<tenant-id>: Your Ripple Custody tenant identifier
<instance-id>: Your Ripple Custody instance identifier
<construct-name>: Always vault for vault deployments
<construct-id>: The vault ID (UUID)

Secrets stored

1. config key: Deployment configuration

{
  "deployment": {
    "vault": {
      "release-name": "vault-01",
      "construct-id": "00000000-0000-0000-0000-000000000001"
    }
  },
  "custody": {
    "environment": "https://api.your-instance.custody.example.com/"
  },
  "infrastructure": {
    "aws": {
      "region": "us-east-1",
      "ec2-iam-role": "vault-role",
      "vpc": {
        "create-new": false,
        "id": "vpc-abc123",
        "subnet-id": "subnet-def456"
      }
    }
  }
}

2. hsm key: HSM credentials

{
  "kms-awscloudhsm": "cluster-a1b2c3d4e5f",
  "kms-awscloudhsm-cacert": "-----BEGIN CERTIFICATE-----\nMI...==\n-----END CERTIFICATE-----\n",
  "kms-password": "CryptoUserPassword",
  "kms-soft-wrapping-key": "0123456789abcdef...",
  "kms-username": "vault-crypto-user",
  "kms-wrap-key": "0000000000000001",
  "kms-awscloudhsm-clientcert": "-----BEGIN CERTIFICATE-----\n...",
  "kms-awscloudhsm-clientkey": "-----BEGIN PRIVATE KEY-----\n..."
}

Security best practices

1. Limit Access:

# Create IAM policy for limited access
cat > secrets-manager-policy.json <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "secretsmanager:GetSecretValue",
        "secretsmanager:DescribeSecret"
      ],
      "Resource": "arn:aws:secretsmanager:us-east-1:123456789012:secret:/ripple/custody/*"
    }
  ]
}
EOF

# Attach policy to role
aws iam put-role-policy \
  --role-name vault-deployer-role \
  --policy-name SecretsManagerAccess \
  --policy-document file://secrets-manager-policy.json

2. Enable Audit Logging:

# Enable CloudTrail for Secrets Manager
aws cloudtrail create-trail \
  --name secrets-manager-audit \
  --s3-bucket-name my-audit-bucket

# Start logging
aws cloudtrail start-logging \
  --name secrets-manager-audit

3. Recovery Window:

AWS Secrets Manager provides a 7-day recovery window for deleted secrets.

# Delete secret (with recovery window)
aws secretsmanager delete-secret \
  --secret-id /ripple/custody/tenant/instance/vault/vault-id/config \
  --recovery-window-in-days 7

# Restore secret within 7 days
aws secretsmanager restore-secret \
  --secret-id /ripple/custody/tenant/instance/vault/vault-id/config

Accessing secrets

# Get config secret
aws secretsmanager get-secret-value \
  --secret-id /ripple/custody/<tenant-id>/<instance-id>/vault/<vault-id>/config \
  --query SecretString \
  --output text | jq .

# Get HSM credentials
aws secretsmanager get-secret-value \
  --secret-id /ripple/custody/<tenant-id>/<instance-id>/vault/<vault-id>/hsm \
  --query SecretString \
  --output text | jq .

Important: While AWS Secrets Manager is used during vault deployment automation, Ripple Custody vault itself does not use AWS Secrets Manager to store wallet private keys. Wallet keys are encrypted with AWS CloudHSM and stored in the Ripple Custody database.

Comparison: ESO vs Vault Injector vs AWS Secrets Manager

Feature	External Secrets Operator	Vault Agent Injector	AWS Secrets Manager
Secret Storage	Kubernetes Secret	Files in shared volume	AWS Secrets Manager
Rotation	Automatic (refreshInterval)	Automatic (lease renewal)	Manual or Lambda-based
Kubernetes Native	Yes (CRDs)	Yes (annotations)	No (AWS-specific)
Multi-Cloud	Yes (Vault, AWS, Azure)	Yes (Vault only)	No (AWS only)
Complexity	Low	Medium	Low
Performance	Good (cached in K8s Secret)	Excellent (direct injection)	Good (cached in AWS)
Security	Good (K8s RBAC)	Excellent (no K8s Secret)	Good (IAM policies)
Use Case	General-purpose	High-security environments	AWS Nitro enclaves
Ripple Custody Support	Recommended	Supported	Vault Deployer UI only

Recommendations

Use External Secrets Operator (ESO) when:

You need multi-cloud support
You want Kubernetes-native secret management
You need automatic rotation with minimal complexity
Recommended for most Ripple Custody deployments

Use Vault Agent Injector when:

You have strict security requirements (no K8s Secrets)
You need fine-grained secret access control
You want secrets to be ephemeral (not persisted in K8s)
You're already using HashiCorp Vault extensively

Use AWS Secrets Manager when:

You're deploying vaults in AWS Nitro enclaves
You're using the Vault Deployer UI
You need AWS-native secret management
Only for Vault Deployer UI deployments

Security best practices

Principle of least privilege

1. Vault Policies:

# eso-reader-policy.hcl
# Read-only access to specific paths
path "secret/data/app/luna-creds" {
  capabilities = ["read"]
}

# Deny all other paths
path "secret/*" {
  capabilities = ["deny"]
}

2. Kubernetes RBAC:

# rbac.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: secret-reader
  namespace: custody-core
rules:
- apiGroups: [""]
  resources: ["secrets"]
  resourceNames: ["kms-luna-credentials"]
  verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: secret-reader-binding
  namespace: custody-core
subjects:
- kind: ServiceAccount
  name: harmonize-notary
  namespace: custody-core
roleRef:
  kind: Role
  name: secret-reader
  apiGroup: rbac.authorization.k8s.io

Secret rotation

1. Automated Rotation with ESO:

# externalsecret.yaml
spec:
  refreshInterval: 1h  # Check Vault every hour

  # Optional: Force immediate sync on annotation change
  metadata:
    annotations:
      force-sync: "2025-06-26T10:00:00Z"

2. Rotation Procedure:

# 1. Update secret in Vault
vault kv put secret/app/luna-creds \
  ca.pem=@ca-new.pem \
  client.pem=@client-new.pem \
  client.key=@client-new.key \
  pin="new-pin"

# 2. Force ESO sync (optional, or wait for refreshInterval)
kubectl annotate externalsecret external-kms-luna-secret \
  force-sync=$(date +%s) \
  -n custody-core

# 3. Restart pods to pick up new secret
kubectl rollout restart deployment/harmonize-notary -n custody-core

# 4. Verify new secret is in use
kubectl logs -n custody-core deployment/harmonize-notary | grep "Connected to HSM"

Audit logging

1. Vault Audit Logging:

# Enable audit logging
vault audit enable file file_path=/vault/logs/audit.log

# Query audit logs
cat /vault/logs/audit.log | jq 'select(.request.path == "secret/data/app/luna-creds")'

2. Kubernetes Audit Logging:

# audit-policy.yaml
apiVersion: audit.k8s.io/v1
kind: Policy
rules:
- level: RequestResponse
  resources:
  - group: ""
    resources: ["secrets"]
  namespaces: ["custody-core"]

Encryption at rest

1. Vault Transit Engine:

# Enable transit engine
vault secrets enable transit

# Create encryption key
vault write -f transit/keys/custody-secrets

# Encrypt secret before storing
vault write transit/encrypt/custody-secrets \
  plaintext=$(base64 <<< "my-secret-value")

2. Kubernetes Encryption at Rest:

# encryption-config.yaml
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      - aescbc:
          keys:
            - name: key1
              secret: <base64-encoded-32-byte-key>
      - identity: {}

Network security

1. Vault TLS:

# Configure Vault with TLS
vault write pki/config/urls \
  issuing_certificates="https://vault.example.com:8200/v1/pki/ca" \
  crl_distribution_points="https://vault.example.com:8200/v1/pki/crl"

2. Network Policies:

# network-policy.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-vault-access
  namespace: custody-core
spec:
  podSelector:
    matchLabels:
      app: harmonize-notary
  policyTypes:
  - Egress
  egress:
  - to:
    - podSelector:
        matchLabels:
          app: vault
    ports:
    - protocol: TCP
      port: 8200

Troubleshooting

Issue: ESO not syncing secrets

Symptoms:

ExternalSecret shows SecretSyncedError
Kubernetes Secret not created or not updated

Possible causes:

Vault authentication failed
Vault path incorrect
Vault policy doesn't allow read access

Solution:

# Check ExternalSecret status
kubectl describe externalsecret external-kms-luna-secret -n custody-core

# Check ESO logs
kubectl logs -n external-secrets-system deployment/external-secrets

# Test Vault authentication manually
kubectl exec -it -n custody-core deployment/harmonize-notary -- \
  vault login -method=kubernetes role=eso-reader

# Verify Vault path
vault kv get secret/app/luna-creds

# Verify Vault policy
vault policy read eso-reader

Issue: Vault Injector secrets not injected

Symptoms:

vault-agent sidecar not present
/vault/secrets/ directory empty

Possible causes:

Annotations not set correctly.
Vault role not configured.
Service account not authorized.

Solution:

# Check pod annotations
kubectl get pod <pod-name> -n custody-core -o yaml | grep vault.hashicorp.com

# Check vault-agent logs
kubectl logs <pod-name> -n custody-core -c vault-agent

# Verify Vault role
vault read auth/kubernetes/role/eso-reader

# Test authentication
kubectl exec -it <pod-name> -n custody-core -c vault-agent -- \
  cat /vault/secrets/ca.pem

Issue: AWS Secrets Manager access denied

Symptoms:

Vault Deployer UI shows "Access Denied"
Cannot retrieve secrets

Possible causes:

IAM role doesn't have permissions.
Secret doesn't exist.
Secret is in different region.

Solution:

# Verify IAM role permissions
aws iam get-role-policy \
  --role-name vault-deployer-role \
  --policy-name SecretsManagerAccess

# Verify secret exists
aws secretsmanager describe-secret \
  --secret-id /ripple/custody/<tenant-id>/<instance-id>/vault/<vault-id>/config

# Check region
aws secretsmanager list-secrets --region us-east-1 | grep ripple

# Update IAM policy if needed
aws iam put-role-policy \
  --role-name vault-deployer-role \
  --policy-name SecretsManagerAccess \
  --policy-document file://secrets-manager-policy.json

Platform-specific examples

MPC with ESO

See MPC integration guide for complete example.

Support

For assistance with advanced secret management:

ESO issues: External Secrets Operator GitHub
Vault issues: HashiCorp Vault support
AWS issues: AWS support
Ripple Custody Issues: Contact your Ripple liaison

Required information for support requests:

Secret management tool (ESO, Vault Injector, AWS Secrets Manager)
Error messages from logs
ExternalSecret or pod annotations (sanitized)
Vault policy and role configuration (sanitized)
Kubernetes version and ESO version

Advanced secret management

Table of contents

Overview

What is advanced secret management?

Benefits

External Secrets Operator (ESO)

Overview

Prerequisites

Step 1: Create SecretStore

Step 2: Create ExternalSecret

Step 3: Reference secret in Helm values

Step 4: Verify secret rotation

HashiCorp Vault Agent Injector

Overview

Prerequisites

Step 1: Annotate pod for injection

Step 2: Deploy and verify

AWS Secrets Manager

Overview

Secrets stored

Security best practices

Accessing secrets

Comparison: ESO vs Vault Injector vs AWS Secrets Manager

Recommendations

Security best practices

Principle of least privilege

Secret rotation

Audit logging

Encryption at rest

Network security

Troubleshooting

Issue: ESO not syncing secrets

Issue: Vault Injector secrets not injected

Issue: AWS Secrets Manager access denied

Platform-specific examples

AWS CloudHSM with ESO

Thales Luna HSM with ESO

BlockSafe HSM with ESO

MPC with ESO

Related resources

Support

Was this page helpful?