Telemetry configuration

This guide describes all Helm chart configuration settings for telemetry in Ripple Custody, including general telemetry, OTEL, Syslog, and OTEL SDK environment variables.

Before you configure telemetry, you need:

• An understanding of telemetry concepts (OpenTelemetry, OTLP, traces, metrics, logs)
• Familiarity with the OpenTelemetry Protocol (OTLP)

The Ripple Custody Helm charts provide granular telemetry configuration through the values.yaml file. Most fields are Go templates that import and concatenate neighboring fields, allowing you to configure telemetry at a fine-grained level. For example, you can send logs and traces to Elasticsearch and metrics to Prometheus.

The telemetry section has three main sub-sections:

1. General telemetry configuration - Protocol type, transport protocol, and hostname
2. OTEL configuration - OTEL-specific endpoints, ports, and SDK environment variables
3. Syslog configuration - Syslog daemon host, port, and gateway settings

harmonize:
  telemetry:

    # Possible values: `otlp` | `disabled`
    type: 'otlp'

      # Possible values: `http` | `https`
    protocol: 'https'

    host: 'my-telemetry-hostname.com'

By default, the Helm charts check whether a local OTEL Collector exists and use it for the telemetry configuration.

If the OTEL Collector sub-chart is enabled, the Helm charts import the telemetry configuration from OTEL Collector and inject it into the Ripple Custody telemetry configuration:

opentelemetry-collector:
  enabled: true

Configure the Ripple Custody Helm charts to use an external telemetry endpoint, either an OTEL Collector or an OTEL-compatible telemetry backend.

harmonize:
  telemetry:
    type: 'otlp'
    protocol: 'https'
    host: 'my-telemetry-hostname.com'

harmonize:
  telemetry:
    otel:
      enabled: true
      host: 'my-otel-host.com'

      # Possible values: `http` | `https`
      protocol: 'https'

      sdk-environment-variables:
        OTEL_TRACES_EXPORTER: 'otlp'
        OTEL_METRICS_EXPORTER: 'otlp'
        OTEL_LOGS_EXPORTER: 'otlp'
        OTEL_LOG_LEVEL: 'info'
        OTEL_METRIC_EXPORT_INTERVAL: '15000'
        OTEL_METRIC_EXPORT_TIMEOUT: '10000'

The Ripple Custody Helm charts allow you to specify data export to three different telemetry backends using the logs, traces, and metrics sub-fields. Each sub-field follows the same pattern as the parent otel configuration, without the sdk-environment-variables field.

harmonize:
  telemetry:
    otel:
      enabled: true
      host: 'my-otel-host.com'
      protocol: 'https'

      sdk-environment-variables:
        OTEL_TRACES_EXPORTER: 'otlp'
        OTEL_METRICS_EXPORTER: 'otlp'
        OTEL_LOGS_EXPORTER: 'otlp'
        OTEL_LOG_LEVEL: 'info'
        OTEL_METRIC_EXPORT_INTERVAL: '15000'
        OTEL_METRIC_EXPORT_TIMEOUT: '10000'

Per-signal backend configuration example with three separate endpoints:

harmonize:
  telemetry:
    enabled: true
    otel:
      logs:
        host: 'my-otel-endpoint-for-logs.com'
      traces:
        host: 'my-otel-endpoint-for-traces.com'
      metrics:
        host: 'my-otel-endpoint-for-metrics.com'

Per-signal sub-field configuration:

harmonize:
  telemetry:
    otel:
      # Logs config
      logs:
        # Enable or disable
        enabled: true
        host: 'my-otel-endpoint-for-logs.com'
        protocol: 'https'

syslog:
  enabled: true
  host: 'my-syslog-receiver-hostname.com'
  port: '54526'
  gatewayName: harmonize_syslog

By default, the Go template imports values for some of the fields from the OTEL Collector sub-chart configuration.

You can specify OTEL SDK environment variables in Ripple Custody Helm charts. The system injects every environment variable specified in this field into every pod that uses the OTEL SDK.

For more information about how to configure telemetry instrumentation and export, see the OTEL SDK environment variable specification.

The Ripple Custody Helm charts configure the following environment variables by default:

harmonize:
  telemetry:
    otel:
      sdk-environment-variables:
        OTEL_TRACES_EXPORTER: 'otlp'
        OTEL_METRICS_EXPORTER: 'otlp'
        OTEL_LOGS_EXPORTER: 'otlp'
        OTEL_LOG_LEVEL: 'info'
        OTEL_METRIC_EXPORT_INTERVAL: '15000'
        OTEL_METRIC_EXPORT_TIMEOUT: '10000'

WARN - OTLP exporter endpoint port is likely incorrect for protocol version "http/protobuf". Port 4317 typically uses grpc.
ERROR - Failed to export spans. Connection reset by peer

OTEL_EXPORTER_OTLP_PROTOCOL="grpc"

JAVA_TOOL_OPTIONS="-Dotel.exporter.otlp.protocol=grpc"

Some OTEL SDK environment variables are already defined at either the application level or by the Helm charts.

The application manages this variable:

• OTEL_SERVICE_NAME (with a unique value for each Ripple Custody component and indexer)

The Helm chart Go templates dynamically define the following values based on the YAML values found at harmonize.telemetry.otel:

• OTEL_SDK_DISABLED
• OTEL_EXPORTER_OTLP_ENDPOINT
• OTEL_EXPORTER_OTLP_LOGS_ENDPOINT
• OTEL_EXPORTER_OTLP_TRACES_ENDPOINT
• OTEL_EXPORTER_OTLP_METRICS_ENDPOINT

You must configure the Batch LogRecord Processor (BLRP) optimally to prevent loss of logs during intermittent outages of the OTEL endpoint (either OTEL Collector or another telemetry backend). These best practices also apply to traces with the OTEL SDK Batch Span Processor.

For more information on how the BLRP works, see the OTEL SDK log specification.

The goal is to strike a balance between memory consumption, export frequency, and network overhead. Regularly monitor the system to fine-tune the BLRP settings and ensure log export runs smoothly.

The OTEL_BLRP_SCHEDULE_DELAY variable specifies the interval between export attempts. If you reduce the default value, OTEL attempts to export logs more frequently, which can be beneficial for pushing logs when the endpoint is still alive or as soon as the endpoint is back online. A much shorter delay could lead to increased operational overhead if the endpoint remains unavailable for an extended period.

More frequent export attempts mean more network packets sent out. This increases the load on the network, potentially causing congestion, especially if multiple instances or services are trying to export logs simultaneously. If the endpoint or any intermediate system (such as a firewall, gateway, router, or Kubernetes network policy) has rate-limiting mechanisms in place, very frequent export attempts might hit these limits, causing valid and important requests to be dropped or delayed.

You can apply Kubernetes traffic shaping using the following pod annotations:

• kubernetes.io/ingress-bandwidth
• kubernetes.io/egress-bandwidth

apiVersion: v1
kind: Pod
metadata:
  annotations:
    kubernetes.io/ingress-bandwidth: 1M
    kubernetes.io/egress-bandwidth: 1M

For more information, see network resource quota.

Each export attempt requires CPU cycles to prepare and send the data. More frequent attempts can result in higher CPU usage.

In the Helm chart configuration, you must set appropriate Kubernetes CPU resource settings for each component:

• resources.requests.cpu
• resources.limits.cpu

components:
  target-harmonize-component:
    resources:
      requests:
        cpu: 1              # MUST be custom based on needs.
      limits:
        cpu: 1              # MUST be custom based on needs.

The OTEL_BLRP_EXPORT_TIMEOUT variable grants a more extended window for successful log exports, particularly during transient networking or OTEL endpoint glitches. Increasing this value directly affects the time the OTEL SDK is willing to wait for an acknowledgment or response from the OTEL endpoint after attempting to export logs.

You also ideally need to increase this value in tandem with OTEL_BLRP_MAX_QUEUE_SIZE and pod memory allocation, to avoid potential backlog problems in the OTEL SDK queue.

If each export attempt waits for an extended timeout period, it can cause subsequent export attempts to queue up or delay, especially if the system is generating logs at a high rate. This can create a backlog of logs waiting to be exported.

The OTEL_BLRP_MAX_QUEUE_SIZE variable controls the in-memory queue buffer. When the endpoint is unavailable, logs start to accumulate in the BLRP's queue. The OTEL SDK stores this queue in memory.

Increasing this value offers a larger buffer to retain logs during short-lived OTEL and OTEL-compatible endpoint disruptions.

In tandem with this change, you must also adjust the pod's memory resources:

• resources.requests.memory
• resources.limits.memory

components:
  target-harmonize-component:
    resources:
      requests:
        memory: "4Gi"       # MUST be custom based on needs.
      limits:
        memory: "4Gi"       # MUST be custom based on needs.

When you double the value of OTEL_BLRP_MAX_QUEUE_SIZE, also increase pod memory limits proportionally. This ensures the pod has adequate memory to manage the enlarged log queue without risking an OutOfMemory error.

The OTEL_BLRP_MAX_EXPORT_BATCH_SIZE variable determines the maximum number of log records the system bundles together and exports in a single batch to the endpoint.

If you've increased the queue size, consider adjusting this variable accordingly. A balanced batch size ensures that the restored endpoint isn't inundated with an overly large volume of logs all at once.

The size of the batch should be in line with the capacity of the endpoint to process incoming logs. If the batch size exceeds the endpoint's processing capability, it may reject or drop the batch. For example, if you know that your endpoint can reliably handle 1000 log records per second, setting the batch size to a value far greater might overload the endpoint.

A larger batch size means more data sent at once, which can result in network congestion or increased latency, especially in bandwidth-limited environments. Make sure the network configuration is adequate to handle large batch exports.

When the OTEL endpoint is unavailable, logs start to accumulate in the BLRP's queue. Once the endpoint is available again, the OTEL SDK exports log batches larger than usual to clear the queue. A larger export batch size can help clear the queued logs faster, ensuring quicker recovery. However, balance this against the potential risk of overwhelming the endpoint.

We recommend you start with a value that's aligned with the endpoint's known processing capacity and the available network bandwidth. Monitor the system's behavior, especially during and after endpoint outages. Adjust OTEL_BLRP_MAX_EXPORT_BATCH_SIZE based on real-world observations, ensuring that you achieve a balance between speedy recovery after outages and not overwhelming the endpoint or network.

• OTEL Collector deployment - Deploy and configure the OTEL Collector
• Networking configuration - Network security and TLS settings