Telemetry configuration

Telemetry configuration is an important part of on-premise Ripple Custody deployments. It defines how the platform collects, processes, and sends telemetry data to the appropriate server.

Most of the fields in the values.yaml file of Ripple Custody Helm charts are Go templates that import and concatenate several neighboring fields. You can use them to configure telemetry in Ripple Custody in a granular way. For example, send the logs and traces to Elasticsearch and the metrics to Prometheus.

The telemetry section has three main sub-sections:

• General telemetry configuration
• Syslog configuration
• OTEL configuration

The telemetry configuration fields for Ripple Custody are as follows:

harmonize:
  telemetry:

    # Possible values: `otlp` | `disabled`
    type: 'otlp'

      # Possible values: `http` | `https`
    protocol: 'https'

    host: 'my-telemetry-hostname.com'

The following describes each of the three telemetry sub-sections.

You can configure the Ripple Custody Helm charts to use an external telemetry endpoint, that is OTEL Collector or an OTEL-compatible telemetry backend.

In the values.yaml file, the general telemetry configuration is in the following format:

Ripple Custody:
  telemetry:
    type: 'otlp'
    protocol: 'https'
    host: 'my-telemetry-hostname.com'

By default, the Helm charts check whether a local OTEL Collector exists and use it for the telemetry configuration.

If the following field in is set to true:

opentelemetry-collector:
  enabled: true

The Helm charts import the telemetry configuration from OTEL Collector and inject it into the Ripple Custody telemetry configuration.

harmonize:
  telemetry:
    otel:
      enabled: true
      host: 'my-otel-host.com'

      # Possible values: `http` | `https`
      protocol: 'https'

      sdk-environment-variables:
        OTEL_TRACES_EXPORTER: 'otlp'
        OTEL_METRICS_EXPORTER: 'otlp'
        OTEL_LOGS_EXPORTER: 'otlp'
        OTEL_LOG_LEVEL: 'info'
        OTEL_METRIC_EXPORT_INTERVAL: '15000'
        OTEL_METRIC_EXPORT_TIMEOUT: '10000'

The Ripple Custody Helm charts give you the option to specify data export to three different telemetry backends:

harmonize:
  telemetry:
  enabled: true
    otel:
      logs: 
        host: 'my-otel-endpoint-for-logs.com'
      traces: 
        host: 'my-otel-endpoint-for-traces.com'
      metrics: 
        host: 'my-otel-endpoint-for-metrics.com'

The sub-fields logs, traces and metrics follow the same pattern, without the field sdk-environment-variables.

harmonize:
  telemetry:
    otel:
      # Logs config
      logs:
        # Enable or disable
        enabled: true
        host: 'my-otel-endpoint-for-logs.com'
        protocol: 'https'

The following configuration shows two different telemetry backends:

1. One telemetry backend for logs and traces.
2. One telemetry backend for metrics.

harmonize:
    telemetry:
      type: otlp
      protocol: http
      host: 'my-otel-endpoint-for-logs-and-traces.com'  # E.g. Elasticsearch

      otel:
        metrics:
          host: 'my-otel-endpoint-for-metrics.com'      # E.g Prometheus

The Syslog configuration is in the following format:

syslog:
  enabled: true
  host: 'my-syslog-receiver-hostname.com'
  port: '54526'
  gatewayName: harmonize_syslog

By default, the Go template imports values for some of the fields from the OTEL Collector sub-chart configuration. For more information, see Telemetry data reception.

You can specify OTEL SDK environment variables in Ripple Custody Helm charts.

For more information about how to configure telemetry instrumentation and export, see the Open Telemetry OTEL SDK environment variable specification.

By default, the Ripple Custody Helm charts configure some environment variables with default values:

harmonize:
  telemetry:
    otel:
      sdk-environment-variables:
        OTEL_TRACES_EXPORTER: 'otlp'
        OTEL_METRICS_EXPORTER: 'otlp'
        OTEL_LOGS_EXPORTER: 'otlp'
        OTEL_LOG_LEVEL: 'info'
        OTEL_METRIC_EXPORT_INTERVAL: '15000'
        OTEL_METRIC_EXPORT_TIMEOUT: '10000'

The system injects every environment variable specified in this field into every pod that uses the OTEL SDK.

WARN - OTLP exporter endpoint port is likely incorrect for protocol version "http/protobuf". Port 4317 typically uses grpc.
ERROR - Failed to export spans. Connection reset by peer

OTEL_EXPORTER_OTLP_PROTOCOL="grpc"

JAVA_TOOL_OPTIONS="-Dotel.exporter.otlp.protocol=grpc"

Some OTEL SDK environment variables are already defined at either the application level or by the Helm charts.

The application handles these OTEL SDK environment variables:

• OTEL_SERVICE_NAME (with a unique value for each Ripple Custody component and indexer)

The Helm charts Go Templates dynamically define those values based on the YAML values found in the values.yaml file at location harmonize.telemetry.otel:

• OTEL_SDK_DISABLED
• OTEL_EXPORTER_OTLP_ENDPOINT
• OTEL_EXPORTER_OTLP_LOGS_ENDPOINT
• OTEL_EXPORTER_OTLP_TRACES_ENDPOINT
• OTEL_EXPORTER_OTLP_METRICS_ENDPOINT

You must configure the Batch LogRecord Processor (BLRP) optimally to prevent loss of logs during intermittent outages of the OTEL endpoint (either OTEL Collector or another telemetry backend).

The logs best practices also apply to traces with the OTEL SDK Batch Span Processor.

For more information on how the BLRP works, see the OTEL SDK log specification.

The goal is to strike a balance between memory consumption, export frequency, and network overhead. Regularly monitor the system to fine-tune the BLRP settings and ensure log export runs smoothly.

The following table lists the environment variables available to configure the BLRP:

This variable specifies the interval between export attempts. If you reduce the default value, OTEL attempts to export logs more frequently, which can be beneficial for pushing logs when the endpoint is still alive or as soon as the endpoint is back online. A much shorter delay could lead to increased operational overhead if the endpoint remains unavailable for an extended period.

More frequent export attempts mean more network packets sent out. This increases the load on the network, potentially causing congestion, especially if multiple instances or services are trying to export logs simultaneously. If the endpoint or any intermediate system, such as a firewall, gateway, router, or Kubernetes network policy, has rate-limiting mechanisms in place, very frequent export attempts might hit these limits, causing valid and important requests to be dropped or delayed.

You can apply Kubernetes traffic shaping using the following pod annotations:

• kubernetes.io/ingress-bandwidth
• kubernetes.io/egress-bandwidth

apiVersion: v1
kind: Pod
metadata:
  annotations:
    kubernetes.io/ingress-bandwidth: 1M
    kubernetes.io/egress-bandwidth: 1M

For more information, see network resource quota.

Each export attempt requires CPU cycles to prepare and send the data. More frequent attempts can result in higher CPU usage.

In the Helm chart configuration, you must set appropriate Kubernetes CPU resource settings for each component:

• resources.requests.cpu
• resources.limits.cpu

components:
  target-harmonize-component:
    resources:
      requests:
        cpu: 1              # MUST be custom based on needs.
      limits:
        cpu: 1              # MUST be custom based on needs.

This variable grants a more extended window for successful log exports, particularly pertinent during transient networking or OTEL endpoint glitches. Increasing this value directly affects the time the OTEL SDK is willing to wait for an acknowledgment or response from the OTEL endpoint after attempting to export logs.

You also ideally need to increase this value in tandem with OTEL_BLRP_MAX_QUEUE_SIZE and pod memory allocation, to avoid potential backlog problems in the OTEL SDK queue.

If each export attempt waits for an extended timeout period, it can cause subsequent export attempts to queue up or delay, especially if the system is generating logs at a high rate. This can create a backlog of logs waiting to be exported.

When the endpoint is unavailable, logs start to accumulate in the BLRP's queue. The OTEL SDK stores this queue in memory.

Increasing this value offers a larger buffer to retain logs during short-lived OTEL and OTEL-compatible endpoint disruptions.

In tandem with this change, you must also adjust the pod's memory resources.

In the configuration, appropriate Kubernetes Memory resource settings must be set for each component:

• resources.requests.memory
• resources.limits.memory

components:
  target-harmonize-component:
    resources:
      requests:
        memory: "4Gi"       # MUST be custom based on needs.
      limits:
        memory: "4Gi"       # MUST be custom based on needs.

When you double the value of OTEL_BLRP_MAX_QUEUE_SIZE, also increase pod memory limits proportionally.

This ensures the pod has adequate memory to manage the enlarged log queue without risking an OutOfMemory error.

This variable determines the maximum number of log records the system bundles together and exports in a single batch to the endpoint. Configuring this parameter effectively is crucial for both performance and reliability.

If you've increased the queue size, consider adjusting this variable accordingly. A balanced batch size ensures that the restored endpoint isn't inundated with an overly large volume of logs all at once.

The size of the batch should be in line with the capacity of the endpoint to process incoming logs. If the batch size exceeds the endpoint's processing capability, it may reject or drop the batch. For example, if you know that your endpoint can reliably handle 1000 log records per second, setting the batch size to a value far greater might overload the endpoint.

A larger batch size means more data sent at once, which can result in network congestion or increased latency, especially in bandwidth-limited environments. Make sure the network configuration is adequate to handle large batch exports.

When the OTEL endpoint is unavailable, logs start to accumulate in the BLRP's queue. Once the endpoint is available again, the OTEL SDK exports log batches larger than usual to clear the queue. A larger export batch size can help clear the queued logs faster, ensuring quicker recovery. However, balance this against the potential risk of overwhelming the endpoint.

We recommend you start with a value that's aligned with the endpoint's known processing capacity and the available network bandwidth. Monitor the system's behavior, especially during and after endpoint outages. Adjust OTEL_BLRP_MAX_EXPORT_BATCH_SIZE based on real-world observations, ensuring that you achieve a balance between speedy recovery after outages and not overwhelming the endpoint or network.