# Telemetry configuration This guide describes all Helm chart configuration settings for telemetry in Ripple Custody, including general telemetry, OTEL, Syslog, and OTEL SDK environment variables. This guide applies to on-premise deployments only. ## Prerequisites Before you configure telemetry, you need: - An understanding of telemetry concepts (OpenTelemetry, OTLP, traces, metrics, logs) - Familiarity with the OpenTelemetry Protocol (OTLP) ## Overview The Ripple Custody Helm charts provide granular telemetry configuration through the `values.yaml` file. Most fields are Go templates that import and concatenate neighboring fields, allowing you to configure telemetry at a fine-grained level. For example, you can send logs and traces to Elasticsearch and metrics to Prometheus. The `telemetry` section has three main sub-sections: 1. **General telemetry configuration** - Protocol type, transport protocol, and hostname 2. **OTEL configuration** - OTEL-specific endpoints, ports, and SDK environment variables 3. **Syslog configuration** - Syslog daemon host, port, and gateway settings ```yaml harmonize: telemetry: # Possible values: `otlp` | `disabled` type: 'otlp' # Possible values: `http` | `https` protocol: 'https' host: 'my-telemetry-hostname.com' ``` By default, the Helm charts check whether a local OTEL Collector exists and use it for the telemetry configuration. Disclaimer The local OTEL Collector setup is intended for testing purposes only. Do not use it in production. If the OTEL Collector sub-chart is enabled, the Helm charts import the telemetry configuration from OTEL Collector and inject it into the Ripple Custody telemetry configuration: ```yaml opentelemetry-collector: enabled: true ``` ## General telemetry configuration Configure the Ripple Custody Helm charts to use an external telemetry endpoint, either an OTEL Collector or an OTEL-compatible telemetry backend. | Parameter | Type | Default | Description | | --- | --- | --- | --- | | `type` | string | `disabled` | Type of telemetry used. Possible values: `disabled`, `otlp`, `jeager`, `azure` | | `protocol` | string | `http` | Protocol used by Ripple Custody components to send telemetry data. Possible values: `http`, `https` | | `host` | string | None | Hostname the telemetry data is sent to | ```yaml harmonize: telemetry: type: 'otlp' protocol: 'https' host: 'my-telemetry-hostname.com' ``` The hostname must not include a port. Port information is set up automatically for `otel-grpc` (4317), `otel-http` (4318), and `syslog` (54526). ## OTEL configuration | Parameter | Type | Default | Description | | --- | --- | --- | --- | | `otel.enabled` | string/boolean | Go template | Determines whether the OTEL Collector is enabled | | `otel.host` | string | Go template | Specifies the host for the OTEL endpoint. By default, determined by the `host` field in the `telemetry` configuration | | `otel.protocol` | string | Go template | Specifies the protocol used by the OTEL endpoint (level 7). By default, determined by the `protocol` field in the `telemetry` configuration | | `otel.httpPort` | string | Go template | Specifies the HTTP port for the OTEL endpoint | | `otel.grpcPort` | string | Go template | Specifies the gRPC port for the OTEL endpoint | | `otel.httpEndpoint` | string | Go template | HTTP endpoint URI, constructed from `protocol`, `host`, and `port` | | `otel.grpcEndpoint` | string | Go template | gRPC endpoint URI, constructed from `protocol`, `host`, and `port` | | `otel.sdk-environment-variables` | object | See below | Key-value pairs for OTEL SDK environment variables | | `otel.logs` | object | - | Per-signal backend override for logs. See [Multiple backends](#multiple-backends) | | `otel.traces` | object | - | Per-signal backend override for traces. See [Multiple backends](#multiple-backends) | | `otel.metrics` | object | - | Per-signal backend override for metrics. See [Multiple backends](#multiple-backends) | ```yaml harmonize: telemetry: otel: enabled: true host: 'my-otel-host.com' # Possible values: `http` | `https` protocol: 'https' sdk-environment-variables: OTEL_TRACES_EXPORTER: 'otlp' OTEL_METRICS_EXPORTER: 'otlp' OTEL_LOGS_EXPORTER: 'otlp' OTEL_LOG_LEVEL: 'info' OTEL_METRIC_EXPORT_INTERVAL: '15000' OTEL_METRIC_EXPORT_TIMEOUT: '10000' ``` The configuration provides parameters (port/endpoint) for both gRPC and HTTP because different Ripple Custody components send telemetry data using either one or the other protocol. ### Multiple backends The Ripple Custody Helm charts allow you to specify data export to three different telemetry backends using the `logs`, `traces`, and `metrics` sub-fields. Each sub-field follows the same pattern as the parent `otel` configuration, without the `sdk-environment-variables` field. ![Diagram showing OTEL SDK configured with separate endpoints for logs, traces, and metrics telemetry data](/assets/otel-sdk-multiple-otel-endpoints.43d68abbfebe6049a80a17df2b815b655f37ba73fe4fcd87eb8b88bf29bcfe78.1d1f12f8.svg) Standard OTLP ```yaml harmonize: telemetry: otel: enabled: true host: 'my-otel-host.com' protocol: 'https' sdk-environment-variables: OTEL_TRACES_EXPORTER: 'otlp' OTEL_METRICS_EXPORTER: 'otlp' OTEL_LOGS_EXPORTER: 'otlp' OTEL_LOG_LEVEL: 'info' OTEL_METRIC_EXPORT_INTERVAL: '15000' OTEL_METRIC_EXPORT_TIMEOUT: '10000' ``` Multiple backends ```yaml harmonize: telemetry: type: otlp protocol: http host: 'my-otel-endpoint-for-logs-and-traces.com' # E.g. Elasticsearch otel: metrics: host: 'my-otel-endpoint-for-metrics.com' # E.g Prometheus ``` ![Diagram showing OTEL SDK sending logs and traces to Elasticsearch and metrics to Prometheus](/assets/otel-sdk-elastic-search-prometheus.6eef3c9d761ac0e7fa48427064db1f4cc8d0c6764831bffdd4638ed1cbb53345.1d1f12f8.svg) Syslog only ```yaml harmonize: telemetry: type: 'disabled' syslog: enabled: true host: 'my-syslog-receiver-hostname.com' port: '54526' gatewayName: harmonize_syslog ``` Per-signal backend configuration example with three separate endpoints: ```yaml harmonize: telemetry: enabled: true otel: logs: host: 'my-otel-endpoint-for-logs.com' traces: host: 'my-otel-endpoint-for-traces.com' metrics: host: 'my-otel-endpoint-for-metrics.com' ``` Per-signal sub-field configuration: ```yaml harmonize: telemetry: otel: # Logs config logs: # Enable or disable enabled: true host: 'my-otel-endpoint-for-logs.com' protocol: 'https' ``` ## Syslog configuration | Parameter | Type | Default | Description | | --- | --- | --- | --- | | `syslog.enabled` | boolean | `true` | Whether syslog is enabled | | `syslog.protocol` | string | `tcp` | The protocol used by Syslog (level 4) | | `syslog.host` | string | Determined by the `host` field in the `otel` configuration | The host for the syslog | | `syslog.port` | integer | `54526` | The port for the syslog | | `syslog.gatewayName` | string | `harmonize_syslog` | The name of the syslog gateway | ```yaml syslog: enabled: true host: 'my-syslog-receiver-hostname.com' port: '54526' gatewayName: harmonize_syslog ``` By default, the Go template imports values for some of the fields from the OTEL Collector sub-chart configuration. ## OTEL SDK environment variables You can specify OTEL SDK environment variables in Ripple Custody Helm charts. The system injects every environment variable specified in this field into every pod that uses the OTEL SDK. For more information about how to configure telemetry instrumentation and export, see the OTEL SDK environment variable specification. ### Default values The Ripple Custody Helm charts configure the following environment variables by default: ```yaml harmonize: telemetry: otel: sdk-environment-variables: OTEL_TRACES_EXPORTER: 'otlp' OTEL_METRICS_EXPORTER: 'otlp' OTEL_LOGS_EXPORTER: 'otlp' OTEL_LOG_LEVEL: 'info' OTEL_METRIC_EXPORT_INTERVAL: '15000' OTEL_METRIC_EXPORT_TIMEOUT: '10000' ``` An important configuration change is required for customers upgrading to Custody Application version LTS 1.34.x. As part of our ongoing commitment to security and performance, version 1.34.x includes a major upgrade to the underlying OpenTelemetry (OTel) Java Agent to address recent upstream security vulnerabilities (CVEs). While this upgrade ensures your deployment remains secure, it introduces a behavioral change in how telemetry data is exported. This change only affects your environment if both of the following apply: - You are deploying or upgrading to Custody Application v1.34.x. - You have explicitly enabled OpenTelemetry monitoring in your deployment. In the newer version of the OpenTelemetry agent, the upstream maintainers changed the default export protocol from `grpc` to `http/protobuf`. However, standard OTel gRPC collectors typically listen on port `4317`. If you upgrade to v1.34.x without adjusting your configuration, the agent will attempt to send `http/protobuf` traffic to the gRPC port, resulting in your telemetry collector rejecting the data. You may see the following errors in your service logs, and your observability dashboards will stop receiving new data: ```text WARN - OTLP exporter endpoint port is likely incorrect for protocol version "http/protobuf". Port 4317 typically uses grpc. ERROR - Failed to export spans. Connection reset by peer ``` This issue only impacts observability and telemetry collection. Core application functionality, performance, and data integrity remain completely unaffected. To maintain your current telemetry flow and preserve the gRPC export behavior on port `4317`, you must explicitly define the protocol in your environment before starting the upgraded services. Apply one of the following configuration changes depending on your deployment method. **Option 1: Environment variable (recommended for Docker/Helm)** Add the following environment variable to your container or OS startup environment: ```bash OTEL_EXPORTER_OTLP_PROTOCOL="grpc" ``` **Option 2: Java Virtual Machine (JVM) options** If you manage JVM arguments directly, append the following system property: ```bash JAVA_TOOL_OPTIONS="-Dotel.exporter.otlp.protocol=grpc" ``` If you have any questions or require assistance implementing this configuration change, contact your assigned CPE or the [Technical Services team](https://metaco.atlassian.net/servicedesk/customer/portal/1). ### Managed environment variables Some OTEL SDK environment variables are already defined at either the application level or by the Helm charts. The application manages this variable: - `OTEL_SERVICE_NAME` (with a unique value for each Ripple Custody component and indexer) The Helm chart Go templates dynamically define the following values based on the YAML values found at `harmonize.telemetry.otel`: - `OTEL_SDK_DISABLED` - `OTEL_EXPORTER_OTLP_ENDPOINT` - `OTEL_EXPORTER_OTLP_LOGS_ENDPOINT` - `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` - `OTEL_EXPORTER_OTLP_METRICS_ENDPOINT` The charts override all of the environment variables listed above. Do not set these manually. ## Batch LogRecord Processor (BLRP) You must configure the Batch LogRecord Processor (BLRP) optimally to prevent loss of logs during intermittent outages of the OTEL endpoint (either OTEL Collector or another telemetry backend). These best practices also apply to traces with the OTEL SDK Batch Span Processor. For more information on how the BLRP works, see the OTEL SDK log specification. The goal is to strike a balance between memory consumption, export frequency, and network overhead. Regularly monitor the system to fine-tune the BLRP settings and ensure log export runs smoothly. ### BLRP configuration reference | Parameter | Type | Default | Description | | --- | --- | --- | --- | | `OTEL_BLRP_SCHEDULE_DELAY` | integer (ms) | `1000` | Interval between logs export attempts. See [Schedule delay](#schedule-delay) | | `OTEL_BLRP_EXPORT_TIMEOUT` | integer (ms) | `30000` | Time window for successful log exports. See [Export timeout](#export-timeout) | | `OTEL_BLRP_MAX_QUEUE_SIZE` | integer | `2048` | In-memory queue maximum buffer size retaining logs during OTEL endpoint unavailability. See [Max queue size](#max-queue-size) | | `OTEL_BLRP_MAX_EXPORT_BATCH_SIZE` | integer | `512` | Maximum logs export batch size. Must be less than or equal to `OTEL_BLRP_MAX_QUEUE_SIZE`. See [Max export batch size](#max-export-batch-size) | ### Schedule delay The `OTEL_BLRP_SCHEDULE_DELAY` variable specifies the interval between export attempts. If you reduce the default value, OTEL attempts to export logs more frequently, which can be beneficial for pushing logs when the endpoint is still alive or as soon as the endpoint is back online. A much shorter delay could lead to increased operational overhead if the endpoint remains unavailable for an extended period. #### Throttling and rate limiting More frequent export attempts mean more network packets sent out. This increases the load on the network, potentially causing congestion, especially if multiple instances or services are trying to export logs simultaneously. If the endpoint or any intermediate system (such as a firewall, gateway, router, or Kubernetes network policy) has rate-limiting mechanisms in place, very frequent export attempts might hit these limits, causing valid and important requests to be dropped or delayed. You can apply Kubernetes traffic shaping using the following pod annotations: - `kubernetes.io/ingress-bandwidth` - `kubernetes.io/egress-bandwidth` ```yaml apiVersion: v1 kind: Pod metadata: annotations: kubernetes.io/ingress-bandwidth: 1M kubernetes.io/egress-bandwidth: 1M ``` For more information, see network resource quota. #### CPU overhead Each export attempt requires CPU cycles to prepare and send the data. More frequent attempts can result in higher CPU usage. In the Helm chart configuration, you must set appropriate Kubernetes CPU resource settings for each component: - `resources.requests.cpu` - `resources.limits.cpu` ```yaml components: target-harmonize-component: resources: requests: cpu: 1 # MUST be custom based on needs. limits: cpu: 1 # MUST be custom based on needs. ``` ### Export timeout The `OTEL_BLRP_EXPORT_TIMEOUT` variable grants a more extended window for successful log exports, particularly during transient networking or OTEL endpoint glitches. Increasing this value directly affects the time the OTEL SDK is willing to wait for an acknowledgment or response from the OTEL endpoint after attempting to export logs. You also ideally need to increase this value in tandem with `OTEL_BLRP_MAX_QUEUE_SIZE` and pod memory allocation, to avoid potential backlog problems in the OTEL SDK queue. If each export attempt waits for an extended timeout period, it can cause subsequent export attempts to queue up or delay, especially if the system is generating logs at a high rate. This can create a backlog of logs waiting to be exported. ### Max queue size The `OTEL_BLRP_MAX_QUEUE_SIZE` variable controls the in-memory queue buffer. When the endpoint is unavailable, logs start to accumulate in the BLRP's queue. The OTEL SDK stores this queue in memory. Increasing this value offers a larger buffer to retain logs during short-lived OTEL and OTEL-compatible endpoint disruptions. In tandem with this change, you must also adjust the pod's memory resources: - `resources.requests.memory` - `resources.limits.memory` ```yaml components: target-harmonize-component: resources: requests: memory: "4Gi" # MUST be custom based on needs. limits: memory: "4Gi" # MUST be custom based on needs. ``` When you double the value of `OTEL_BLRP_MAX_QUEUE_SIZE`, also increase pod memory limits proportionally. This ensures the pod has adequate memory to manage the enlarged log queue without risking an `OutOfMemory` error. ### Max export batch size The `OTEL_BLRP_MAX_EXPORT_BATCH_SIZE` variable determines the maximum number of log records the system bundles together and exports in a single batch to the endpoint. The `OTEL_BLRP_MAX_EXPORT_BATCH_SIZE` value must always be equal to or lower than `OTEL_BLRP_MAX_QUEUE_SIZE`. If you've increased the queue size, consider adjusting this variable accordingly. A balanced batch size ensures that the restored endpoint isn't inundated with an overly large volume of logs all at once. The size of the batch should be in line with the capacity of the endpoint to process incoming logs. If the batch size exceeds the endpoint's processing capability, it may reject or drop the batch. For example, if you know that your endpoint can reliably handle 1000 log records per second, setting the batch size to a value far greater might overload the endpoint. A larger batch size means more data sent at once, which can result in network congestion or increased latency, especially in bandwidth-limited environments. Make sure the network configuration is adequate to handle large batch exports. When the OTEL endpoint is unavailable, logs start to accumulate in the BLRP's queue. Once the endpoint is available again, the OTEL SDK exports log batches larger than usual to clear the queue. A larger export batch size can help clear the queued logs faster, ensuring quicker recovery. However, balance this against the potential risk of overwhelming the endpoint. We recommend you start with a value that's aligned with the endpoint's known processing capacity and the available network bandwidth. Monitor the system's behavior, especially during and after endpoint outages. Adjust `OTEL_BLRP_MAX_EXPORT_BATCH_SIZE` based on real-world observations, ensuring that you achieve a balance between speedy recovery after outages and not overwhelming the endpoint or network. ## Related topics - [OTEL Collector deployment](/products/custody/v1.36/operations-and-maintenance/monitoring/otel-collector) - Deploy and configure the OTEL Collector - [Networking configuration](/products/custody/v1.36/deployment/reference/networking) - Network security and TLS settings