# Telemetry configuration Telemetry configuration is an important part of on-premise Ripple Custody deployments. It defines how the platform collects, processes, and sends telemetry data to the appropriate server. This page covers Helm chart configuration for telemetry. For an overview of telemetry concepts, see [Configure telemetry](/products/custody/v1.34/deployment/reference/telemetry). For OTEL Collector deployment, see [OTEL Collector deployment](/products/custody/v1.34/deployment/reference/telemetry-collector). ### Helm chart values.yaml Most of the fields in the `values.yaml` file of Ripple Custody Helm charts are Go templates that import and concatenate several neighboring fields. You can use them to configure telemetry in Ripple Custody in a granular way. For example, send the logs and traces to Elasticsearch and the metrics to Prometheus. The `telemetry` section has three main sub-sections: - General telemetry configuration - Syslog configuration - OTEL configuration The telemetry configuration fields for Ripple Custody are as follows: ```yaml harmonize: telemetry: # Possible values: `otlp` | `disabled` type: 'otlp' # Possible values: `http` | `https` protocol: 'https' host: 'my-telemetry-hostname.com' ``` The following describes each of the three `telemetry` sub-sections. #### General telemetry You can configure the Ripple Custody Helm charts to use an external telemetry endpoint, that is OTEL Collector or an OTEL-compatible telemetry backend. In the `values.yaml` file, the general telemetry configuration is in the following format: ```yaml Ripple Custody: telemetry: type: 'otlp' protocol: 'https' host: 'my-telemetry-hostname.com' ``` | Telemetry Field | type | default value | possible values | Description | | --- | --- | --- | --- | --- | | `type` | string | `disabled` | `disabled`, `otlp`, `jeager`, `azure` | Type of telemetry used | | `protocol` | string | `http` | `http`, `https` | Protocol used by Ripple Custody components to send telemetry data | | `host` | string | None | Any host | Hostname the telemetry data is sent to | The hostname must not include a port. This information is set up automatically for `otel-grpc` (4317), `otel-http` (4318), and `syslog` (54526). By default, the Helm charts check whether a local OTEL Collector exists and use it for the telemetry configuration. Disclaimer This setup is intended for testing purposes only. Do not use it in production. If the following field in is set to `true`: ```yaml opentelemetry-collector: enabled: true ``` The Helm charts import the telemetry configuration from OTEL Collector and inject it into the Ripple Custody telemetry configuration. #### OTEL ```yaml harmonize: telemetry: otel: enabled: true host: 'my-otel-host.com' # Possible values: `http` | `https` protocol: 'https' sdk-environment-variables: OTEL_TRACES_EXPORTER: 'otlp' OTEL_METRICS_EXPORTER: 'otlp' OTEL_LOGS_EXPORTER: 'otlp' OTEL_LOG_LEVEL: 'info' OTEL_METRIC_EXPORT_INTERVAL: '15000' OTEL_METRIC_EXPORT_TIMEOUT: '10000' ``` | Field name | Type | Format | Default Value | Description | | --- | --- | --- | --- | --- | | `enabled` | string/boolean | | Go template | Determines whether the OTEL Collector is enabled or not. | | `host` | string | Hostname | Go template | Specifies the host for the OTEL endpoint. By default, the value is determined by the `host` field in the `telemetry` configuration. | | `protocol` | string | Network Protocol (level 7) | Go template | Specifies the protocol used by the OTEL endpoint. By default, The value is determined by the `protocol` field in the `telemetry` configuration. | | `httpPort` | string | Port | Go template | Specifies the HTTP port for the OTEL endpoint. | | `grpcPort` | string | Port | Go template | Specifies the gRPC port for the OTEL endpoint. | | `httpEndpoint` | string | URI | Go template | Specifies the HTTP endpoint for the OTEL endpoint. The endpoint is constructed using the neighboring `protocol`, `host`, and `port`. | | `grpcEndpoint` | string | URI | Go template | Specifies the gRPC endpoint for the OTEL endpoint. The endpoint is constructed using the neighboring `protocol`, `host`, and `port`. | | `sdk-environment-variables` | Object | field of key/value pairs | | See OTEL SDK environment variables | | `logs` | object | Field of key-value pairs | | See [Configuration of multiple backends](#configuration-of-multiple-backends) | | `traces` | object | Field of key-value pairs | | See [Configuration of multiple backends](#configuration-of-multiple-backends) | | `metrics` | object | Field of key-value pairs | | See [Configuration of multiple backends](#configuration-of-multiple-backends) | The configuration provides parameters (port/endpoint) for both `gRPC` and `HTTP`. This is because different Ripple Custody components send telemetry data using either one or the other protocol. ##### Multiple backends The Ripple Custody Helm charts give you the option to specify data export to three different telemetry backends: ```yaml harmonize: telemetry: enabled: true otel: logs: host: 'my-otel-endpoint-for-logs.com' traces: host: 'my-otel-endpoint-for-traces.com' metrics: host: 'my-otel-endpoint-for-metrics.com' ``` ![Diagram showing OTEL SDK configured with separate endpoints for logs, traces, and metrics telemetry data](/assets/otel-sdk-multiple-otel-endpoints.43d68abbfebe6049a80a17df2b815b655f37ba73fe4fcd87eb8b88bf29bcfe78.322370a9.svg) The sub-fields `logs`, `traces` and `metrics` follow the same pattern, without the field `sdk-environment-variables`. ```yaml harmonize: telemetry: otel: # Logs config logs: # Enable or disable enabled: true host: 'my-otel-endpoint-for-logs.com' protocol: 'https' ``` The following configuration shows two different telemetry backends: 1. One telemetry backend for logs and traces. 2. One telemetry backend for metrics. ```yaml harmonize: telemetry: type: otlp protocol: http host: 'my-otel-endpoint-for-logs-and-traces.com' # E.g. Elasticsearch otel: metrics: host: 'my-otel-endpoint-for-metrics.com' # E.g Prometheus ``` ![Diagram showing OTEL SDK sending logs and traces to Elasticsearch and metrics to Prometheus](/assets/otel-sdk-elastic-search-prometheus.6eef3c9d761ac0e7fa48427064db1f4cc8d0c6764831bffdd4638ed1cbb53345.322370a9.svg) #### Syslog The Syslog configuration is in the following format: ```yaml syslog: enabled: true host: 'my-syslog-receiver-hostname.com' port: '54526' gatewayName: harmonize_syslog ``` | Field name | Type | Format | Default Value | Description | | --- | --- | --- | --- | --- | | `enabled` | boolean | | `true` | Whether the syslog is enabled or not. | | `protocol` | string | Network Protocol (level 4) | `tcp` | The protocol used by Syslog. | | `host` | string | Hostname | Determined by the `host` field in the `otel` configuration | The host for the syslog. | | `port` | int | | 54526 | The port for the Syslog. | | `gatewayName` | string | | `harmonize_syslog` | The name of the syslog gateway. | By default, the Go template imports values for some of the fields from the OTEL Collector sub-chart configuration. For more information, see [Telemetry data reception](#telemetry-data-reception). ### OTEL SDK environment variables You can specify OTEL SDK environment variables in Ripple Custody Helm charts. For more information about how to configure telemetry instrumentation and export, see the Open Telemetry OTEL SDK environment variable specification. By default, the Ripple Custody Helm charts configure some environment variables with default values: ```yaml harmonize: telemetry: otel: sdk-environment-variables: OTEL_TRACES_EXPORTER: 'otlp' OTEL_METRICS_EXPORTER: 'otlp' OTEL_LOGS_EXPORTER: 'otlp' OTEL_LOG_LEVEL: 'info' OTEL_METRIC_EXPORT_INTERVAL: '15000' OTEL_METRIC_EXPORT_TIMEOUT: '10000' ``` The system injects every environment variable specified in this field into every pod that uses the OTEL SDK. An important configuration change is required for customers upgrading to Custody Application version LTS 1.34.x. As part of our ongoing commitment to security and performance, version 1.34.x includes a major upgrade to the underlying OpenTelemetry (OTel) Java Agent to address recent upstream security vulnerabilities (CVEs). While this upgrade ensures your deployment remains secure, it introduces a behavioral change in how telemetry data is exported. This change only affects your environment if both of the following apply: - You are deploying or upgrading to Custody Application v1.34.x. - You have explicitly enabled OpenTelemetry monitoring in your deployment. In the newer version of the OpenTelemetry agent, the default export protocol has been changed by the upstream maintainers from `grpc` to `http/protobuf`. However, standard OTel gRPC collectors typically listen on port `4317`. If you upgrade to v1.34.x without adjusting your configuration, the agent will attempt to send `http/protobuf` traffic to the gRPC port, resulting in your telemetry collector rejecting the data. You may see the following errors in your service logs, and your observability dashboards will stop receiving new data: ```text WARN - OTLP exporter endpoint port is likely incorrect for protocol version "http/protobuf". Port 4317 typically uses grpc. ERROR - Failed to export spans. Connection reset by peer ``` This issue only impacts observability and telemetry collection. Core application functionality, performance, and data integrity remain completely unaffected. To maintain your current telemetry flow and preserve the gRPC export behavior on port `4317`, you must explicitly define the protocol in your environment before starting the upgraded services. Apply one of the following configuration changes depending on your deployment method. **Option 1: Environment variable (recommended for Docker/Helm)** Add the following environment variable to your container or OS startup environment: ```bash OTEL_EXPORTER_OTLP_PROTOCOL="grpc" ``` **Option 2: Java Virtual Machine (JVM) options** If you manage JVM arguments directly, append the following system property: ```bash JAVA_TOOL_OPTIONS="-Dotel.exporter.otlp.protocol=grpc" ``` If you have any questions or require assistance implementing this configuration change, contact your assigned CPE or the [Technical Services team](https://metaco.atlassian.net/servicedesk/customer/portal/1). #### Managed environment variables Some OTEL SDK environment variables are already defined at either the application level or by the Helm charts. The application handles these OTEL SDK environment variables: - `OTEL_SERVICE_NAME` (with a unique value for each Ripple Custody component and indexer) The Helm charts Go Templates dynamically define those values based on the YAML values found in the `values.yaml` file at location `harmonize.telemetry.otel`: - `OTEL_SDK_DISABLED` - `OTEL_EXPORTER_OTLP_ENDPOINT` - `OTEL_EXPORTER_OTLP_LOGS_ENDPOINT` - `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` - `OTEL_EXPORTER_OTLP_METRICS_ENDPOINT` The charts override all of the environment variables listed above. #### Logs best practices You must configure the Batch LogRecord Processor (BLRP) optimally to prevent loss of logs during intermittent outages of the OTEL endpoint (either OTEL Collector or another telemetry backend). The logs best practices also apply to traces with the OTEL SDK Batch Span Processor. For more information on how the BLRP works, see the OTEL SDK log specification. The goal is to strike a balance between memory consumption, export frequency, and network overhead. Regularly monitor the system to fine-tune the BLRP settings and ensure log export runs smoothly. The following table lists the environment variables available to configure the BLRP: | Environment variable Name | Type | Format | Default Value | Description | | --- | --- | --- | --- | --- | | `OTEL_BLRP_SCHEDULE_DELAY` | integer | milliseconds | 1000 (OTEL SDK default value) | OTEL SDK interval between logs export attempts. See [OTEL_BLRP_SCHEDULE_DELAY](#otel_blrp_schedule_delay). | | `OTEL_BLRP_EXPORT_TIMEOUT` | integer | milliseconds | 30000 (OTEL SDK default value) | OTEL SDK time window for successful log exports. See [OTEL_BLRP_EXPORT_TIMEOUT](#otel_blrp_export_timeout). | | `OTEL_BLRP_MAX_QUEUE_SIZE` | integer | milliseconds | 2048 (OTEL SDK default value) | OTEL SDK in-memory queue maximum buffer size retaining logs in case of OTEL endpoint unavailability. See [OTEL_BLRP_MAX_QUEUE_SIZE](#otel_blrp_max_queue_size). | | `OTEL_BLRP_MAX_EXPORT_BATCH_SIZE` | integer | milliseconds | 512 (OTEL SDK default value) | OTEL SDK maximum logs export batch size. Must be less than or equal to `OTEL_BLRP_MAX_QUEUE_SIZE`. See [OTEL_BLRP_MAX_EXPORT_BATCH_SIZE](#otel_blrp_max_export_batch_size). | ##### OTEL_BLRP_SCHEDULE_DELAY This variable specifies the interval between export attempts. If you reduce the default value, OTEL attempts to export logs more frequently, which can be beneficial for pushing logs when the endpoint is still alive or as soon as the endpoint is back online. A much shorter delay could lead to increased operational overhead if the endpoint remains unavailable for an extended period. ###### Throttling and rate limiting More frequent export attempts mean more network packets sent out. This increases the load on the network, potentially causing congestion, especially if multiple instances or services are trying to export logs simultaneously. If the endpoint or any intermediate system, such as a firewall, gateway, router, or Kubernetes network policy, has rate-limiting mechanisms in place, very frequent export attempts might hit these limits, causing valid and important requests to be dropped or delayed. You can apply Kubernetes traffic shaping using the following pod annotations: - `kubernetes.io/ingress-bandwidth` - `kubernetes.io/egress-bandwidth` ```yaml apiVersion: v1 kind: Pod metadata: annotations: kubernetes.io/ingress-bandwidth: 1M kubernetes.io/egress-bandwidth: 1M ``` For more information, see network resource quota. ###### CPU overhead Each export attempt requires CPU cycles to prepare and send the data. More frequent attempts can result in higher CPU usage. In the Helm chart configuration, you must set appropriate Kubernetes CPU resource settings for each component: - `resources.requests.cpu` - `resources.limits.cpu` ```yaml components: target-harmonize-component: resources: requests: cpu: 1 # MUST be custom based on needs. limits: cpu: 1 # MUST be custom based on needs. ``` ##### OTEL_BLRP_EXPORT_TIMEOUT This variable grants a more extended window for successful log exports, particularly pertinent during transient networking or OTEL endpoint glitches. Increasing this value directly affects the time the OTEL SDK is willing to wait for an acknowledgment or response from the OTEL endpoint after attempting to export logs. You also ideally need to increase this value in tandem with `OTEL_BLRP_MAX_QUEUE_SIZE` and pod memory allocation, to avoid potential backlog problems in the OTEL SDK queue. If each export attempt waits for an extended timeout period, it can cause subsequent export attempts to queue up or delay, especially if the system is generating logs at a high rate. This can create a backlog of logs waiting to be exported. ##### OTEL_BLRP_MAX_QUEUE_SIZE When the endpoint is unavailable, logs start to accumulate in the BLRP's queue. The OTEL SDK stores this queue in memory. Increasing this value offers a larger buffer to retain logs during short-lived OTEL and OTEL-compatible endpoint disruptions. In tandem with this change, you must also adjust the pod's memory resources. In the configuration, appropriate Kubernetes **Memory** resource settings must be set for each component: - `resources.requests.memory` - `resources.limits.memory` ```yaml components: target-harmonize-component: resources: requests: memory: "4Gi" # MUST be custom based on needs. limits: memory: "4Gi" # MUST be custom based on needs. ``` When you double the value of `OTEL_BLRP_MAX_QUEUE_SIZE`, also increase pod memory limits proportionally. This ensures the pod has adequate memory to manage the enlarged log queue without risking an `OutOfMemory` error. ##### OTEL_BLRP_MAX_EXPORT_BATCH_SIZE This variable determines the maximum number of log records the system bundles together and exports in a single batch to the endpoint. Configuring this parameter effectively is crucial for both performance and reliability. The `OTEL_BLRP_MAX_EXPORT_BATCH_SIZE` value must always be equal to or lower than `OTEL_BLRP_MAX_QUEUE_SIZE`. If you've increased the queue size, consider adjusting this variable accordingly. A balanced batch size ensures that the restored endpoint isn't inundated with an overly large volume of logs all at once. The size of the batch should be in line with the capacity of the endpoint to process incoming logs. If the batch size exceeds the endpoint's processing capability, it may reject or drop the batch. For example, if you know that your endpoint can reliably handle 1000 log records per second, setting the batch size to a value far greater might overload the endpoint. A larger batch size means more data sent at once, which can result in network congestion or increased latency, especially in bandwidth-limited environments. Make sure the network configuration is adequate to handle large batch exports. When the OTEL endpoint is unavailable, logs start to accumulate in the BLRP's queue. Once the endpoint is available again, the OTEL SDK exports log batches larger than usual to clear the queue. A larger export batch size can help clear the queued logs faster, ensuring quicker recovery. However, balance this against the potential risk of overwhelming the endpoint. We recommend you start with a value that's aligned with the endpoint's known processing capacity and the available network bandwidth. Monitor the system's behavior, especially during and after endpoint outages. Adjust `OTEL_BLRP_MAX_EXPORT_BATCH_SIZE` based on real-world observations, ensuring that you achieve a balance between speedy recovery after outages and not overwhelming the endpoint or network.