# Configure telemetry Telemetry configuration is an important part of on-premise Ripple Custody deployments. It defines how telemetry data is collected, processed, and sent to the appropriate server. This task applies to on-premise deployments only. ## Prerequisites Before you configure telemetry, you need a basic understanding of the following: - Kubernetes - Helm chart configuration - Helm chart configuration, especially values.yaml - The Open Telemetry Protocol - Open Telemetry Collector ## Telemetry in Ripple Custody Ripple Custody supports two types of telemetry protocols: - Open Telemetry Protocol (OTLP) - Remote System Logs (rSysLog) ### Open Telemetry (OTEL) OTEL is an open-source observability framework designed to generate, collect, and describe telemetry data from applications such as Ripple Custody. Open Telemetry Protocol (OTLP) is a set of APIs, libraries, agents, and instrumentation standards for the generation, collection, and description of telemetry data (traces, metrics, and logs). OTEL refers to the Open Telemetry project, whereas OTLP refers to the set of standards for implementing and using OTEL. Implementation of OTEL in Ripple Custody can help you diagnose and fix issues, optimize performance, and understand user behavior. For more information about the vendors officially supporting OTEL, see Vendor list supporting OTEL. For a deep dive into how it works and how to use it, see the OTEL project's official documentation. #### Open Telemetry Protocol (OTLP) OTLP is the encoding protocol used by OTEL to transmit telemetry data from client applications to backends. The OTLP specification document describes the encoding, transport, and delivery mechanism of telemetry data between telemetry sources, intermediate nodes such as collectors, and telemetry backends. OTEL automatically captures telemetry data, such as latency or error data, from a Ripple Custody deployment. This data is enriched with context, such as user or transaction information, processed, and then sent to OTEL observability backends for analysis. #### Data sources OTEL captures data from the following sources: - Traces: Flow and latency of requests and transactions through microservices and distributed systems. - Metrics: State of the system at a specific point in time, for example CPU usage, memory consumption, or number of active users. - Logs: Application, system, and device logs. #### Open Telemetry Software Development Kit (OTEL SDK) The OTEL SDK library facilitates the export of telemetry data through OTLP to OTEL compatible endpoints. Most Ripple Custody components and indexers can export telemetry data using the OTEL SDK. ![OTEL SDK](/assets/otel-sdk.5e6a9005a29280832237fb5253123ae918d98470519e221435089cdef11a7dcb.6c4d45f5.svg) You can find more information about support for the OTEL SDK in the Open Telemetry Compliance Matrix. ### Remote System Logging (rSyslog) rSyslog is a syslog daemon standard in Unix-based systems for sending logging messages across IP networks. It uses Syslog, a standard for message logging in Unix-based systems for troubleshooting, auditing, and monitoring systems. rSyslog allows processes to send event notification messages over TCP, UDP, and RELP (reliable event logging protocol) to event message collectors. Messages contain data from logs generated by the operating system, collected by the Syslog API, and sent to an rSyslog daemon. Syslog cannot change the format or the information which is included. ![OTEL Collector to Elasticsearch](/assets/syslog.eddcc45a53d55fa5aaf9360fd28088e7753e7a3ce0d684257f92d4d458994f06.6c4d45f5.svg) Many telemetry platforms can ingest Syslog/rSyslog messages directly, using the following methods: - Agents: Some platforms use agents, for example OTEL Collector, to pull or receive syslog events and transform them into a format suitable for the telemetry platform. - Brokers/processors: Middleware, for example Apache Kafka or Logstash, can consume logs from rSyslog and distribute them to multiple destinations, including telemetry systems. ## Telemetry best practices This table provides an overview of the strengths and weaknesses of the protocols. Although both protocols are supported by Ripple Custody, we recommend you use OLTP. | Criteria | OTLP | rSyslog | | --- | --- | --- | | Purpose | Specifically designed for telemetry | Designed for event and logging messages | | Design | Supports request/response and streaming export | Doesn't natively understand telemetry nuances | | Efficiency | Efficient serialization/deserialization | Text-based encoding may be less efficient | | Integration with observability tools | Integrated into broad ecosystem with good support in modern tools | Widely supported due to longevity, but might not be optimized for modern telemetry tools | | Flexibility and evolution | Expected to evolve quickly, not constrained by legacy | Evolution might be challenging due to wide adoption | | Multiplexing | Supports multiplexed telemetry types | Doesn't natively support multiplexing | | Reliability | Built-in features like acknowledgments, retries, and back-off | Mechanisms for reliability exist, but might not be as comprehensive | | Context preservation | Can preserve and transmit context information | Doesn't have native mechanisms for telemetry context | ### OTEL best practices We recommend you use the following best practices for OTEL integration. #### OTEL SDK Before you use the OTEL SDK, you need to familiarize yourself with its internal workings and configure it using the OTEL environment variable specification. An important part of the OTEL SDK configuration is the Batch LogRecord Processor (BLRP). [Logs best practices](#logs-best-practices) describes the BLRP environment variables in the context of Ripple Custody. ##### OTEL endpoints OTLP is a request-response style protocol: the clients send requests, and the server replies with corresponding responses. The client continuously sends a sequence of requests to the server and expects to receive a response to each request: ![OTLP Request Response diagram](/assets/otlp-request-response.c7904b271217572bc40526e3a70f574b77a3b76fe13a8883638386ae43893589.6c4d45f5.png) OTLP aims to ensure that no data is lost in transit between client-server node pairs. However, many telemetry collection systems have intermediary nodes that the data must travel through before reaching the final destination, for example, application -> agent -> collector -> backend. OTLP does not provide end-to-end delivery guarantees. The acknowledgements described below happen between a single client-server pair and do not span intermediary nodes in multi-hop delivery paths. The OTEL SDK provides out-of-the-box support for data reception acknowledgement and for temporarily queuing the logs in memory if the OTEL endpoint is down. If the OTEL endpoint of the OTEL SDK is not the final destination, but an intermediary between the OTEL SDK and the final telemetry data store, for example, OTEL Collector, the telemetry data stays in the working memory of the intermediary between the OTLP reception acknowledgment and its export to the final data store. If the intermediary crashes, there is no guarantee that the telemetry data will persist and it may be lost. We recommend that you configure the telemetry collection systems so that the OTEL SDK directly exports the logs and other telemetry data directly to the final destination to guarantee the logs are not lost in transit, as long as the endpoint supports OTLP. #### OTEL Collector OTEL Collector is a vendor-agnostic implementation to receive, process, and export telemetry data. It removes the need to maintain multiple agents and collectors to which instrumentation libraries export their telemetry data. It also supports sending Open Source observability data format, for example, Jaeger, Prometheus, and Fluent Bit, to open source or commercial backends. ![OTEL Collector to Elasticsearch](/assets/otel-collector.25bc82c63c2c02c19c6464274937f33cc73f1ed6ec7b50acb0118b863ee1b6eb.6c4d45f5.svg) The Ripple Custody components use the OTEL SDK and rSyslog to send telemetry data to OTEL Collector. You can use OTEL Collector in infrastructures where the telemetry backends do not support OTLP out-of-the-box. OTEL Collector handles logs, traces, and metrics through a pipeline. Each telemetry type (logs, traces, or metrics) is handled through a dedicated pipeline. OTEL collector logs We recommend that you handle logs exported from the OTEL SDK and logs exported from Syslog through separate pipelines in OTEL Collector. They have different key-value fields that may not be mutually compatible. For information on how to deploy OTEL Collector, see [OTEL Collector deployment](#otel-collector-deployment). ### rSyslog best practices When using the rSyslog protocol with a Syslog receiver, we recommend you use the TCP network protocol, since log truncation is a known limitation of using rSyslog with the UDP protocol. Known UDP network protocol limitations with rSyslog: - UDP size limitations: The maximum size of a UDP datagram is technically 65,535 bytes, but in practice, due to the limitations imposed by the IPv4 protocol (which caps the maximum IP packet size at 65,535 bytes, including header information) and Ethernet frame size, syslog messages are often limited to much smaller sizes. Typically, most devices and platforms will truncate or split syslog messages sent over UDP that exceed 1,500 bytes, or, in many cases, even less. 1. Reliability: UDP is a connectionless protocol, which means there's no guarantee that the packets will arrive at their destination nor in the correct order. There's no built-in mechanism for retransmitting lost packets, as there is with TCP. ## Telemetry configuration Telemetry configuration is an important part of on-premise Ripple Custody deployments. It defines how telemetry data is collected, processed, and sent to the appropriate server. ### Helm chart values.yaml file Most of the fields in the `values.yaml` file of Ripple Custody Helm charts are Go templates that import and concatenate several neighboring fields. You can use them to configure telemetry in Ripple Custody in a granular way. For example, send the logs and traces to Elasticsearch and the metrics to Prometheus. The `telemetry` section has three main sub-sections: - General telemetry configuration - Syslog configuration - OTEL configuration The telemetry configuration fields for Ripple Custody are as follows: ```yaml harmonize: telemetry: # Possible values: `otlp` | `disabled` type: 'otlp' # Possible values: `http` | `https` protocol: 'https' host: 'my-telemetry-hostname.com' ``` The sub-sections are described below. #### General telemetry configuration The Ripple Custody Helm charts can be configured to use an external telemetry endpoint, that is OTEL Collector or an OTEL-compatible telemetry backend. In the `values.yaml` file, the general telemetry configuration is in the following format: ```yaml Ripple Custody: telemetry: type: 'otlp' protocol: 'https' host: 'my-telemetry-hostname.com' ``` | Telemetry Field | type | default value | possible values | Description | | --- | --- | --- | --- | --- | | `type` | string | `disabled` | `disabled`, `otlp`, `jeager`, `azure` | Type of telemetry used | | `protocol` | string | `http` | `http`, `https` | Protocol used by Ripple Custody components to send telemetry data | | `host` | string | None | Any host | Hostname the telemetry data is sent to | The hostname must not include a port. This information is set up automatically for `otel-grpc` (4317), `otel-http` (4318), and `syslog` (54526). By default, the Helm charts are designed to check whether a local OTEL Collector is deployed, and use it for the telemetry configuration. Disclaimer This setup is intended for testing purposes only. It is not suitable for use in production. If the following field in is set to `true`: ```yaml opentelemetry-collector: enabled: true ``` The Helm charts import the telemetry configuration from OTEL Collector and inject it into the Ripple Custody telemetry configuration. #### OTEL configuration ```yaml harmonize: telemetry: otel: enabled: true host: 'my-otel-host.com' # Possible values: `http` | `https` protocol: 'https' sdk-environment-variables: OTEL_TRACES_EXPORTER: 'otlp' OTEL_METRICS_EXPORTER: 'otlp' OTEL_LOGS_EXPORTER: 'otlp' OTEL_LOG_LEVEL: 'info' OTEL_METRIC_EXPORT_INTERVAL: '15000' OTEL_METRIC_EXPORT_TIMEOUT: '10000' ``` | Field name | Type | Format | Default Value | Description | | --- | --- | --- | --- | --- | | `enabled` | string/boolean | | Go template | Determines whether the OTEL Collector is enabled or not. | | `host` | string | Hostname | Go template | Specifies the host for the OTEL endpoint. By default, the value is determined by the `host` field in the `telemetry` configuration. | | `protocol` | string | Network Protocol (level 7) | Go template | Specifies the protocol used by the OTEL endpoint. By default, The value is determined by the `protocol` field in the `telemetry` configuration. | | `httpPort` | string | Port | Go template | Specifies the HTTP port for the OTEL endpoint. | | `grpcPort` | string | Port | Go template | Specifies the gRPC port for the OTEL endpoint. | | `httpEndpoint` | string | URI | Go template | Specifies the HTTP endpoint for the OTEL endpoint. The endpoint is constructed using the neighboring `protocol`, `host`, and `port`. | | `grpcEndpoint` | string | URI | Go template | Specifies the gRPC endpoint for the OTEL endpoint. The endpoint is constructed using the neighboring `protocol`, `host`, and `port`. | | `sdk-environment-variables` | Object | field of key/value pairs | | See OTEL SDK environment variables | | `logs` | object | Field of key-value pairs | | See [Configuration of multiple backends](#configuration-of-multiple-backends) | | `traces` | object | Field of key-value pairs | | See [Configuration of multiple backends](#configuration-of-multiple-backends) | | `metrics` | object | Field of key-value pairs | | See [Configuration of multiple backends](#configuration-of-multiple-backends) | The parameters (port/endpoint) for both `gRPC` and a `HTTP` are provided. This is because different Ripple Custody components send telemetry data using either one or the other protocol. ##### Configuration of multiple backends The Ripple Custody Helm charts give you the option to specify data export to three different telemetry backends: ```yaml harmonize: telemetry: enabled: true otel: logs: host: 'my-otel-endpoint-for-logs.com' traces: host: 'my-otel-endpoint-for-traces.com' metrics: host: 'my-otel-endpoint-for-metrics.com' ``` ![OTEL SDK](/assets/otel-sdk-multiple-otel-endpoints.43d68abbfebe6049a80a17df2b815b655f37ba73fe4fcd87eb8b88bf29bcfe78.6c4d45f5.svg) The sub-fields `logs`, `traces` and `metrics` follow the same pattern, without the field `sdk-environment-variables`. ```yaml harmonize: telemetry: otel: # Logs config logs: # Enable or disable enabled: true host: 'my-otel-endpoint-for-logs.com' protocol: 'https' ``` The following configuration shows two different telemetry backends: 1. One telemetry backend for logs and traces. 2. One telemetry backend for metrics. ```yaml harmonize: telemetry: type: otlp protocol: http host: 'my-otel-endpoint-for-logs-and-traces.com' # E.g. Elasticsearch otel: metrics: host: 'my-otel-endpoint-for-metrics.com' # E.g Prometheus ``` ![OTEL SDK](/assets/otel-sdk-elastic-search-prometheus.6eef3c9d761ac0e7fa48427064db1f4cc8d0c6764831bffdd4638ed1cbb53345.6c4d45f5.svg) #### Syslog configuration The Syslog configuration is in the following format: ```yaml syslog: enabled: true host: 'my-syslog-receiver-hostname.com' port: '54526' gatewayName: harmonize_syslog ``` | Field name | Type | Format | Default Value | Description | | --- | --- | --- | --- | --- | | `enabled` | boolean | | `true` | Whether the syslog is enabled or not. | | `protocol` | string | Network Protocol (level 4) | `tcp` | The protocol used by Syslog. | | `host` | string | Hostname | Determined by the `host` field in the `otel` configuration | The host for the syslog. | | `port` | int | | 54526 | The port for the Syslog. | | `gatewayName` | string | | `harmonize_syslog` | The name of the syslog gateway. | By default, the Go template imports values for some of the fields from the OTEL Collector sub-chart configuration. For more information, see [Telemetry data reception](#telemetry-data-reception). ### OTEL SDK environment variables You can specify OTEL SDK environment variables in Ripple Custody Helm charts. For more information about how to configure telemetry instrumentation and export, see the Open Telemetry OTEL SDK environment variable specification. By default, the Ripple Custody Helm charts are configured with default values for some environment variables: ```yaml harmonize: telemetry: otel: sdk-environment-variables: OTEL_TRACES_EXPORTER: 'otlp' OTEL_METRICS_EXPORTER: 'otlp' OTEL_LOGS_EXPORTER: 'otlp' OTEL_LOG_LEVEL: 'info' OTEL_METRIC_EXPORT_INTERVAL: '15000' OTEL_METRIC_EXPORT_TIMEOUT: '10000' ``` Every environment variable specified in this field will be injected in every pod that uses the OTEL SDK. #### Managed environment variables Some OTEL SDK environment variables are already either defined at the application level or by the Helm charts. These OTEL SDK environment variables are handled at the application level: - `OTEL_SERVICE_NAME` (with a unique value for each Ripple Custody component and indexer) Those values are dynamically defined by the Helm charts Go Templates, based on the YAML values found in the `values.yaml` file at location `harmonize.telemetry.otel`: - `OTEL_SDK_DISABLED` - `OTEL_EXPORTER_OTLP_ENDPOINT` - `OTEL_EXPORTER_OTLP_LOGS_ENDPOINT` - `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` - `OTEL_EXPORTER_OTLP_METRICS_ENDPOINT` The charts override all of the environment variables listed above. #### Logs best practices You must configure the Batch LogRecord Processor (BLRP) optimally to prevent loss of logs during intermittent outages of the OTEL endpoint (either OTEL Collector or another telemetry backend). The logs best practices also apply to traces with the OTEL SDK Batch Span Processor. For more information on how the BLRP works, see the OTEL SDK log specification. The goal is to strike a balance between memory consumption, export frequency, and network overhead. You must ensure the system is regularly monitored to fine-tune the BLRP settings and ensure log export runs smoothly. The following table lists the environment variables available to configure the BLRP: | Environment variable Name | Type | Format | Default Value | Description | | --- | --- | --- | --- | --- | | `OTEL_BLRP_SCHEDULE_DELAY` | integer | milliseconds | 1000 (OTEL SDK default value) | OTEL SDK interval between logs export attempts. See [OTEL_BLRP_SCHEDULE_DELAY](#otel_blrp_schedule_delay). | | `OTEL_BLRP_EXPORT_TIMEOUT` | integer | milliseconds | 30000 (OTEL SDK default value) | OTEL SDK time window for successful log exports. See [OTEL_BLRP_EXPORT_TIMEOUT](#otel_blrp_export_timeout). | | `OTEL_BLRP_MAX_QUEUE_SIZE` | integer | milliseconds | 2048 (OTEL SDK default value) | OTEL SDK in-memory queue maximum buffer size retaining logs in case of OTEL endpoint unavailability. See [OTEL_BLRP_MAX_QUEUE_SIZE](#otel_blrp_max_queue_size). | | `OTEL_BLRP_MAX_EXPORT_BATCH_SIZE` | integer | milliseconds | 512 (OTEL SDK default value) | OTEL SDK maximum logs export batch size. Must be less than or equal to `OTEL_BLRP_MAX_QUEUE_SIZE`. See [OTEL_BLRP_MAX_EXPORT_BATCH_SIZE](#otel_blrp_max_export_batch_size). | ##### OTEL_BLRP_SCHEDULE_DELAY This variable specifies the interval between export attempts. If you reduce the default value, OTEL attempts to export logs more frequently, which can be beneficial for pushing logs when the endpoint is still alive or as soon as the endpoint is back online. A much shorter delay could lead to increased operational overhead if the endpoint remains unavailable for an extended period. ###### Throttling and rate-limiting More frequent export attempts mean more network packets sent out. This increases the load on the network, potentially causing congestion, especially if multiple instances or services are trying to export logs simultaneously. If the endpoint or any intermediate system, such as a firewall, gateway, router, or Kubernetes network policy, has rate-limiting mechanisms in place, very frequent export attempts might hit these limits, causing valid and important requests to be dropped or delayed. Kubernetes traffic shaping can be applied using the following pod annotations: - `kubernetes.io/ingress-bandwidth` - `kubernetes.io/egress-bandwidth` ```yaml apiVersion: v1 kind: Pod metadata: annotations: kubernetes.io/ingress-bandwidth: 1M kubernetes.io/egress-bandwidth: 1M ``` For more information, see network resource quota. ###### CPU overhead Each export attempt requires CPU cycles to prepare and send the data. More frequent attempts can result in higher CPU usage. In the Helm chart configuration, you must set appropriate Kubernetes CPU resource settings for each component: - `resources.requests.cpu` - `resources.limits.cpu` ```yaml components: target-harmonize-component: resources: requests: cpu: 1 # MUST be custom based on needs. limits: cpu: 1 # MUST be custom based on needs. ``` ##### OTEL_BLRP_EXPORT_TIMEOUT Elevate this to grant a more extended window for successful log exports, particularly pertinent during transient networking or OTEL endpoint glitches. Increasing this value directly affects the time the OTEL SDK is willing to wait for an acknowledgment or response from the OTEL endpoint after attempting to export logs. Increase this environment variable in tandem with `OTEL_BLRP_MAX_QUEUE_SIZE` and pod memory allocation to avoid potential backlog problems in the OTEL SDK queue. If each export attempt is waiting for an extended timeout period, it can cause subsequent export attempts to queue up or delay, especially if the system is generating logs at a high rate. This can create a backlog of logs waiting to be exported. Elevating value grants a more extended window for successful log exports, particularly pertinent during transient networking or OTEL endpoint glitches. Increasing this value directly affects the time the OTEL SDK is willing to wait for an acknowledgment or response from the OTEL endpoint after attempting to export logs. You also ideally need to increase this value in tandem with `OTEL_BLRP_MAX_QUEUE_SIZE` and pod memory allocation, to avoid potential backlog problems in the OTEL SDK queue. If each export attempt waits for an extended timeout period, it can cause subsequent export attempts to queue up or delay, especially if the system is generating logs at a high rate. This can create a backlog of logs waiting to be exported. ##### OTEL_BLRP_MAX_QUEUE_SIZE When the endpoint is unavailable, logs start to accumulate in the BLRP's queue. This queue is stored in memory and handled by the OTEL SDK. Increasing this value offers a larger buffer to retain logs during short-lived OTEL and OTEL-compatible endpoint disruptions. In tandem with this change, the pod's memory resources must also be adjusted. In the configuration, appropriate Kubernetes **Memory** resource settings must be set for each component: - `resources.requests.memory` - `resources.limits.memory` ```yaml components: target-harmonize-component: resources: requests: memory: "4Gi" # MUST be custom based on needs. limits: memory: "4Gi" # MUST be custom based on needs. ``` When the value of `OTEL_BLRP_MAX_QUEUE_SIZE` is doubled, also increase pod memory limits proportionally. This ensures the pod has adequate memory to manage the enlarged log queue without risking an `OutOfMemory` error. ##### OTEL_BLRP_MAX_EXPORT_BATCH_SIZE This variable determines the maximum number of log records that will be bundled together and exported in a single batch to the endpoint. Configuring this parameter effectively is crucial for both performance and reliability. If you've increased the queue size, consider adjusting this variable accordingly, but ensure it doesn't exceed the `OTEL_BLRP_MAX_QUEUE_SIZE`. A balanced batch size ensures that the restored endpoint isn't inundated with an overly large volume of logs all at once. The size of the batch should be in line with the capacity of the endpoint to process incoming logs. If the batch size exceeds the endpoint's processing capability, it may reject or drop the batch. For example, if you know that your endpoint can reliably handle 1000 log records per second, if you set the batch size to a value far greater, you might overload the endpoint. A larger batch size means more data sent at once, which can result in network congestion or increased latency, especially in bandwidth-limited environments, so make sure the network configuration is adequate to handle large batch exports. When the OTEL endpoint is unavailable, logs start to accumulate in the BLRP's queue, with the maximum queue size determined by the environment variable `OTEL_BLRP_MAX_QUEUE_SIZE`. Once the endpoint is available again, the OTEL SDK will export log batches larger than usual to clear the queue. A larger export batch size can help clear the queued logs faster when the endpoint becomes available after an outage, ensuring quicker recovery and a return to the normal amount of queued logs in the component memory. However, this advantage needs to be balanced against the potential risk of overwhelming the endpoint. We recommend you start with a value that's aligned with the endpoint's known processing capacity and the available network bandwidth. Monitor the system's behavior, especially during and after endpoint outages. Adjust `OTEL_BLRP_MAX_EXPORT_BATCH_SIZE` based on real-world observations, that is monitoring failed batches and adjusting based on observed failures, ensuring that you achieve a balance between speedy recovery after outages and not overwhelming the endpoint or network. This variable determines the maximum number of log records that will be bundled together and exported in a single batch to the endpoint. Configuring this parameter effectively is crucial for both performance and reliability. The `OTEL_BLRP_MAX_EXPORT_BATCH_SIZE` value must always be equal to or lower than `OTEL_BLRP_MAX_QUEUE_SIZE`. A reasonable batch size ensures that the restored endpoint isn't inundated with an overly large volume of logs all at once. The size of the batch should be in line with the capacity of the endpoint to process incoming logs. If the batch size exceeds the endpoint's processing capability, it may reject or drop the batch. For instance, if you know the OTEL endpoint can reliably handle maximum 1000 logs records per second, setting the batch size to a value far greater might overload the OTEL endpoint. A larger batch size means more data sent at once, which might result in network congestion or increased latency, especially in bandwidth-limited environments. You need to make sure the network configuration is large enough to have large batch export. ### OTEL Collector deployment The Ripple Custody Helm charts include an OTEL Collector instance: ```yaml opentelemetry-collector: enabled: false # Set to `false` by default ``` This instance is only to be used for testing purposes. You must not use it for production workloads. We recommend you set up a dedicated and segregated OTEL Collector instance. You must deploy OTEL Collector in line with your own requirements. For more information on OTEL Collector deployment, see the following: - Kubernetes Open Telemetry documentation - The official OTEL Collector Helm charts The recommended best practice for production workload is to set up a dedicated OTEL Collector deployment that is segregated from Ripple Custody. All the YAML code snippets in the steps below are specified in the `opentelemetry-collector` section, as a reminder that this is the OTEL Collector Helm `values.yaml` file configuration and not the Ripple Custody Helm configuration. For detailed OTEL Collector Values File configuration, see the Official Helm Chart for OTEL Collector. #### Deploy OTEL Collector You can set up OTEL Collector using two different configurations: 1. A dedicated OTEL Collector instance for both the trusted and untrusted components. 2. Two dedicated and separate instances (one instance for the trusted components and one instance for the untrusted components). For more information on trusted and untrusted components, see [Security](/products/custody/v1.26/overview/architecture/security#key-concepts). To set up an OTEL Collector instance: 1. Set the `enabled` variable to `true`: ```yaml opentelemetry-collector: enabled: true ``` This automatically deploys an OTEL Collector pod alongside the Ripple Custody deployment and sets the appropriate environment variables for the pods to discover the OTEL Collector service. 1. The default resource values might be too low for a typical deployment. Set the resource value to an appropriate level, as follows: 1. Run load testing on the Ripple Custody instance, at the peak usage you expect the Ripple Custody instance to encounter and observe resource usage. 2. Set the following values accordingly: ```yaml resources: requests: cpu: 4 memory: 4Gi limits: cpu: 8 memory: 16Gi ``` The Helm `values.yaml` file settings must reflect your usage of Ripple Custody. Those values need to be high enough that the **opentelemetry-collector** pod does not encounter a `Pending` or `CrashLoopBackOff` Kubernetes state. The higher the number of users, the greater the volume of telemetry data generated, and the higher the resource consumption for OTEL Collector. 1. Add third party contributors as parameters in the OTEL Collector configuration. For a list of supported third party contributors, see the OTEL Collector Contrib Github Repository. The OTEL Collector Contrib Github Repository contains three main folders for handling telemetry data: - Third Party Receivers - Third Party Processors - Third Party Exporters The recommendations for configuring the contributors are as follows: - For receiver configuration, set the appropriate network parameters: ```yaml opentelemetry-collector: ports: syslog: protocol: TCP # or UDP ``` - For processor configuration, set the appropriate values for the following: - Log and trace formatting and enrichment. - Trace throttling, as traces are very verbose. - For exporter configuration, set the exporters to export the telemetry data from OTEL Collector to another OTEL Receiver. The OTEL SDK fields can be modified when exported to another format. Contributions from various telemetry solution vendors (such as Microsoft, AWS, Elasticsearch, and Prometheus) can be found in OTEL Collector, but not all vendors are present. Support may be found for the post popular telemetry solution vendors, but it is not guaranteed. If a contribution from a telemetry vendor is present, it might not include everything. For example, exporter contributions from Elasticsearch are only available for logs and traces. #### Configure OTEL Collector ##### Telemetry data reception By default, OTEL Collector listens to the **OTLP** ports (`4317` for `gRPC` and `4318` for `HTTP/HTTPS`), but not for the **rSyslog** protocol. The Ripple Custody Helm charts configure OTEL Collector to listen to port `54526` with the `TCP` network protocol for rSyslog telemetry data. ```yaml opentelemetry-collector: ports: syslog: enabled: true containerPort: 54526 servicePort: 54526 hostPort: 54526 protocol: TCP ``` ##### Host telemetry data By default, OTEL Collector only receives telemetry data sent by the telemetry network protocol (for example, `OTLP` or `rSyslog`) from the Ripple Custody components. OTEL Collector also offers the possibility to read and export host telemetry data (for example stdout logs and host metrics). Since OTEL Collector uses the Kubernetes API to access the host telemetry data, it needs the correct Role Based Access Control (RBAC) permissions. Appropriate RBAC permissions must be set in the Kubernetes cluster for OTEL Collector to be able to access the desired host telemetry data. For more information, see the OTEL Collector documentation. OTEL Collector log collection can be enabled by using the stdout of the Ripple Custody components: ```yaml opentelemetry-collector: enabled: true presets: logsCollection: enabled: true ``` More telemetry collection options are available by configuring the `presets` field: ```yaml opentelemetry-collector: enabled: true presets: logsCollection: enabled: true includeCollectorLogs: true storeCheckpoints: true hostMetrics: enabled: true kubernetesAttributes: enabled: true kubernetesEvents: enabled: true clusterMetrics: enabled: true kubeletMetrics: enabled: true ``` For more information, see OTEL Collector presets. ##### Pipeline configuration The default `pipelines` configuration must be updated for the environment: ```yaml opentelemetry-collector: config: service: pipelines: logs/syslog: receivers: [syslog] processors: [batch] exporters: [logging] logs: receivers: [otlp] processors: [batch] exporters: [logging] traces: receivers: [otlp] processors: [tail_sampling] exporters: [logging] metrics: receivers: [otlp] processors: [batch] exporters: [logging] ``` There are 4 pipelines: - logs/syslog - logs - traces - metrics Each pipeline is executed in 3 steps: Receivers —> Processors —> Exporters The OTEL Collector Contrib Github Repository provides an exhaustive list of all the Public implementation and documentation elements for OTEL Collector. For the full list of available third party contributions to OTEL Collector: - Receivers - Processors - Exporters ##### Processors In the context of Open Telemetry, a processor is a component that sits between the receiver and exporter in the data pipeline. Processors are responsible for modifying the telemetry data, such as traces, metrics, and logs that pass through them. Some common tasks a processor might handle include filtering, batching, and data enrichment. These processors play a crucial role in refining and preparing the telemetry data, making it ready for further analysis and insights. The Helm charts provide a default processor configuration named `tail_sampling` specifically for handling traces. ```yaml opentelemetry-collector: config: # OTEL Collector Processors processors: tail_sampling: decision_wait: 30s num_traces: 50000 expected_new_traces_per_sec: 50 policies: [...] ``` The `tail_sampling` processor has a number of policies defined for it. Each policy specifies the rules for selecting which traces should be kept for analysis. Some examples of these policies include capturing all domain-related traces, authenticated requests, and forced logs. Policies also exist to filter out traces based on certain criteria, such as specific HTTP targets, sources, payloads, latency, status codes, and more. The final policy, samples-some, is a probabilistic sampling policy that keeps a certain percentage of traces. This is often used to reduce the volume of traces, thus saving storage and processing resources while still providing a representative sample of the overall trace data. ##### Exporters Exporters are the components responsible for transmitting telemetry data from OTEL Collector to external systems or storage. An Exporter packages and sends the telemetry data to various destinations, such as monitoring platforms, logging systems, or analytics tools. Exporters come in different types and formats, depending on the target system or service. Here are some examples: ###### Elasticsearch ```yaml opentelemetry-collector: config: exporters: elasticsearch/trace: endpoints: ['http://my-elastic-search-host:9200'] traces_index: trace_index elasticsearch/log: endpoints: ['http://my-elastic-search-host:9200'] logs_index: log_index sending_queue: enabled: true num_consumers: 20 queue_size: 1000 elasticsearch/syslog: endpoints: ['http://my-elastic-search-host:9200'] logs_index: syslog_index sending_queue: enabled: true num_consumers: 20 queue_size: 1000 pipelines: logs/syslog: exporters: [elasticsearch/syslog] logs: exporters: [elasticsearch/log] traces: exporters: [elasticsearch/trace] ``` ###### Azure Monitor ```yaml opentelemetry-collector: config: exporters: azuremonitor: endpoint: 'https://my-azure-region.applicationinsights.azure.com/' instrumentation_key: 6ac20654-450e-29e4-65e2-1bdecb7db7c4 maxbatchsize: 1024 maxbatchinterval: 10s service: pipelines: logs/syslog: exporters: [azuremonitor] logs: exporters: [azuremonitor] traces: exporters: [azuremonitor] metrics: exporters: [azuremonitor] ```