# OTEL Collector deployment This page covers deploying and configuring the [OpenTelemetry (OTEL) Collector](https://opentelemetry.io/docs/collector/) for Ripple Custody. For an overview of telemetry concepts, see [Configure telemetry](/products/custody/v1.34/deployment/reference/telemetry). For Helm chart configuration, see [Telemetry configuration](/products/custody/v1.34/deployment/reference/telemetry-configuration). The Ripple Custody Helm charts include an OTEL Collector instance: ```yaml opentelemetry-collector: enabled: false # Set to `false` by default ``` Use this instance only for testing purposes. Do not use it for production workloads. We recommend you set up a dedicated and segregated OTEL Collector instance. You must deploy OTEL Collector in line with your own requirements. For more information on OTEL Collector deployment, see the following: - Kubernetes OpenTelemetry documentation - The official OTEL Collector Helm charts The recommended best practice for production workload is to set up a dedicated OTEL Collector deployment that you segregate from Ripple Custody. All the YAML code snippets in the steps below use the `opentelemetry-collector` section, as a reminder that this is the OTEL Collector Helm `values.yaml` file configuration and not the Ripple Custody Helm configuration. For detailed OTEL Collector Values File configuration, see the Official Helm Chart for OTEL Collector. ## Deploy OTEL Collector You can set up OTEL Collector using two different configurations: 1. A dedicated OTEL Collector instance for both the trusted and untrusted components. 2. Two dedicated and separate instances (one instance for the trusted components and one instance for the untrusted components). For more information on trusted and untrusted components, see [Security](/products/custody/v1.34/concepts/security). To set up an OTEL Collector instance: 1. Set the `enabled` variable to `true`: ```yaml opentelemetry-collector: enabled: true ``` This automatically deploys an OTEL Collector pod alongside the Ripple Custody deployment and sets the appropriate environment variables for the pods to discover the OTEL Collector service. 1. The default resource values might be too low for a typical deployment. Set the resource value to an appropriate level, as follows: 1. Run load testing on the Ripple Custody instance, at the peak usage you expect the Ripple Custody instance to encounter and observe resource usage. 2. Set the following values accordingly: ```yaml resources: requests: cpu: 4 memory: 4Gi limits: cpu: 8 memory: 16Gi ``` The Helm `values.yaml` file settings must reflect your usage of Ripple Custody. Those values need to be high enough that the **opentelemetry-collector** pod does not encounter a `Pending` or `CrashLoopBackOff` Kubernetes state. The higher the number of users, the greater the volume of telemetry data generated, and the higher the resource consumption for OTEL Collector. 1. Add third-party contributors as parameters in the OTEL Collector configuration. For a list of supported third-party contributors, see the OTEL Collector Contrib Github Repository. The OTEL Collector Contrib Github Repository contains three main folders for handling telemetry data: - Third Party Receivers - Third Party Processors - Third Party Exporters The recommendations for configuring the contributors are as follows: - For receiver configuration, set the appropriate network parameters: ```yaml opentelemetry-collector: ports: syslog: protocol: TCP # or UDP ``` - For processor configuration, set the appropriate values for the following: - Log and trace formatting and enrichment. - Trace throttling, as traces are very verbose. - For exporter configuration, set the exporters to export the telemetry data from OTEL Collector to another OTEL Receiver. The OTEL SDK fields can be modified when exported to another format. Contributions from various telemetry solution vendors (such as Microsoft, AWS, Elasticsearch, and Prometheus) can be found in OTEL Collector, but not all vendors are present. You may find support for the most popular telemetry solution vendors, but we do not guarantee it. If a contribution from a telemetry vendor is present, it might not include everything. For example, exporter contributions from Elasticsearch are only available for logs and traces. ## Configure OTEL Collector ### Telemetry data reception By default, OTEL Collector listens to the **OTLP** ports (`4317` for `gRPC` and `4318` for `HTTP/HTTPS`), but not for the **rSyslog** protocol. The Ripple Custody Helm charts configure OTEL Collector to listen to port `54526` with the `TCP` network protocol for rSyslog telemetry data. ```yaml opentelemetry-collector: ports: syslog: enabled: true containerPort: 54526 servicePort: 54526 hostPort: 54526 protocol: TCP ``` ##### Host telemetry data By default, OTEL Collector only receives telemetry data sent by the telemetry network protocol (for example, `OTLP` or `rSyslog`) from the Ripple Custody components. OTEL Collector also offers the possibility to read and export host telemetry data (for example stdout logs and host metrics). Since OTEL Collector uses the Kubernetes API to access the host telemetry data, it needs the correct Role Based Access Control (RBAC) permissions. You must set appropriate RBAC permissions in the Kubernetes cluster for OTEL Collector to access the desired host telemetry data. For more information, see the OTEL Collector documentation. You can enable OTEL Collector log collection by using the stdout of the Ripple Custody components: ```yaml opentelemetry-collector: enabled: true presets: logsCollection: enabled: true ``` More telemetry collection options are available by configuring the `presets` field: ```yaml opentelemetry-collector: enabled: true presets: logsCollection: enabled: true includeCollectorLogs: true storeCheckpoints: true hostMetrics: enabled: true kubernetesAttributes: enabled: true kubernetesEvents: enabled: true clusterMetrics: enabled: true kubeletMetrics: enabled: true ``` For more information, see OTEL Collector presets. ##### Pipeline configuration You must update the default `pipelines` configuration for the environment: ```yaml opentelemetry-collector: config: service: pipelines: logs/syslog: receivers: [syslog] processors: [batch] exporters: [logging] logs: receivers: [otlp] processors: [batch] exporters: [logging] traces: receivers: [otlp] processors: [tail_sampling] exporters: [logging] metrics: receivers: [otlp] processors: [batch] exporters: [logging] ``` There are 4 pipelines: - logs/syslog - logs - traces - metrics OTEL Collector executes each pipeline in 3 steps: Receivers —> Processors —> Exporters The OTEL Collector Contrib Github Repository provides an exhaustive list of all the Public implementation and documentation elements for OTEL Collector. For the full list of available third party contributions to OTEL Collector: - Receivers - Processors - Exporters ##### Processors In the context of OpenTelemetry, a processor is a component that sits between the receiver and exporter in the data pipeline. Processors are responsible for modifying the telemetry data, such as traces, metrics, and logs that pass through them. Some common tasks a processor might handle include filtering, batching, and data enrichment. These processors play a crucial role in refining and preparing the telemetry data, making it ready for further analysis and insights. The Helm charts provide a default processor configuration named `tail_sampling` specifically for handling traces. ```yaml opentelemetry-collector: config: # OTEL Collector Processors processors: tail_sampling: decision_wait: 30s num_traces: 50000 expected_new_traces_per_sec: 50 policies: [...] ``` The `tail_sampling` processor has a number of policies defined for it. Each policy specifies the rules for selecting which traces should be kept for analysis. Some examples of these policies include capturing all domain-related traces, authenticated requests, and forced logs. Policies also exist to filter out traces based on certain criteria, such as specific HTTP targets, sources, payloads, latency, status codes, and more. The final policy, samples-some, is a probabilistic sampling policy that keeps a certain percentage of traces. Operators often use this approach to reduce the volume of traces, thus saving storage and processing resources while still providing a representative sample of the overall trace data. ##### Exporters Exporters are the components responsible for transmitting telemetry data from OTEL Collector to external systems or storage. An Exporter packages and sends the telemetry data to various destinations, such as monitoring platforms, logging systems, or analytics tools. Exporters come in different types and formats, depending on the target system or service. Here are some examples: ###### Elasticsearch ```yaml opentelemetry-collector: config: exporters: elasticsearch/trace: endpoints: ['http://my-elastic-search-host:9200'] traces_index: trace_index elasticsearch/log: endpoints: ['http://my-elastic-search-host:9200'] logs_index: log_index sending_queue: enabled: true num_consumers: 20 queue_size: 1000 elasticsearch/syslog: endpoints: ['http://my-elastic-search-host:9200'] logs_index: syslog_index sending_queue: enabled: true num_consumers: 20 queue_size: 1000 pipelines: logs/syslog: exporters: [elasticsearch/syslog] logs: exporters: [elasticsearch/log] traces: exporters: [elasticsearch/trace] ``` ###### Azure Monitor ```yaml opentelemetry-collector: config: exporters: azuremonitor: endpoint: 'https://my-azure-region.applicationinsights.azure.com/' instrumentation_key: 6ac20654-450e-29e4-65e2-1bdecb7db7c4 maxbatchsize: 1024 maxbatchinterval: 10s service: pipelines: logs/syslog: exporters: [azuremonitor] logs: exporters: [azuremonitor] traces: exporters: [azuremonitor] metrics: exporters: [azuremonitor] ```