OTEL Collector deployment

This page covers deploying and configuring the OpenTelemetry (OTEL) Collector for Ripple Custody.

The Ripple Custody Helm charts include an OTEL Collector instance:

opentelemetry-collector:
  enabled: false        # Set to `false` by default

You must deploy OTEL Collector in line with your own requirements.

For more information on OTEL Collector deployment, see the following:

• Kubernetes OpenTelemetry documentation
• The official OTEL Collector Helm charts

For detailed OTEL Collector Values File configuration, see the Official Helm Chart for OTEL Collector.

You can set up OTEL Collector using two different configurations:

1. A dedicated OTEL Collector instance for both the trusted and untrusted components.
2. Two dedicated and separate instances (one instance for the trusted components and one instance for the untrusted components).

For more information on trusted and untrusted components, see Security.

To set up an OTEL Collector instance:

1. Set the enabled variable to true:

opentelemetry-collector:
  enabled: true

This automatically deploys an OTEL Collector pod alongside the Ripple Custody deployment and sets the appropriate environment variables for the pods to discover the OTEL Collector service.

2. The default resource values might be too low for a typical deployment. Set the resource value to an appropriate level, as follows:

   1. Run load testing on the Ripple Custody instance, at the peak usage you expect the Ripple Custody instance to encounter and observe resource usage.

   2. Set the following values accordingly:

        resources:
          requests:
            cpu: 4
            memory: 4Gi
          limits:
            cpu: 8
            memory: 16Gi

3. Add third-party contributors as parameters in the OTEL Collector configuration. For a list of supported third-party contributors, see the OTEL Collector Contrib Github Repository. The OTEL Collector Contrib Github Repository contains three main folders for handling telemetry data:

   • Third Party Receivers
   • Third Party Processors
   • Third Party Exporters

   The recommendations for configuring the contributors are as follows:

   • For receiver configuration, set the appropriate network parameters:
     opentelemetry-collector:
       ports:
         syslog:
           protocol: TCP     # or UDP

   • For processor configuration, set the appropriate values for the following:

     • Log and trace formatting and enrichment.
     • Trace throttling, as traces are very verbose.

   • For exporter configuration, set the exporters to export the telemetry data from OTEL Collector to another OTEL Receiver. The OTEL SDK fields can be modified when exported to another format.

By default, OTEL Collector listens to the OTLP ports (4317 for gRPC and 4318 for HTTP/HTTPS), but not for the rSyslog protocol.

The Ripple Custody Helm charts configure OTEL Collector to listen to port 54526 with the TCP network protocol for rSyslog telemetry data.

opentelemetry-collector:
  ports:
    syslog:
      enabled: true
      containerPort: 54526
      servicePort: 54526
      hostPort: 54526
      protocol: TCP

By default, OTEL Collector only receives telemetry data sent by the telemetry network protocol (for example, OTLP or rSyslog) from the Ripple Custody components.

OTEL Collector also offers the possibility to read and export host telemetry data (for example stdout logs and host metrics).

You can enable OTEL Collector log collection by using the stdout of the Ripple Custody components:

opentelemetry-collector:
  enabled: true

  presets:
    logsCollection:
      enabled: true

More telemetry collection options are available by configuring the presets field:

opentelemetry-collector:
  enabled: true

  presets:
    logsCollection:
      enabled: true
      includeCollectorLogs: true
      storeCheckpoints: true
    hostMetrics:
      enabled: true
    kubernetesAttributes:
      enabled: true
    kubernetesEvents:
      enabled: true
    clusterMetrics:
      enabled: true
    kubeletMetrics:
      enabled: true

For more information, see OTEL Collector presets.

You must update the default pipelines configuration for the environment:

opentelemetry-collector:
  config:

    service:
      pipelines:
        logs/syslog:
          receivers: [syslog]
          processors: [batch]
          exporters: [logging]
        logs:
          receivers: [otlp]
          processors: [batch]
          exporters: [logging]
        traces:
          receivers: [otlp]
          processors: [tail_sampling]
          exporters: [logging]
        metrics:
          receivers: [otlp]
          processors: [batch]
          exporters: [logging]

There are 4 pipelines:

• logs/syslog
• logs
• traces
• metrics

OTEL Collector executes each pipeline in 3 steps:

Receivers —> Processors —> Exporters

The OTEL Collector Contrib Github Repository provides an exhaustive list of all the Public implementation and documentation elements for OTEL Collector.

For the full list of available third party contributions to OTEL Collector:

• Receivers
• Processors
• Exporters

In the context of OpenTelemetry, a processor is a component that sits between the receiver and exporter in the data pipeline. Processors are responsible for modifying the telemetry data, such as traces, metrics, and logs that pass through them. Some common tasks a processor might handle include filtering, batching, and data enrichment.

These processors play a crucial role in refining and preparing the telemetry data, making it ready for further analysis and insights.

The Helm charts provide a default processor configuration named tail_sampling specifically for handling traces.

opentelemetry-collector:
  config:

    # OTEL Collector Processors
    processors:
      tail_sampling:
        decision_wait: 30s
        num_traces: 50000
        expected_new_traces_per_sec: 50
        policies: [...]

The tail_sampling processor has a number of policies defined for it.
Each policy specifies the rules for selecting which traces should be kept for analysis.
Some examples of these policies include capturing all domain-related traces, authenticated requests, and forced logs.
Policies also exist to filter out traces based on certain criteria, such as specific HTTP targets, sources, payloads, latency, status codes, and more.

The final policy, samples-some, is a probabilistic sampling policy that keeps a certain percentage of traces.
Operators often use this approach to reduce the volume of traces, thus saving storage and processing resources while still providing a representative sample of the overall trace data.

Exporters are the components responsible for transmitting telemetry data from OTEL Collector to external systems or storage. An Exporter packages and sends the telemetry data to various destinations, such as monitoring platforms, logging systems, or analytics tools.

Exporters come in different types and formats, depending on the target system or service. Here are some examples:

opentelemetry-collector:
  config:

    exporters:

      elasticsearch/trace:
        endpoints: ['http://my-elastic-search-host:9200']
        traces_index: trace_index
      elasticsearch/log:
        endpoints: ['http://my-elastic-search-host:9200']
        logs_index: log_index
        sending_queue:
          enabled: true
          num_consumers: 20
          queue_size: 1000
      elasticsearch/syslog:
        endpoints: ['http://my-elastic-search-host:9200']
        logs_index: syslog_index
        sending_queue:
          enabled: true
          num_consumers: 20
          queue_size: 1000

      pipelines:
        logs/syslog:
          exporters: [elasticsearch/syslog]
        logs:
          exporters: [elasticsearch/log]
        traces:
          exporters: [elasticsearch/trace]

opentelemetry-collector:
  config:

    exporters:
      azuremonitor:
        endpoint: 'https://my-azure-region.applicationinsights.azure.com/'
        instrumentation_key: 6ac20654-450e-29e4-65e2-1bdecb7db7c4
        maxbatchsize: 1024
        maxbatchinterval: 10s

    service:
      pipelines:
        logs/syslog:
          exporters: [azuremonitor]
        logs:
          exporters: [azuremonitor]
        traces:
          exporters: [azuremonitor]
        metrics:
          exporters: [azuremonitor]