Skip to content

Optimize

Losslessly compact events collected by OpenTelemetry Collector before they ship to output (e.g., ElasticSearch, S3). This module is a component of the Edge Optimizer app.

Architecture

graph LR
    A["<div style='font-size: 14px;'>📂 OTel Receivers</div><div style='font-size: 10px;'>filelog, otlp</div>"] --> B["<div style='font-size: 14px;'>📤 syslog</div><div style='font-size: 10px;'>exporter</div>"]
    B --> C["<div style='font-size: 14px;'>🔌 Unix Socket</div><div style='font-size: 10px;'>RFC5424</div>"]
    C --> D["<div style='font-size: 14px;'>⚡ 10x Optimizer</div><div style='font-size: 10px;'>encode()</div>"]
    D --> E["<div style='font-size: 14px;'>🔌 Unix Socket</div><div style='font-size: 10px;'>Forward</div>"]
    E --> F["<div style='font-size: 14px;'>📥 fluentforward</div><div style='font-size: 10px;'>receiver</div>"]
    F --> G["<div style='font-size: 14px;'>📤 Final Exporters</div><div style='font-size: 10px;'>ES, Splunk, S3</div>"]

    classDef receiver fill:#2563eb,stroke:#1d4ed8,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef exporter fill:#ea580c,stroke:#c2410c,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef socket fill:#0891b2,stroke:#0e7490,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef engine fill:#7c3aed,stroke:#6d28d9,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef output fill:#16a34a,stroke:#15803d,color:#ffffff,stroke-width:2px,rx:8,ry:8

    class A receiver
    class B,F exporter
    class C,E socket
    class D engine
    class G output

Data Flow

  • 📂 OTel Receivers - Collect logs from files, OTLP, or other sources
  • 📤 Syslog Exporter - Sends logs to Log10x via Unix socket (RFC5424 format)
  • ⚡ 10x Optimizer - Losslessly compacts events to reduce log volume 50-80%
  • 🔌 Forward Output - Returns COMPACT events via Forward protocol
  • 📥 FluentForward Receiver - OTel Collector receives compact events
  • 📤 Final Exporters - Compact events ship to final destinations at reduced size

Key Characteristics

Feature Description
📦 Lossless Compact Compacts events to reduce log volume 50-80%
🔗 Template Extraction Repetitive structures become reusable templates
💰 Cost Savings Reduced storage and transfer costs
🔄 Two Pipelines logs/to-tenx sends original, logs/from-tenx receives compact

Sidecar Relay

This module configures a Unix socket input/output pair that receives syslog events from OpenTelemetry Collector, losslessly compacts events, and returns compact events via Forward protocol. The sidecar relays compact events back to OTel Collector to ship to outputs (e.g., Splunk, S3).

Install

See the Log10x Edge Optimizer OTel Collector run instructions

Deploy to k8s via Helm

See the Log10x Edge Optimizer OTel Collector deployment instructions

Configuration

To configure the OpenTelemetry Collector optimizer module, Edit these settings.

Below is the default configuration from: optimize/config.yaml.

Edit Online

Edit config.yaml Locally 

# 🔟❎ 'run' OpenTelemetry Collector optimizer configuration
#
# Configure an input that reads events from OpenTelemetry Collector via syslog
# and an output that writes encoded (optimized) TenXObjects back via Forward protocol.
#
# Architecture:
#   OTel Collector --[syslog/unix]--> Log10x Optimizer --[forward/unix]--> OTel Collector --> Final destinations
#
# Events are encoded using 10x-encoding to reduce log volume by 50-80%.
#
# Usage:
#   tenx run @run/input/forwarder/otel-collector/optimize @apps/edge/optimizer

# Set the 10x pipeline to 'run'
tenx: run

# =============================== Dependencies ================================

include:

  # Include output encoding format options
  - run/modules/input/forwarder/otel-collector/output/module.yaml

  # Read events from OTel Collector via Unix socket with syslog parsing
  - run/modules/input/forwarder/otel-collector/input

  # Write encoded events back to OTel Collector via Forward protocol
  - run/modules/input/forwarder/otel-collector/output/unix

# ========================= Configuration Options =============================

otelCollector:
  # Write TenXObjects using encode() function for compression
  encodeObjects: true


This module is defined in optimize/module.yaml.