Skip to content

Optimize

Losslessly compact log/trace events collected by Logstash forwarders before they ship to output (e.g., ElasticSearch, S3). This module is a component of the Edge Optimizer app.

Architecture

graph LR
    A["<div style='font-size: 14px;'>📂 Logstash</div><div style='font-size: 10px;'>inputs</div>"] --> B["<div style='font-size: 14px;'>🔧 Pipe Output</div><div style='font-size: 10px;'>plugin</div>"]
    B --> C["<div style='font-size: 14px;'>⚡ 10x Optimizer</div><div style='font-size: 10px;'>encode()</div>"]
    C --> D["<div style='font-size: 14px;'>🔌 Unix/TCP</div><div style='font-size: 10px;'>input</div>"]
    D --> E["<div style='font-size: 14px;'>📤 Logstash</div><div style='font-size: 10px;'>outputs</div>"]

    classDef input fill:#2563eb,stroke:#1d4ed8,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef filter fill:#ea580c,stroke:#c2410c,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef engine fill:#7c3aed,stroke:#6d28d9,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef socket fill:#0891b2,stroke:#0e7490,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef output fill:#16a34a,stroke:#15803d,color:#ffffff,stroke-width:2px,rx:8,ry:8

    class A input
    class B filter
    class C engine
    class D socket
    class E output

Data Flow

  • 📂 Logstash Inputs - Collect logs from files, beats, TCP, or other sources
  • 🔧 Pipe Output Plugin - Pipes events to 10x sidecar via stdin
  • ⚡ 10x Optimizer - Losslessly compacts events to reduce log volume 50-80%
  • 🔌 Unix/TCP Input - Receives COMPACT events back from the sidecar
  • 📤 Logstash Outputs - Compact events ship to final destinations at reduced size

Key Characteristics

Feature Description
📦 Lossless Compact Compacts events to reduce log volume 50-80%
🔗 Template Extraction Repetitive structures become reusable templates
💰 Cost Savings Reduced storage and transfer costs
🔧 Pipe Output Uses Logstash's pipe output plugin for stdin piping

Sidecar Relay

This module configures a Logstash pipe output plugin and Unix/TCP input plugin. The Logstash output plugin launches a 10x sidecar process and pipes events to it to encode. The sidecar relays compact events back to Logstash Unix/TCP input plugin to ship to output (e.g., ElasticSearch).

Install

See the Log10x Edge Optimizer Logstash run instructions

Currently not supported

Configuration

To configure the Logstash optimizer module, Edit these settings.

Below is the default configuration from: optimize/config.yaml.

Edit Online

Edit config.yaml Locally 

# 🔟❎ 'run' Logstash optimizer configuration

# Configure an input that reads events from a Logstash forwarder and
# writes optimized TenXObjects back to Logstash to ship to output (e.g., ElasticSearch, Splunk).

# To learn more see https://doc.log10x.com/run/input/forwarder/logstash/optimize/

# This file is operates in conjunction with the following Logstash '.conf' files:

# 1) tenx-pipe-out.conf: configures a Logstash plugin to launch a
#    10x sub-process and write incoming events to its stdin pipe.

# 2) tenx-pipe-in-<unix/tcp>.conf: configures a Logstash input to read
#    encoded events back via a unix or tcp socket (win):
#      unix: https://github.com/log-10x/config/blob/main/pipelines/run/input/forwarder/logstash/optimize/tenx-pipe-in-unix.conf
#      win : https://github.com/log-10x/config/blob/main/pipelines/run/input/forwarder/logstash/regulate/tenx-pipe-in-tcp.conf

# 3) tenx-elastic.conf

# Set the 10x pipeline to 'run'
tenx: run

# =============================== Dependencies ================================

include:

   # Include output encoding format options
  - run/modules/input/forwarder/logstash/output/module.yaml

    # Include the stdin input to read events from the forwarder
  - run/modules/input/forwarder/logstash/input

    # Include Unix domain/TCP socket output to write TenXObjects to the forwarder
  - '$="run/modules/input/forwarder/logstash/output/" + (TenXString.includes(TenXEnv.get("os.name"), "Windows") ? "tcp" : "unix")'

# ============================== Logstash Options =============================

logstash:
  # 'encodeObjects' is set to true to write TenXObjects using the encode() function
  #  To learn more see https://doc.log10x.com/api/js/#TenXObject+encode
  encodeObjects: true

Below is the default configuration from: optimize/tenx-pipe-in-tcp.conf.

Edit Online

Edit tenx-pipe-in-tcp.conf Locally 

# 10x Logstash TCP socket input configuration
#
# Add this Logstash configuration to the 'pipelines.yml' to launch a 10x process from which 
# to read events via a TCP socket.
#
# Add this file as a pipeline for your Logstash pipelines.yml config to incorporate
# it into your configuration.
#
# This file is used in conjunction with:
# run/config/forwarder/logstash/optimize/tenx-pipe-out.conf
#

# Read back data from 10x via a TCP socket. 
# Port 5160 is specified as a default, as Logstash does not define a default TCP port.
# To learn more see https://www.elastic.co/guide/en/logstash/current/plugins-inputs-tcp.html 
# The port value is captured by the 10x 'run' pipeline using the 'tenx_marker_pipeline_tcp_port' marker in '/run/modules/input/forwarder/logstash/output/tcp/stream.yaml'
#
input {
  tcp {
    id => "tenx_tcp_input"
    port => "${tenx_marker_pipeline_tcp_port:5160}"
    tags => ["tenx"]
    codec => json_lines
  }
}

# Process incoming events; templates get sent to Elastic, and everything else gets
# processed by the rest of the Logstash pipeline.
#
output {
  if [template] and [templateHash] {
    # Send 10x templates generated to your ElasticSearch cluster to allow
    # elasticsearch to run queries on 10x encoded indices.
    #
    elasticsearch {
      # Fill in your ElasticSearch cluster connection details here.
      #
      # For more info see:
      # https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-options
      #
      hosts => "localhost"

      # Explicitly set the id key inside elastic to the value of templateHash
      # This is important as the l1es plugin depends on being able to find explicitly
      # templates based on their hash.
      #
      document_id => "%{[templateHash]}"

      # Index to store 10x templates.
      # It needs to match the configuration of the l1es plugin and should most likely not be changed.
      #
      index => l1es_dml
    }
  } else {
    # Send all other data received from the 10x executable to the next logstash
    # pipeline for further processing, depending on your configuration.
    #
    # Alternatively, any other Logstash output plugin may be used when incorporating
    # 10x into your Logstash config
    #
    pipeline {
      send_to => user_output_pipeline
    }
  }
}

Below is the default configuration from: optimize/tenx-pipe-in-unix.conf.

Edit Online

Edit tenx-pipe-in-unix.conf Locally 

# 10x Logstash Unix domain socket input configuration
#
# Add this Logstash configuration to 'pipelines.yml' to launch
# a 10x subprocess from which to read events via a Unix domain socket.
#
# Add this file as a pipeline for your Logstash pipelines.yml config to incorporate
# it into your configuration.
#
# This file is used in conjunction with:
# run/config/forwarder/logstash/optimize/tenx-pipe-out.conf
#

# Read back data from 10x via a Unix domain socket.
# The 10x 'run' pipeline captures the socket address using 'tenx_marker_pipeline_unix_address' in '/run/modules/input/forwarder/logstash/output/unix/stream.yaml'
#
input {
  unix {
    id => "tenx_unix_input"
    path => "${tenx_marker_pipeline_unix_address:/tmp/tenx_logstash.sock}"
    force_unlink => true
    tags => ["tenx"]
    codec => json_lines
  }
}

# Process incoming events; templates get sent to Elastic, and everything else gets
# processed by the rest of the Logstash pipeline.
#
output {
  if [template] and [templateHash] {
    # Send 10x templates generated to your ElasticSearch cluster to allow
    # elasticsearch to run queries on 10x encoded indices.
    #
    elasticsearch {
      # Fill in your ElasticSearch cluster connection details here.
      #
      # For more info see:
      # https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-options
      #
      hosts => "localhost"

      # Explicitly set the id key inside elastic to the value of templateHash
      # This is important as the l1es plugin depends on being able to find explicitly
      # templates based on their hash.
      #
      document_id => "%{[templateHash]}"

      # Index to store 10x templates.
      # It needs to match the configuration of the l1es plugin and should most likely not be changed.
      #
      index => l1es_dml
    }
  } else {
    # Send all other data received from the 10x executable to the next logstash
    # pipeline for further processing, depending on your configuration.
    #
    # Alternatively, any other Logstash output plugin may be used when directly incorporating
    # 10x into your Logstash config
    #
    pipeline {
      send_to => user_output_pipeline
    }
  }
}

Below is the default configuration from: optimize/tenx-pipe-out.conf.

Edit Online

Edit tenx-pipe-out.conf Locally 

# 10x Logstash pipe output configuration
#
# Add this Logstash configuration to 'pipelines.yml' to launch
# an 10x subprocess into which to write events to optimize.
#
# This file is used in conjunction with a Logstash input '.conf' file
# used to read optimized events from the 10x subprocess:
#   unix: run/config/forwarder/logstash/optimize/tenx-pipe-in-unix.conf
#   win:  run/config/forwarder/logstash/optimize/tenx-pipe-in-tcp.conf


input {
  # Getting data into this Logstash pipeline.
  # When using the pipelines.yml approach, this sets Logstash's internal address
  # for data input into this pipeline.
  #
  # Alternatively, any other Logstash input plugin may be used when incorporating
  # 10x into your Logstash config
  #
  pipeline {
    address => tenx_input_pipeline
  }
}

output {
  # Send incoming events to the 10x executable via the 'pipe' output plugin.
  #
  pipe {
    id => "tenx_pipeline"

    # Don't close the 10x process due to events inactivity.
    ttl => -1

    # NIX
    command => "${TENX_BIN:/opt/tenx-edge/bin/tenx} run @run/input/forwarder/logstash/optimize/config.yaml @run/apps/edge/optimizer"

    # WIN
    #command => "${TENX_BIN:'c:\program files\tenx\tenx.exe'} run @run/input/forwarder/logstash/optimize/config.yaml @run/apps/edge/optimizer"
  }
}


This module is defined in optimize/module.yaml.