Skip to content

Optimize

Losslessly compact events collected by Fluentd forwarders before they ship to output (e.g., ElasticSearch, S3). This module is a component of the Edge Optimizer app.

Architecture

graph LR
    A["<div style='font-size: 14px;'>📂 Fluentd</div><div style='font-size: 10px;'>sources</div>"] --> B["<div style='font-size: 14px;'>🔧 exec_filter</div><div style='font-size: 10px;'>plugin</div>"]
    B --> C["<div style='font-size: 14px;'>⚡ 10x Optimizer</div><div style='font-size: 10px;'>encode()</div>"]
    C --> B
    B --> D["<div style='font-size: 14px;'>📤 Fluentd</div><div style='font-size: 10px;'>outputs</div>"]

    classDef input fill:#2563eb,stroke:#1d4ed8,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef filter fill:#ea580c,stroke:#c2410c,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef engine fill:#7c3aed,stroke:#6d28d9,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef output fill:#16a34a,stroke:#15803d,color:#ffffff,stroke-width:2px,rx:8,ry:8

    class A input
    class B filter
    class C engine
    class D output

Data Flow

  • 📂 Fluentd Sources - Collect logs from files, TCP, HTTP, or other sources
  • 🔧 exec_filter Plugin - Pipes events to 10x sidecar via stdin
  • ⚡ 10x Optimizer - Losslessly compacts events to reduce log volume 50-80%
  • 🔄 Bidirectional Pipe - COMPACT events return via stdout to exec_filter
  • 📤 Fluentd Outputs - Compact events ship to final destinations at reduced size

Key Characteristics

Feature Description
📦 Lossless Compact Compacts events to reduce log volume 50-80%
🔗 Template Extraction Repetitive structures become reusable templates
💰 Cost Savings Reduced storage and transfer costs
🔧 exec_filter Uses Fluentd's native exec_filter for stdin/stdout piping

Sidecar Relay

This module configures a Fluentd exec-filter that launches a 10x sidecar process and passes it collected events to encode. The sidecar relays compact events back to the Fluentd filter to ship to outputs (e.g., Splunk, S3).

Install

See the Log10x Edge Optimizer Fluentd run instructions

Deploy to k8s via Helm

See the Log10x Edge Optimizer Fluentd deployment instructions

Configuration

To configure the Fluentd optimizer module, Edit these settings.

Below is the default configuration from: optimize/config.yaml.

Edit Online

Edit config.yaml Locally 

# 🔟❎ 'run' Fluentd optimizer configuration

# Configure an input that reads events from a Fluentd forwarder and
# writes optimized TenXObjects back to Fluentd to ship to output (e.g., ElasticSearch, Splunk).

# To learn more see https://doc.log10x.com/run/input/forwarder/fluentd/optimize/

# This config file runs in conjunction with the instructions provided in:
# https://doc.log10x.com/run/input/forwarder/fluentd/optimize/#install

# Set the 10x pipeline to 'run'
tenx: run

# =============================== Dependencies ================================

include:

  # define output encoding format options
  - run/modules/input/forwarder/fluentd/output/module.yaml

  # define the stdin input from which to read events the forwarder
  - run/modules/input/forwarder/fluentd/input

  # define Unix domain/stdout output to write events to the forwarder
  - '$="run/modules/input/forwarder/fluentd/output/" + (TenXString.includes(TenXEnv.get("os.name"), "Windows") ? "stdio" : "unix")'

# ============================== Fluentd Options ==============================

fluentd:
  # 'encodeObjects' is set to true to write TenXObjects to the forwarder using the encode() function.
  #  To learn more see https://doc.log10x.com/api/js/#TenXObject+encode
  encodeObjects: true

Below is the default configuration from: optimize/tenx-splunk.conf.

Edit Online

Edit tenx-splunk.conf Locally 

# Fluentd Splunk 10x Template output configuration

# This Fluent configuration routes 10x templates generated by the 'optimize' sub-process.
# Templates are written to Splunk via the HTTP Event Collector
# https://github.com/splunk/fluent-plugin-splunk-hec
# 
# 10x Templates are used by the 10x Splunk 'Optimize' app to inflate
# encoded events to their original form at search time. 
#  
#
<label @TENX-TEMPLATE>
    <match **>
        @type splunk_hec

        # 'hec_host', 'hec_port' and 'hec_token' are used to locate and
        #  authenticate against a target Splunk cluster
        #
        protocol https
        hec_host 127.0.0.1
        hec_token 00000000-0000-0000-0000-000000000000

        # The default Splunk HEC port is 8088, configurable in Splunk.
        # Make sure this matches your Splunk config.
        hec_port 8088

        # Overriding the default HEC config.
        #
        # While it is possible to set index/source/source type when setting explicitly
        # an HEC input in Splunk (https://docs.splunk.com/Documentation/Splunk/latest/Data/UsetheHTTPEventCollector)
        # They should also be expressly set here.
        #
        # Index should be set based on indexing policies.
        #
        index main

        # Source should be set to something unique to help you identify events.
        #
        source fluentd-tenx

        # Sourcetype must match the config of the Splunk 10x app and should most likely not be changed.
        #
        sourcetype tenx_dml_raw_json
    </match>
</label>

Below is the default configuration from: optimize/tenx-elastic.conf.

Edit Online

Edit tenx-elastic.conf Locally 

# Fluentd Elastic 10x Template output configuration

# This Fluent configuration routes 10x templates generated by the 'optimize' sub-process.
# Templates are written to Elastic Search via the 'elasticsearch' plugin
# https://docs.fluentd.org/output/elasticsearch
# 
# 10x Templates are used by the 10x Elastic Search 'Optimize' app to inflate
# encoded events to their original form at search-time. 
#  
<label @TENX-TEMPLATE>
    <match **>
        @type elasticsearch

        # Base host config, should be modified based on how you
        # connect to your ES cluster
        #
        host localhost
        port 9200

        # Index to store 10x templates.
        # It needs to match the configuration of the l1es plugin and should most likely not be changed.
        #
        index_name l1es_dml

        # Explicitly set the id key inside elastic to the value of templateHash.
        # This is important as the l1es plugin depends on being able to find 
        # templates based on their hash.
        #
        id_key templateHash

        <buffer>
            # Flush tenxTemplates to ensure the availability of encoded events within the search
            flush_interval 10s
        </buffer>
    </match>
</label>


This module is defined in optimize/module.yaml.