Optimize

Losslessly compact events collected by Fluentd forwarders before they ship to output (e.g., ElasticSearch, S3). This module is a component of the Edge Optimizer app.

Architecture

graph LR
    A["<div style='font-size: 14px;'>📂 Fluentd</div><div style='font-size: 10px;'>sources</div>"] --> B["<div style='font-size: 14px;'>🔧 exec_filter</div><div style='font-size: 10px;'>plugin</div>"]
    B --> C["<div style='font-size: 14px;'>⚡ 10x Optimizer</div><div style='font-size: 10px;'>encode()</div>"]
    C --> B
    B --> D["<div style='font-size: 14px;'>📤 Fluentd</div><div style='font-size: 10px;'>outputs</div>"]

    classDef input fill:#2563eb,stroke:#1d4ed8,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef filter fill:#ea580c,stroke:#c2410c,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef engine fill:#7c3aed,stroke:#6d28d9,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef output fill:#16a34a,stroke:#15803d,color:#ffffff,stroke-width:2px,rx:8,ry:8

    class A input
    class B filter
    class C engine
    class D output

Data Flow

📂 Fluentd Sources - Collect logs from files, TCP, HTTP, or other sources
🔧 exec_filter Plugin - Pipes events to 10x sidecar via stdin
⚡ 10x Optimizer - Losslessly compacts events to reduce log volume 50-80%
🔄 Bidirectional Pipe - COMPACT events return via stdout to exec_filter
📤 Fluentd Outputs - Compact events ship to final destinations at reduced size

Key Characteristics

Feature	Description
📦 Lossless Compact	Compacts events to reduce log volume 50-80%
🔗 Template Extraction	Repetitive structures become reusable templates
💰 Cost Savings	Reduced storage and transfer costs
🔧 exec_filter	Uses Fluentd's native exec_filter for stdin/stdout piping

Sidecar Relay

This module configures a Fluentd exec-filter that launches a 10x sidecar process and passes it collected events to encode. The sidecar relays compact events back to the Fluentd filter to ship to outputs (e.g., Splunk, S3).

Install

Nix/Win/OSX k8s

See the Log10x Edge Optimizer Fluentd run instructions

Deploy to k8s via Helm

See the Log10x Edge Optimizer Fluentd deployment instructions

Config Files

To configure the Fluentd optimizer module, Edit these files.

ConfigTenx-splunk.confTenx-elastic.conf

Below is the default configuration from: optimize/config.yaml.

Edit Online

# 🔟❎ 'run' Fluentd optimizer configuration

# Configure an input that reads events from a Fluentd forwarder and
# writes optimized TenXObjects back to Fluentd to ship to output (e.g., ElasticSearch, Splunk).

# To learn more see https://doc.log10x.com/run/input/forwarder/fluentd/optimize/

# This config file runs in conjunction with the instructions provided in:
# https://doc.log10x.com/run/input/forwarder/fluentd/optimize/#install

# Set the 10x pipeline to 'run'
tenx: run

# =============================== Dependencies ================================

include:

  # Include optimizer output options (encodeObjects, outputFields, outputEncodeType)
  - run/modules/input/forwarder/fluentd/optimize/module.yaml

  # define the stdin input from which to read events the forwarder
  - run/modules/input/forwarder/fluentd/input

  # define Unix domain/stdout output to write events to the forwarder
  - '$="run/modules/input/forwarder/fluentd/output/" + (TenXString.includes(TenXEnv.get("os.name"), "Windows") ? "stdio" : "unix")'

# ============================== Fluentd Options ==============================

fluentd:
  # 'encodeObjects' is set to true to write TenXObjects to the forwarder using the encode() function.
  #  To learn more see https://doc.log10x.com/api/js/#TenXObject+encode
  encodeObjects: true

  # 'outputFields' specifies TenXObject field names to emit alongside the encoded event.
  #  When set, enrichment fields are included in the sidecar output for forwarder post-processing.
  #  Example: outputFields: [level, group, symbolMessage]
  outputFields: []

  # 'outputEncodeType' controls the output format when outputFields is set.
  #  Possible values: 'json' or 'delimited'
  outputEncodeType: delimited

Below is the default configuration from: optimize/tenx-splunk.conf.

Edit Online

# Fluentd Splunk 10x Template output configuration

# This Fluent configuration routes 10x templates generated by the 'optimize' sub-process.
# Templates are written to Splunk via the HTTP Event Collector
# https://github.com/splunk/fluent-plugin-splunk-hec
# 
# 10x Templates are used by the 10x Splunk 'Optimize' app to inflate
# encoded events to their original form at search time. 
#  
#
<label @TENX-TEMPLATE>
    <match **>
        @type splunk_hec

        # 'hec_host', 'hec_port' and 'hec_token' are used to locate and
        #  authenticate against a target Splunk cluster
        #
        protocol https
        hec_host 127.0.0.1
        hec_token 00000000-0000-0000-0000-000000000000

        # The default Splunk HEC port is 8088, configurable in Splunk.
        # Make sure this matches your Splunk config.
        hec_port 8088

        # Overriding the default HEC config.
        #
        # While it is possible to set index/source/source type when setting explicitly
        # an HEC input in Splunk (https://docs.splunk.com/Documentation/Splunk/latest/Data/UsetheHTTPEventCollector)
        # They should also be expressly set here.
        #
        # Index should be set based on indexing policies.
        #
        index main

        # Source should be set to something unique to help you identify events.
        #
        source fluentd-tenx

        # Sourcetype must match the config of the Splunk 10x app and should most likely not be changed.
        #
        sourcetype tenx_dml_raw_json
    </match>
</label>

Below is the default configuration from: optimize/tenx-elastic.conf.

Edit Online

# Fluentd Elastic 10x Template output configuration

# This Fluent configuration routes 10x templates generated by the 'optimize' sub-process.
# Templates are written to Elastic Search via the 'elasticsearch' plugin
# https://docs.fluentd.org/output/elasticsearch
# 
# 10x Templates are used by the 10x Elastic Search 'Optimize' app to inflate
# encoded events to their original form at search-time. 
#  
<label @TENX-TEMPLATE>
    <match **>
        @type elasticsearch

        # Base host config, should be modified based on how you
        # connect to your ES cluster
        #
        host localhost
        port 9200

        # Index to store 10x templates.
        # It needs to match the configuration of the l1es plugin and should most likely not be changed.
        #
        index_name l1es_dml

        # Explicitly set the id key inside elastic to the value of templateHash.
        # This is important as the l1es plugin depends on being able to find 
        # templates based on their hash.
        #
        id_key templateHash

        <buffer>
            # Flush tenxTemplates to ensure the availability of encoded events within the search
            flush_interval 10s
        </buffer>
    </match>
</label>

Options

Specify the options below to configure the Fluentd optimizer:

Name	Description
fluentdEncodeObjects	Controls whether events written to a Fluentd forwarder are encoded or written in raw form
fluentdOutputFields	List of TenXObject field names to include alongside the main event in output
fluentdOutputEncodeType	Output format when outputFields are set. Possible values: [json, delimited]

`fluentdEncodeObjects`

Controls whether events written to a Fluentd forwarder are encoded or written in raw form.

Type	Default
Boolean	true

Sets the format in which to serialize TenXObjects back to the Fluentd forwarder. If true, write events using the encode function. If false, write events in their original form as read from the Fluentd forwarder using the fullText field.

`fluentdOutputFields`

List of TenXObject field names to include alongside the main event in output.

Type	Default
List	[]

Specifies TenXObject field names to emit alongside the main event text when writing back to the Fluentd forwarder. When empty (default), only the main event field is written. Example fields: level, group, symbolMessage.

`fluentdOutputEncodeType`

Output format when outputFields are set. Possible values: [json, delimited].

Type	Default
String	delimited

Specifies how the combined output (main event field plus outputFields) is encoded when writing back to the Fluentd forwarder. Possible values:

json: formats all fields as a JSON object
delimited: formats field values separated by the output delimiter Only takes effect when fluentdOutputFields is set.

This module is defined in optimize/module.yaml.