Optimize

Losslessly compact log/trace events collected by Fluent Bit forwarders before they ship to output (e.g., Splunk, S3). This module is a component of the Edge Optimizer app.

Architecture

graph LR
    A["<div style='font-size: 14px;'>📂 Fluent Bit</div><div style='font-size: 10px;'>inputs</div>"] --> B["<div style='font-size: 14px;'>🔧 Lua Filter</div><div style='font-size: 10px;'>tenx.lua</div>"]
    B --> C["<div style='font-size: 14px;'>⚡ 10x Optimizer</div><div style='font-size: 10px;'>encode()</div>"]
    C --> D["<div style='font-size: 14px;'>🔌 Unix/TCP</div><div style='font-size: 10px;'>input</div>"]
    D --> E["<div style='font-size: 14px;'>📤 Fluent Bit</div><div style='font-size: 10px;'>outputs</div>"]

    classDef input fill:#2563eb,stroke:#1d4ed8,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef filter fill:#ea580c,stroke:#c2410c,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef engine fill:#7c3aed,stroke:#6d28d9,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef socket fill:#0891b2,stroke:#0e7490,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef output fill:#16a34a,stroke:#15803d,color:#ffffff,stroke-width:2px,rx:8,ry:8

    class A input
    class B filter
    class C engine
    class D socket
    class E output

Data Flow

• 📂 Fluent Bit Inputs - Collect logs from files, containers, or other sources
• 🔧 Lua Filter - Intercepts events and pipes them to 10x sidecar
• ⚡ 10x Optimizer - Losslessly compacts events to reduce log volume 50-80%
• 🔌 Unix/TCP Input - Receives COMPACT events back from the sidecar
• 📤 Fluent Bit Outputs - Compact events ship to final destinations at reduced size

Key Characteristics

Feature	Description
📦 Lossless Compact	Compacts events to reduce log volume 50-80%
🔗 Template Extraction	Repetitive structures become reusable templates
💰 Cost Savings	Reduced storage and transfer costs
🔧 Lua Filter	Uses Fluent Bit's native Lua scripting for sidecar launch

Sidecar Relay

This module configures a Fluent Bit Lua filter and Unix/TCP input. The Lua filter launches a 10x sidecar process and passes it collected events to encode. The sidecar relays compact events back to Fluent Bit via the configured Unix/TCP input to ship to outputs (e.g., Splunk, S3).

Install

Nix/Win/OSX k8s

See the Log10x Edge Optimizer Fluent Bit run instructions

Deploy to k8s via Helm

See the Log10x Edge Optimizer Fluent Bit deployment instructions

Config Files

To configure the Fluent Bit optimizer module, Edit these files.

ConfigTenx-splunk.confTenx-elastic.conf

Below is the default configuration from: optimize/config.yaml.

Edit Online Edit Locally

Edit config.yaml Locally

Download and save to:

• Linux / Docker / macOS
• Custom directory
  $TENX_CONFIG/run/input/forwarder/fluentbit/optimize/config.yaml
• Within cloned repo
  ./pipelines/run/input/forwarder/fluentbit/optimize/config.yaml

Download

# 🔟❎ 'run' Fluent Bit optimizer configuration

# Configure an input that reads events from a Fluent Bit forwarder and
# writes optimized TenXObjects back to Fluent Bit to ship to output (e.g., ElasticSearch, Splunk).
# To learn more see https://doc.log10x.com/run/input/forwarder/fluentbit/optimize/

# This config runs in conjunction with the instructions provided in:
# https://doc.log10x.com/run/input/forwarder/fluentbit/optimize/#install

# Set the 10x pipeline to 'run'
tenx: run

# =============================== Dependencies ================================

include:

  # Include optimizer output options (encodeObjects, outputFields, outputEncodeType)
  - run/modules/input/forwarder/fluentbit/optimize/module.yaml

  # Include the stdin input from which to read events from the forwarder
  - run/modules/input/forwarder/fluentbit/input

  # Include Unix domain/TCP socket output to which events into the forwarder
  - '$="run/modules/input/forwarder/fluentbit/output/" + (TenXString.includes(TenXEnv.get("os.name"), "Windows") ? "tcp" : "unix")'

# ============================= Fluentbit Options =============================

fluentbit:
  # 'encodeObjects' is set to true to write TenXObjects
  #  back to the forwarder using the encode() function.
  #  To learn more see https://doc.log10x.com/api/js/#TenXObject+encode
  encodeObjects: true

  # 'outputFields' specifies TenXObject field names to emit alongside the encoded event.
  #  When set, enrichment fields are included in the sidecar output for forwarder post-processing.
  #  Example: outputFields: [level, group, symbolMessage]
  outputFields: []

  # 'outputEncodeType' controls the output format when outputFields is set.
  #  Possible values: 'json' or 'delimited'
  outputEncodeType: delimited

Below is the default configuration from: optimize/tenx-splunk.conf.

Edit Online Edit Locally

Edit tenx-splunk.conf Locally

Download and save to:

• Linux / Docker / macOS
• Custom directory
  $TENX_CONFIG/run/input/forwarder/fluentbit/optimize/tenx-splunk.conf
• Within cloned repo
  ./pipelines/run/input/forwarder/fluentbit/optimize/tenx-splunk.conf

Download

# Sample Fluent-Bit Splunk 10x output configuration
#
# This Fluent-Bit configuration routes 10x templates and encoded events to Splunk
# via the HTTP Event Collector (HEC).
# https://docs.fluentbit.io/manual/pipeline/outputs/splunk
#
# 10x Templates are used by the 10x Splunk 'Optimize' app to inflate
# encoded events to their original form at search time.
#
# IMPORTANT: You must create HEC tokens in Splunk and replace <your-hec-token> below.
# See: https://docs.splunk.com/Documentation/Splunk/latest/Data/UsetheHTTPEventCollector
#
# Index should be set based on indexing policies.
#
# Source should be set to something unique to help you identify events.
#
# Sourcetype must match the config of the Splunk 10x app and should most likely not be changed.
#
# ========================= ENCODED EVENTS OUTPUT =========================
# Use this output for encoded log events (match all except templates).
# IMPORTANT: Use 'event_key $log' to send only the log content without JSON wrapping.
# Without event_key, Splunk receives {"log":"~hash,var1,var2..."} instead of just the encoded data.
#
# [OUTPUT]
#     Name              splunk
#     Match_Regex       ^(?!tenx-template).*
#     Host              127.0.0.1
#     Port              8088
#     Splunk_Token      <your-encoded-events-hec-token>
#     event_index       tenx_encoded
#     event_sourcetype  tenx_encoded
#     event_key         $log
#     TLS               On
#     TLS.Verify        On
#
# ========================= TEMPLATES OUTPUT ==============================
# Use this output for 10x templates (match tenx-template tag only).
#
[OUTPUT]
    Name                splunk
    Match               tenx-template
    Host                127.0.0.1
    Port                8088
    Splunk_Token        <your-hec-token>
    event_index         tenx_dml
    event_source        fluent-bit-tenx
    event_sourcetype    tenx_dml_raw_json
    TLS                 On
    TLS.Verify          On

Below is the default configuration from: optimize/tenx-elastic.conf.

Edit Online Edit Locally

Edit tenx-elastic.conf Locally

Download and save to:

• Linux / Docker / macOS
• Custom directory
  $TENX_CONFIG/run/input/forwarder/fluentbit/optimize/tenx-elastic.conf
• Within cloned repo
  ./pipelines/run/input/forwarder/fluentbit/optimize/tenx-elastic.conf

Download

# Sample Fluent-Bit Elastic 10x Template output configuration
#
# This Fluent-Bit configuration routes 10x templates generated by the 'optimize' sub-process.
# Templates are written to Elastic Search via the 'elasticsearch' plugin
# https://docs.fluentbit.io/manual/pipeline/outputs/elasticsearch
# 
# 10x Templates are used by the 10x elastic search 'Optimize' app to decode
# encoded events to their original form at search-time. 
#
[OUTPUT]
    Name es
    Match tenx-template
    Host localhost
    Port 9200
    Index l1es_dml
    Id_Key templateHash
    Retry_Limit False

Options

Specify the options below to configure the Fluent Bit optimizer:

Name	Description
fluentbitEncodeObjects	Controls whether events written to a Fluent Bit forwarder are encoded or written in raw form
fluentbitOutputFields	List of TenXObject field names to include alongside the main event in output
fluentbitOutputEncodeType	Output format when outputFields are set. Possible values: [json, delimited]

fluentbitEncodeObjects

Controls whether events written to a Fluent Bit forwarder are encoded or written in raw form.

Type	Default
Boolean	true

Sets the format in which to serialize TenXObjects back to the Fluent Bit forwarder. If true, write events using the encode function. If false, write events in their original form as read from the Fluent Bit forwarder using the fullText field.

fluentbitOutputFields

List of TenXObject field names to include alongside the main event in output.

Type	Default
List	[]

Specifies TenXObject field names to emit alongside the main event text when writing back to the Fluent Bit forwarder. When empty (default), only the main event field is written. Example fields: level, group, symbolMessage.

fluentbitOutputEncodeType

Output format when outputFields are set. Possible values: [json, delimited].

Type	Default
String	delimited

Specifies how the combined output (main event field plus outputFields) is encoded when writing back to the Fluent Bit forwarder. Possible values:

• json: formats all fields as a JSON object
• delimited: formats field values separated by the output delimiter Only takes effect when fluentbitOutputFields is set.

This module is defined in optimize/module.yaml.