Optimize
Losslessly compact log/trace events collected by Fluent Bit forwarders before they ship to output (e.g., Splunk, S3). This module is a component of the Edge Optimizer app.
Architecture
graph LR
A["<div style='font-size: 14px;'>📂 Fluent Bit</div><div style='font-size: 10px;'>inputs</div>"] --> B["<div style='font-size: 14px;'>🔧 Lua Filter</div><div style='font-size: 10px;'>tenx.lua</div>"]
B --> C["<div style='font-size: 14px;'>⚡ 10x Optimizer</div><div style='font-size: 10px;'>encode()</div>"]
C --> D["<div style='font-size: 14px;'>🔌 Unix/TCP</div><div style='font-size: 10px;'>input</div>"]
D --> E["<div style='font-size: 14px;'>📤 Fluent Bit</div><div style='font-size: 10px;'>outputs</div>"]
classDef input fill:#2563eb,stroke:#1d4ed8,color:#ffffff,stroke-width:2px,rx:8,ry:8
classDef filter fill:#ea580c,stroke:#c2410c,color:#ffffff,stroke-width:2px,rx:8,ry:8
classDef engine fill:#7c3aed,stroke:#6d28d9,color:#ffffff,stroke-width:2px,rx:8,ry:8
classDef socket fill:#0891b2,stroke:#0e7490,color:#ffffff,stroke-width:2px,rx:8,ry:8
classDef output fill:#16a34a,stroke:#15803d,color:#ffffff,stroke-width:2px,rx:8,ry:8
class A input
class B filter
class C engine
class D socket
class E outputData Flow
- 📂 Fluent Bit Inputs - Collect logs from files, containers, or other sources
- 🔧 Lua Filter - Intercepts events and pipes them to 10x sidecar
- ⚡ 10x Optimizer - Losslessly compacts events to reduce log volume 50-80%
- 🔌 Unix/TCP Input - Receives COMPACT events back from the sidecar
- 📤 Fluent Bit Outputs - Compact events ship to final destinations at reduced size
Key Characteristics
| Feature | Description |
|---|---|
| 📦 Lossless Compact | Compacts events to reduce log volume 50-80% |
| 🔗 Template Extraction | Repetitive structures become reusable templates |
| 💰 Cost Savings | Reduced storage and transfer costs |
| 🔧 Lua Filter | Uses Fluent Bit's native Lua scripting for sidecar launch |
Sidecar Relay
This module configures a Fluent Bit Lua filter and Unix/TCP input. The Lua filter launches a 10x sidecar process and passes it collected events to encode. The sidecar relays compact events back to Fluent Bit via the configured Unix/TCP input to ship to outputs (e.g., Splunk, S3).
Install
See the Log10x Edge Optimizer Fluent Bit run instructions
Deploy to k8s via Helm
See the Log10x Edge Optimizer Fluent Bit deployment instructions
Configuration
To configure the Fluent Bit optimizer module, Edit these settings.
Below is the default configuration from: optimize/config.yaml.
ewogICJ0eXBlIiA6ICJvYmplY3QiLAogICJwcm9wZXJ0aWVzIiA6IHsKICAgICJpbmNsdWRlIiA6IHsKICAgICAgInR5cGUiIDogImFycmF5IiwKICAgICAgIml0ZW1zIiA6IHsKICAgICAgICAidHlwZSIgOiAic3RyaW5nIgogICAgICB9CiAgICB9LAogICAgInRlbngiIDogewogICAgICAidHlwZSIgOiAic3RyaW5nIgogICAgfQogIH0sCiAgImFkZGl0aW9uYWxQcm9wZXJ0aWVzIiA6IGZhbHNlCn0=
# 🔟❎ 'run' Fluent Bit optimizer configuration
# Configure an input that reads events from a Fluent Bit forwarder and
# writes optimized TenXObjects back to Fluent Bit to ship to output (e.g., ElasticSearch, Splunk).
# To learn more see https://doc.log10x.com/run/input/forwarder/fluentbit/optimize/
# This config runs in conjunction with the instructions provided in:
# https://doc.log10x.com/run/input/forwarder/fluentbit/optimize/#install
# Set the 10x pipeline to 'run'
tenx: run
# =============================== Dependencies ================================
include:
# Include output encoding format options
- run/modules/input/forwarder/fluentbit/output/module.yaml
# Include the stdin input from which to read events from the forwarder
- run/modules/input/forwarder/fluentbit/input
# Include Unix domain/TCP socket output to which events into the forwarder
- '$="run/modules/input/forwarder/fluentbit/output/" + (TenXString.includes(TenXEnv.get("os.name"), "Windows") ? "tcp" : "unix")'
# ============================= Fluentbit Options =============================
fluentbit:
# 'encodeObjects' is set to true to write TenXObjects
# back to the forwarder using the encode() function.
# To learn more see https://doc.log10x.com/api/js/#TenXObject+encode
encodeObjects: true
Below is the default configuration from: optimize/tenx-splunk.conf.
ewogICJ0eXBlIiA6ICJvYmplY3QiLAogICJwcm9wZXJ0aWVzIiA6IHsKICAgICJpbmNsdWRlIiA6IHsKICAgICAgInR5cGUiIDogImFycmF5IiwKICAgICAgIml0ZW1zIiA6IHsKICAgICAgICAidHlwZSIgOiAic3RyaW5nIgogICAgICB9CiAgICB9LAogICAgInRlbngiIDogewogICAgICAidHlwZSIgOiAic3RyaW5nIgogICAgfQogIH0sCiAgImFkZGl0aW9uYWxQcm9wZXJ0aWVzIiA6IGZhbHNlCn0=
# Sample Fluent-Bit Splunk 10x output configuration
#
# This Fluent-Bit configuration routes 10x templates and encoded events to Splunk
# via the HTTP Event Collector (HEC).
# https://docs.fluentbit.io/manual/pipeline/outputs/splunk
#
# 10x Templates are used by the 10x Splunk 'Optimize' app to inflate
# encoded events to their original form at search time.
#
# IMPORTANT: You must create HEC tokens in Splunk and replace <your-hec-token> below.
# See: https://docs.splunk.com/Documentation/Splunk/latest/Data/UsetheHTTPEventCollector
#
# Index should be set based on indexing policies.
#
# Source should be set to something unique to help you identify events.
#
# Sourcetype must match the config of the Splunk 10x app and should most likely not be changed.
#
# ========================= ENCODED EVENTS OUTPUT =========================
# Use this output for encoded log events (match all except templates).
# IMPORTANT: Use 'event_key $log' to send only the log content without JSON wrapping.
# Without event_key, Splunk receives {"log":"~hash,var1,var2..."} instead of just the encoded data.
#
# [OUTPUT]
# Name splunk
# Match_Regex ^(?!tenx-template).*
# Host 127.0.0.1
# Port 8088
# Splunk_Token <your-encoded-events-hec-token>
# event_index tenx_encoded
# event_sourcetype tenx_encoded
# event_key $log
# TLS On
# TLS.Verify On
#
# ========================= TEMPLATES OUTPUT ==============================
# Use this output for 10x templates (match tenx-template tag only).
#
[OUTPUT]
Name splunk
Match tenx-template
Host 127.0.0.1
Port 8088
Splunk_Token <your-hec-token>
event_index tenx_dml
event_source fluent-bit-tenx
event_sourcetype tenx_dml_raw_json
TLS On
TLS.Verify On
Below is the default configuration from: optimize/tenx-elastic.conf.
ewogICJ0eXBlIiA6ICJvYmplY3QiLAogICJwcm9wZXJ0aWVzIiA6IHsKICAgICJpbmNsdWRlIiA6IHsKICAgICAgInR5cGUiIDogImFycmF5IiwKICAgICAgIml0ZW1zIiA6IHsKICAgICAgICAidHlwZSIgOiAic3RyaW5nIgogICAgICB9CiAgICB9LAogICAgInRlbngiIDogewogICAgICAidHlwZSIgOiAic3RyaW5nIgogICAgfQogIH0sCiAgImFkZGl0aW9uYWxQcm9wZXJ0aWVzIiA6IGZhbHNlCn0=
# Sample Fluent-Bit Elastic 10x Template output configuration
#
# This Fluent-Bit configuration routes 10x templates generated by the 'optimize' sub-process.
# Templates are written to Elastic Search via the 'elasticsearch' plugin
# https://docs.fluentbit.io/manual/pipeline/outputs/elasticsearch
#
# 10x Templates are used by the 10x elastic search 'Optimize' app to decode
# encoded events to their original form at search-time.
#
[OUTPUT]
Name es
Match tenx-template
Host localhost
Port 9200
Index l1es_dml
Id_Key templateHash
Retry_Limit False
This module is defined in optimize/module.yaml.