Skip to content

Optimize

Losslessly compact events collected by Filebeat forwarders before they ship to output (e.g., ElasticSearch, S3). This module is a component of the Edge Optimizer app.

Architecture

graph LR
    A["<div style='font-size: 14px;'>📂 Filebeat</div><div style='font-size: 10px;'>inputs</div>"] --> B["<div style='font-size: 14px;'>🔧 Script</div><div style='font-size: 10px;'>processor</div>"]
    B --> C["<div style='font-size: 14px;'>⚡ 10x Optimizer</div><div style='font-size: 10px;'>encode()</div>"]
    C --> D["<div style='font-size: 14px;'>🔌 Unix/TCP</div><div style='font-size: 10px;'>input</div>"]
    D --> E["<div style='font-size: 14px;'>📤 Filebeat</div><div style='font-size: 10px;'>outputs</div>"]

    classDef input fill:#2563eb,stroke:#1d4ed8,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef filter fill:#ea580c,stroke:#c2410c,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef engine fill:#7c3aed,stroke:#6d28d9,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef socket fill:#0891b2,stroke:#0e7490,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef output fill:#16a34a,stroke:#15803d,color:#ffffff,stroke-width:2px,rx:8,ry:8

    class A input
    class B filter
    class C engine
    class D socket
    class E output

Data Flow

  • 📂 Filebeat Inputs - Collect logs from files, containers, or other sources
  • 🔧 Script Processor - Intercepts events and pipes them to 10x sidecar
  • ⚡ 10x Optimizer - Losslessly compacts events to reduce log volume 50-80%
  • 🔌 Unix/TCP Input - Receives COMPACT events back from the sidecar
  • 📤 Filebeat Outputs - Compact events ship to final destinations at reduced size

Key Characteristics

Feature Description
📦 Lossless Compact Compacts events to reduce log volume 50-80%
🔗 Template Extraction Repetitive structures become reusable templates
💰 Cost Savings Reduced storage and transfer costs
🔧 Script Processor Uses Filebeat's native JavaScript processor for sidecar launch

Sidecar Relay

This module configures a Filebeat script processor and TCP/Unix input. The script processor launches a 10x sidecar process and pipes collected events to it to encode. The sidecar relays compact events back to Filebeat via the configured Unix/TCP input to ship to outputs (e.g., ElasticSearch).

Install

See the Log10x Edge Optimizer Filebeat run instructions

Deploy to k8s via Helm

See the Log10x Edge Optimizer Filebeat deployment instructions

Configuration

To configure the Filebeat optimizer module, Edit these settings.

Below is the default configuration from: optimize/config.yaml.

Edit Online

Edit config.yaml Locally 

# 🔟❎ 'run' Filebeat optimizer configuration

# Configure an input that reads events from a Filebeat forwarder and output
# optimized TenXObjects back to Filebeat to ship to output (e.g., ElasticSearch, Splunk).
# To learn more see https://doc.log10x.com/run/input/forwarder/filebeat/optimize/

# Set the 10x pipeline to 'run' 
tenx: run

# =============================== Dependencies ================================

include:

  # Input module to read events from Filebeat into 10x pipeline
  - run/modules/input/forwarder/filebeat/input

  # Output module to write encoded events from 10x pipeline into Filebeat
  - run/modules/input/forwarder/filebeat/output

# ---------------------------- Filebeat Options -------------------------------

filebeat:
  # 'encodeObjects' is set to true to write TenXObjects
  #  back to the forwarder using the encode() function field.
  #  To learn more see https://doc.log10x.com/api/js/#TenXObject+encode
  encodeObjects: true


This module is defined in optimize/module.yaml.