Skip to content

Filebeat Input

Filebeat modules execute as a sidecar process to report, regulate, and optimize events before they ship to output (e.g., ElasticSearch).

Architecture

graph LR
    A["<div style='font-size: 14px;'>📂 Input</div><div style='font-size: 10px;'>tail, k8s</div>"] --> B["<div style='font-size: 14px;'>🔧 script</div><div style='font-size: 10px;'>tenx-*.js</div>"]
    B --> C["<div style='font-size: 14px;'>âš¡ 10x Pipeline</div><div style='font-size: 10px;'>Parse/Optimize/Regulate</div>"]
    C --> D["<div style='font-size: 14px;'>🔌 Unix Input</div><div style='font-size: 10px;'>back to Filebeat</div>"]
    D --> E["<div style='font-size: 14px;'>📤 Output</div><div style='font-size: 10px;'>Elasticsearch</div>"]

    classDef input fill:#2563eb,stroke:#1d4ed8,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef script fill:#ea580c,stroke:#c2410c,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef engine fill:#7c3aed,stroke:#6d28d9,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef socket fill:#0891b2,stroke:#0e7490,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef output fill:#16a34a,stroke:#15803d,color:#ffffff,stroke-width:2px,rx:8,ry:8

    class A input
    class B script
    class C engine
    class D socket
    class E output

Data Flow

Component Protocol Description
âš¡ 10x Process launch Launches Filebeat as subprocess
âš¡ 10x Config parsing Reads Filebeat's config paths from stdout
🔧 script processor JSON/stdout tenx-optimize.js writes events to stdout
âš¡ 10x stdin JSON Reads and parses events from Filebeat's stdout
âš¡ 10x Pipeline Internal Processes event (report/regulate/optimize)
🔌 Unix socket JSON Writes processed events to Unix socket
🔌 Filebeat unix input JSON decode Reads from socket, decodes JSON fields
📤 Filebeat Output Output Sends to Elasticsearch with correct document IDs

Expected Event Format

The 10x Engine expects JSON events from Filebeat containing:

Field Description Used For
tenx Boolean marker (true) set by script processor Event filtering via sourceFilter
message The actual log message Message extraction

Unlike other forwarders, Filebeat uses sourceFilter with the pattern \"tenx\":true to identify events from the script processor versus internal Filebeat log messages.

Output Limitation: output.console is not supported

The 10x engine reads events from Filebeat via a stdout pipe (filebeat ... 2>&1 | tenx-edge run ...). The script processor writes "tenx":true-marked events to stdout, and the engine uses sourceFilter to separate these from Filebeat's internal log messages.

output.console must not be used when Log10x is enabled — it writes multi-line JSON to the same stdout pipe, corrupting event boundaries and causing JSON parsing errors in the engine.

Supported outputs include output.elasticsearch, output.logstash, output.file, output.kafka, and any other output that does not write to stdout. For local testing without Elasticsearch, use output.file.

Key Files
File Purpose
script/tenx-optimize.js Filebeat script processor - encodes events to stdout
script/tenx-regulate.js Filebeat script processor for regulate mode
optimize/tenxNix.yml Filebeat Unix socket input config (Linux/macOS)
optimize/tenxWin.yml Filebeat Unix socket input config (Windows)
input/stream.yaml 10x stdin input with Filebeat config parsing
input/log4j2.yaml Log4j config for Filebeat internal logs
output/stream.yaml 10x Unix socket output configuration

Modules

  • Filebeat Reporter

    Read events from Filebeat forwarders.

    More info

  • Filebeat Regulator

    Read events from Filebeat forwarders to regulate which events to ship.

    More info

  • Filebeat Optimizer

    Optimize events collected by Filebeat forwarders.

    More info


This module is defined in filebeat/module.yaml.