Filebeat

Runs 10x Engine as a sidecar to Filebeat for reporting, receiving, and optimizing events before they ship to their destination (Elasticsearch, Logstash, Kafka, S3, …). Filebeat's plugin model doesn't expose the Fluent Forward protocol used by other forwarders, so Log10x and Filebeat exchange events through Filebeat's own native extension points instead: a script processor on every input emits enriched events to Filebeat's stdout, and a unix input loads processed events back over a local socket. Filebeat runs as a child process of the sidecar (filebeat -e 2>&1 | tenx ...); works against any stock Filebeat build (Linux/macOS/Windows) and the upstream elastic/filebeat Helm chart on Kubernetes via an image swap to the prebuilt log10x/filebeat-10x image.

Architecture

graph LR
    A["<div style='font-size: 14px;'>📂 Inputs</div><div style='font-size: 10px;'>filestream, container, log</div>"] --> F["<div style='font-size: 14px;'>🧪 script processor</div><div style='font-size: 10px;'>tenx-*.js</div>"]
    F --> B["<div style='font-size: 14px;'>📤 stdout</div><div style='font-size: 10px;'>JSON line per event</div>"]
    B --> E["<div style='font-size: 14px;'>⚡ 10x Engine</div><div style='font-size: 10px;'>Receive/Optimize</div>"]
    E --> C["<div style='font-size: 14px;'>🔌 unix input</div><div style='font-size: 10px;'>/tmp/tenx_filebeat.sock</div>"]
    C --> D["<div style='font-size: 14px;'>📤 Outputs</div><div style='font-size: 10px;'>ES, Logstash, Kafka, S3</div>"]

    classDef input fill:#2563eb,stroke:#1d4ed8,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef filter fill:#ea580c,stroke:#c2410c,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef engine fill:#7c3aed,stroke:#6d28d9,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef socket fill:#0891b2,stroke:#0e7490,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef output fill:#16a34a,stroke:#15803d,color:#ffffff,stroke-width:2px,rx:8,ry:8

    class A input
    class B filter
    class C socket
    class D output
    class E engine
    class F filter

Data Flow

• 📂 Inputs — Your existing Filebeat inputs (filestream, container, log, journald, …) collect events and pass them through any processors you've configured on the input (add_kubernetes_metadata, decode_json_fields, dissect, …). Enrichment runs here exactly once before the event is handed off to Log10x.
• 🧪 script processor — A small JavaScript processor on each input (tenx-receive.js for the Receiver, tenx-report.js for the Reporter) marks the event, writes it as a single JSON line to Filebeat's stdout, and cancels it from Filebeat's normal output path. This is what keeps your destinations from seeing the unprocessed event.
• ⚡ 10x Engine — Filebeat runs as a child process of the sidecar (filebeat -e 2>&1 | tenx ...), so its stdout is the engine's stdin. The Receiver app applies rate/policy-based filtering and optionally compacts events for volume reduction. The engine also picks Filebeat's own log lines off the same stream and replays them to Filebeat's configured log destinations (logging.to_files, logging.to_stderr, logging.to_syslog) — so enabling the integration doesn't change where Filebeat logs go.
• 🔌 unix input — Processed events come back to Filebeat through a unix input listening on /tmp/tenx_filebeat.sock (loaded via filebeat.config.inputs from a bundled snippet — same path on both sides). The input's processors decode the JSON payload and remove the script-processor marker, so the second pass of tenx-receive.js lets the event through unmodified.
• 📤 Outputs — Your destination output (elasticsearch, logstash, kafka, file, …) ships the returned event. output.console is not supported — it would write to the same stdout pipe that carries events to the engine and corrupt the stream. Use output.file for local testing without a real destination.

What an event looks like on the way back

The record structure of the original Filebeat event is preserved end-to-end — every field comes back to your destination output with the same name and same position. What changes depends on the Receiver app mode:

Mode	Difference vs the event Filebeat collected
Receive (default)	None. Same record.
Receive + symbolMessageHashField <name>	Adds one new field with the symbol-pattern hash (a stable identifier for the message pattern — usable as a dedup key, metric dimension, or correlation ID).
receiverOptimize true	The value of the message field is replaced with a compact encoded form. A separate tenx-template event is emitted carrying the template needed to decode it (Filebeat's decode_json_fields processor on the return socket uses the embedded templateHashDocId to set the Elasticsearch document ID). All other fields stay verbatim.
receiverOptimize true + symbolMessageHashField <name>	Both of the above.

Internally, Log10x's Filebeat input module reads the message text from the event's message field and surfaces the input source (log.file.path, container, etc.) for use in rate-based grouping. When the Receiver app is configured with k8sExtractorName: filebeatK8s, the kubernetes.* sub-object stamped by add_kubernetes_metadata is also materialized as enrichment fields for message-pattern and rate filtering.

Key Files

File	Purpose
stream.yaml	Stdin input (Filebeat events + Filebeat log lines) and Unix-socket output stream definitions
log4j2.yaml	Appenders that replay Filebeat's own log lines to the destinations declared in filebeat.yml's logging.*
script/tenx-receive.js	Receiver processor — marks + emits events to stdout, cancels them so they loop back over the socket
script/tenx-report.js	Reporter processor — read-only variant that emits to stdout without canceling
conf/tenxNix.yml	unix input snippet for Linux/macOS — referenced from your filebeat.yml via filebeat.config.inputs.path, and read by the engine to discover the socket address
conf/tenxWin.yml	Same as above for Windows (uses ${TEMP}\tenx_filebeat.sock)

Quickstart

1. Wire up your Filebeat config — load the return-path unix input and add the script processor to your inputs:

filebeat.yml

# Loads the unix input that receives processed events back from Log10x.
filebeat.config.inputs:
  enabled: true
  # Linux/macOS
  path: ${TENX_MODULES}/pipelines/run/modules/input/forwarder/filebeat/conf/tenxNix.yml
  # Windows
  # path: ${TENX_MODULES}/pipelines/run/modules/input/forwarder/filebeat/conf/tenxWin.yml

filebeat.inputs:
  - type: filestream
    id: app-logs
    paths:
      - /var/log/app.log

    processors:
      # Hands every event off to Log10x via Filebeat's stdout, then cancels
      # it locally so destinations only see events that came back on the
      # unix socket.
      - script:
          lang: javascript
          file: ${TENX_MODULES}/pipelines/run/modules/input/forwarder/filebeat/script/tenx-receive.js

# Use any non-stdout output — output.console would collide with the stdout
# pipe that carries events to the engine.
output.elasticsearch:
  hosts: ["https://elasticsearch:9200"]

2. Run Filebeat through Log10x — Filebeat is launched as a child process of the sidecar, so start them as a single pipeline:

filebeat -c filebeat.yml -e 2>&1 | tenx run @run/input/forwarder/filebeat @apps/receiver

For Kubernetes deployment, swap the chart's default Filebeat image for the prebuilt log10x/filebeat-10x on top of the upstream elastic/filebeat chart — see the Helm chart overlay. For read-only Reporter mode (no event diversion) swap tenx-receive.js for tenx-report.js and run against @apps/reporter — see the Reporter Quickstart.

Config Files

To configure the Filebeat module, Edit these files.

Below is the default configuration from: filebeat/config.yaml.

Edit Online Edit Locally

Edit config.yaml Locally

Download and save to:

• Linux / Docker / macOS
• Custom directory
  $TENX_CONFIG/run/input/forwarder/filebeat/config.yaml
• Within cloned repo
  ./pipelines/run/input/forwarder/filebeat/config.yaml

Download

# 🔟❎ 'run' Filebeat receiver configuration
#
# Configures an input that reads events from a Filebeat forwarder (Filebeat
# runs as a subprocess of the sidecar) and an output that writes processed
# events back to Filebeat over a Unix domain socket.
# When 'receiverOptimize' is enabled, events are encoded for volume reduction.
#
# To learn more see https://doc.log10x.com/run/input/forwarder/filebeat/

tenx: run

# =============================== Dependencies ================================

include:
  - run/input/forwarder/config.yaml
  - run/modules/input/forwarder/filebeat

# ============================== Filebeat Options ==============================

filebeat:

  # ----------------------------- Output Options ----------------------------

  output:

    # 'fields' specifies TenXObject field names to emit alongside the main
    #  event. When set, enrichment fields are included in the sidecar output
    #  for forwarder post-processing.
    #  Example: fields: [level, group, symbolMessage]
    fields: []

    # 'encodeType' controls the output format when fields is set.
    #  Possible values: 'json' or 'delimited'
    encodeType: delimited

Options

Specify the options below to configure the Filebeat:

Name	Description	Category
filebeatOutputFields	List of TenXObject field names to include alongside the main event in output	Output
filebeatOutputEncodeType	Output format when outputFields are set. Possible values: [json, delimited]	Output
filebeatConfigPath	Configuration file path	FilebeatLog
filebeatLogsPath	Log file path	FilebeatLog
filebeatLoggingToFiles	Control log file rotation	FilebeatLog
filebeatLoggingToStderr	Filebeat logging to stderr	FilebeatLog
filebeatLoggingToSyslog	Filebeat logging to syslog	FilebeatLog
filebeatName	Filebeat log name	FilebeatLog
filebeatPath	Output path for Filebeat logs	FilebeatLog
filebeatInterval	Enable Filebeat log file rotation	FilebeatLog
filebeatRotateEveryBytes	Filebeat log file size limit	FilebeatLog
filebeatRotateOnStartup	Rotate existing Filebeat logs upon startup	FilebeatLog
filebeatPermissions	Log file permissions mask	FilebeatLog
filebeatKeepFiles	Number of rotated Filebeat log files to keep	FilebeatLog
rawOutputUnixSocketAddress	Internal	Internal
filebeatInputReady	Internal	Internal

Output

filebeatOutputFields

List of TenXObject field names to include alongside the main event in output.

Type	Default	Category
List	[]	Output

Specifies TenXObject field names to emit alongside the main event text when writing back to the Filebeat forwarder. When empty (default), only the main event field is written. Example fields: level, group, symbolMessage.

filebeatOutputEncodeType

Output format when outputFields are set. Possible values: [json, delimited].

Type	Default	Category
String	delimited	Output

Specifies how the combined output (main event field plus outputFields) is encoded when writing back to the Filebeat forwarder. Possible values:

• json: formats all fields as a JSON object
• delimited: formats field values separated by the output delimiter Only takes effect when filebeatOutputFields is set.

FilebeatLog

filebeatConfigPath

Configuration file path.

Type	Default	Category
File		FilebeatLog

Path containing Filebeat configuration files.

filebeatLogsPath

Log file path.

Type	Default	Category
File		FilebeatLog

Path containing Filebeat log files.

filebeatLoggingToFiles

Control log file rotation.

Type	Default	Category
Boolean	false	FilebeatLog

Sets Filebeat logging to rotating files. Set logging.to_files to false to disable logging to files.

filebeatLoggingToStderr

Filebeat logging to stderr.

Type	Default	Category
Boolean	false	FilebeatLog

Filebeat logging to stderr. Set logging.to_stderr to false to disable logging to stderr.

filebeatLoggingToSyslog

Filebeat logging to syslog.

Type	Default	Category
Boolean	false	FilebeatLog

Filebeat logging to syslog. Set logging.to_syslog to false to disable logging to syslog.

filebeatName

Filebeat log name.

Type	Default	Category
File		FilebeatLog

Name of the Filebeat files where the logs are written to.

filebeatPath

Output path for Filebeat logs.

Type	Default	Category
File		FilebeatLog

Configures the path where the Filebeat logs are written.

filebeatInterval

Enable Filebeat log file rotation.

Type	Default	Category
String	""	FilebeatLog

Enable Filebeat log file rotation on time intervals in addition to the size-based rotation.

filebeatRotateEveryBytes

Filebeat log file size limit.

Type	Default	Category
Number	0	FilebeatLog

Configure Filebeat log file size limit. If limit is reached, log file will be automatically rotated.

filebeatRotateOnStartup

Rotate existing Filebeat logs upon startup.

Type	Default	Category
Boolean	false	FilebeatLog

Rotate existing Filebeat logs on startup rather than appending them to the existing file.

filebeatPermissions

Log file permissions mask.

Type	Default	Category
String	""	FilebeatLog

POSIX permissions mask to apply when rotating Filebeat log files.

filebeatKeepFiles

Number of rotated Filebeat log files to keep.

Type	Default	Category
Number	0	FilebeatLog

Number of rotated Filebeat log files to keep. Oldest files will be deleted first.

Internal

rawOutputUnixSocketAddress

Internal.

Type	Default	Category
String	""	Internal

Internal variable used to load socket address to be used in multiple encoders.

filebeatInputReady

Internal.

Type	Default	Category
String	""	Internal

An internal marker used to indicate Filebeat input ready.

This module is defined in filebeat/module.yaml.