Fluentd

Runs 10x Engine as a sidecar to Fluentd for reporting, receiving, and optimizing events before they ship to their destination (Elasticsearch, Splunk, S3, Kafka, …). Fluentd and Log10x run as peer processes and exchange events over the Fluent Forward protocol — works against any stock Fluentd build (td-agent, fluent-package, OSS) and the official Fluentd Helm chart with a values overlay.

Architecture

graph LR
    A["<div style='font-size: 14px;'>📂 Sources</div><div style='font-size: 10px;'>tail, http, k8s</div>"] --> F["<div style='font-size: 14px;'>🧪 @INGEST</div><div style='font-size: 10px;'>enrichment filters</div>"]
    F --> B["<div style='font-size: 14px;'>📤 out_forward</div><div style='font-size: 10px;'>:24224</div>"]
    B --> E["<div style='font-size: 14px;'>⚡ 10x Engine</div><div style='font-size: 10px;'>Report/Receive/Optimize</div>"]
    E --> C["<div style='font-size: 14px;'>📥 in_forward</div><div style='font-size: 10px;'>:24225 → @OUTPUT</div>"]
    C --> D["<div style='font-size: 14px;'>📤 Destinations</div><div style='font-size: 10px;'>ES, Splunk, S3, Kafka</div>"]

    classDef input fill:#2563eb,stroke:#1d4ed8,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef filter fill:#ea580c,stroke:#c2410c,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef engine fill:#7c3aed,stroke:#6d28d9,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef output fill:#16a34a,stroke:#15803d,color:#ffffff,stroke-width:2px,rx:8,ry:8

    class A input
    class B filter
    class C filter
    class D output
    class E engine
    class F filter

Data Flow

📂 Sources — Your existing Fluentd sources (tail, http, forward, syslog, etc.) route their events into the @INGEST label.
🧪 @INGEST — Your enrichment filters (kubernetes_metadata, record_transformer, parsers, …) run here exactly once before the event is handed off to Log10x.
📤 out_forward → Log10x — Fluentd forwards the enriched event to the Log10x sidecar over TCP :24224 (or a Unix socket on Linux/macOS).
⚡ 10x Engine — The Receiver app applies rate/policy-based filtering and optionally compacts events for volume reduction.
📥 in_forward → @OUTPUT — Processed events come back to Fluentd on :24225 and are routed directly to the @OUTPUT label, which holds your destination <match> blocks. Filters defined under @INGEST are not re-applied, so each event is enriched exactly once.
📤 Destinations — The original Fluentd tag survives the round trip, so destinations that route on $TAG (Splunk index, S3 path, Kafka topic, …) behave the same as if Log10x weren't in the path.

What an event looks like on the way back

The record structure of the original Fluentd event is preserved end-to-end — every field comes back to your @OUTPUT label with the same name and same position. What changes depends on the Receiver app mode:

Mode	Difference vs the event Fluentd sent in
Receive (default)	None. Same record.
Receive + `symbolMessageHashField <name>`	Adds one new field with the symbol-pattern hash (a stable identifier for the message pattern — usable as a dedup key, metric dimension, or correlation ID).
`receiverOptimize true`	The value of the message field (`log` by default, or whatever `fluentdInputMessageField` is set to) is replaced with a compact encoded form. A separate `tenx-template` event is emitted with the template needed to decode it. All other fields stay verbatim.
`receiverOptimize true` + `symbolMessageHashField <name>`	Both of the above.

The original Fluentd tag is carried by the Forward protocol itself and surfaces on the event as its source inside Log10x — used for rate-based grouping and emitted back to Fluentd as the wire tag on the return path. Internally, Log10x's Fluentd input module reads the message text from the field named by fluentdInputMessageField (default log); when the Receiver app is configured with k8sExtractorName: fluentK8s, the kubernetes.* sub-object is also materialized as enrichment fields for use by message-pattern and rate filtering.

Key Files

File	Purpose
`stream.yaml`	Fluentd Forward input + output stream definitions
`conf/tenx-sidecar.conf`	Reference Fluentd config showing `@INGEST` / `@OUTPUT` label routing

Quickstart

1. Run Log10x:

tenx @run/input/forwarder/fluentd @apps/receiver

2. Wire up your Fluentd config — include the sidecar recipe and route your sources to @INGEST:

fluentd.conf

@include "#{ENV['TENX_MODULES']}/pipelines/run/modules/input/forwarder/fluentd/conf/tenx-sidecar.conf"

<source>
  @type tail
  path /var/log/app.log
  tag app.logs
  @label @INGEST          # routes the source into the sidecar
  <parse>
    @type json
  </parse>
</source>

3. Point @OUTPUT at your real destinations (the recipe defaults to stdout for testing):

<label @OUTPUT>
  <match **>
    @type your_output_plugin
    # ... destination config
  </match>
</label>

For Splunk integration see the 10x for Splunk documentation. For Kubernetes deployment via the official Fluentd Helm chart see the Helm sidecar overlay.

Config Files

To configure the Fluentd module, Edit these files.

Below is the default configuration from: fluentd/config.yaml.

Edit Online

# 🔟❎ 'run' Fluentd receiver configuration
#
# To learn more see https://doc.log10x.com/run/input/forwarder/fluentd/

tenx: run

# =============================== Dependencies ================================

include:
  - run/input/forwarder/config.yaml
  - run/modules/input/forwarder/fluentd

# =============================== Fluentd Options =============================

fluentd:

  # ----------------------------- Input Options -----------------------------

  input:

    # 'port' specifies the TCP port to listen on for events from Fluentd's
    #  out_forward plugin.
    port: 24224

    # 'path' switches the input to a Unix domain socket instead of TCP.
    #  When set, takes precedence over 'port'.
    # path: /tmp/tenx-fluentd-in.sock

    # 'messageField' is the name of the JSON field in each Fluentd record
    #  carrying the log line text.
    messageField: log

  # ----------------------------- Output Options ----------------------------

  output:

    # 'host' specifies the TCP host of Fluentd's in_forward source receiving
    #  processed events from Log10x.
    host: 127.0.0.1

    # 'port' specifies the TCP port of Fluentd's in_forward source. MUST differ
    #  from input.port or the two would collide on the same listener.
    port: 24225

    # 'path' switches the output to a Unix domain socket instead of TCP.
    #  When set, takes precedence over 'host'/'port'.
    # path: /tmp/tenx-fluentd-out.sock

    # 'encodeType' is the on-wire format for the Forward record.
    #  - 'delimited': each top-level field of the rendered record becomes its
    #    own Forward record field — preserves the original record's structure.
    #  - 'json': the whole record is wrapped as one stringified field.
    encodeType: delimited

Options

Specify the options below to configure the Fluentd:

Name	Description	Category
fluentdInputPort	TCP port to listen on for events from Fluentd's out_forward plugin	Input
fluentdInputPath	Unix domain socket path to listen on (Linux/macOS)	Input
fluentdInputMessageField	Name of the JSON field carrying the original log line	Input
fluentdOutputHost	TCP host of the Fluentd in_forward source receiving processed events	Output
fluentdOutputPort	TCP port of the Fluentd in_forward source receiving processed events	Output
fluentdOutputPath	Unix domain socket path of the Fluentd in_forward source (Linux/macOS)	Output
fluentdOutputEncodeType	Output format for the Forward record. Possible values: [json, delimited]	Output

Input

`fluentdInputPort`

TCP port to listen on for events from Fluentd's out_forward plugin.

Type	Default	Category
String	24224	Input

TCP port where Log10x listens for events sent by Fluentd's out_forward plugin. Match this against the port in your Fluentd <server> block.

`fluentdInputPath`

Unix domain socket path to listen on (Linux/macOS).

Type	Default	Category
String	""	Input

When set, Log10x listens on a Unix domain socket instead of TCP — useful on Linux/macOS to avoid TCP overhead. Configure Fluentd's out_forward with <transport unix> + a matching path. Ignored on Windows.

`fluentdInputMessageField`

Name of the JSON field carrying the original log line.

Type	Default	Category
String	log	Input

Name of the field in each Fluentd record that contains the actual log message text. The default (log) matches Fluentd's in_tail plugin convention for container and file logs. Set to e.g. message if your sources emit the log line under a different key.

Output

`fluentdOutputHost`

TCP host of the Fluentd in_forward source receiving processed events.

Type	Default	Category
String	127.0.0.1	Output

Hostname or IP where the Fluentd in_forward source is listening for processed events from Log10x. Pairs with fluentdOutputPort.

`fluentdOutputPort`

TCP port of the Fluentd in_forward source receiving processed events.

Type	Default	Category
String	24225	Output

TCP port where the Fluentd in_forward source is listening for processed events from Log10x. MUST differ from the port Log10x's own Forward input listens on (default 24224) or the two would collide on the same socket.

`fluentdOutputPath`

Unix domain socket path of the Fluentd in_forward source (Linux/macOS).

Type	Default	Category
String	""	Output

Optional Unix domain socket path for the Fluentd in_forward source. When set, takes precedence over fluentdOutputHost / fluentdOutputPort. Linux/macOS only — ignored on Windows.

`fluentdOutputEncodeType`

Output format for the Forward record. Possible values: [json, delimited].

Type	Default	Category
String	delimited	Output

How the rendered event field(s) are encoded on the outgoing Forward record. Possible values:

delimited: each top-level field of the rendered record becomes its own Forward record field — preserves the original record's structure on the wire.
json: the whole rendered record is wrapped as one stringified field. Defaults to delimited, which is almost always what you want.

This module is defined in fluentd/module.yaml.