Skip to content

Receive

Read events from a Fluentd forwarder to transform them into typed TenXObjects to filter using local/centralized receiver policy. This module is a component of the Receiver app.

Architecture

graph LR
    A["<div style='font-size: 14px;'>📂 Fluentd</div><div style='font-size: 10px;'>sources</div>"] --> B["<div style='font-size: 14px;'>🔧 exec_filter</div><div style='font-size: 10px;'>plugin</div>"]
    B --> C["<div style='font-size: 14px;'>⚡ 10x Receiver</div><div style='font-size: 10px;'>filter policy</div>"]
    C --> B
    B --> D["<div style='font-size: 14px;'>📤 Fluentd</div><div style='font-size: 10px;'>outputs</div>"]

    classDef input fill:#2563eb,stroke:#1d4ed8,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef filter fill:#ea580c,stroke:#c2410c,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef engine fill:#7c3aed,stroke:#6d28d9,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef output fill:#16a34a,stroke:#15803d,color:#ffffff,stroke-width:2px,rx:8,ry:8

    class A input
    class B filter
    class C engine
    class D output

Data Flow

  • 📂 Fluentd Sources - Collect logs from files, TCP, HTTP, or other sources
  • 🔧 exec_filter Plugin - Pipes ALL events to 10x sidecar via stdin
  • ⚡ 10x Receiver - Applies rate/policy-based filtering, drops noisy events
  • 🔄 Bidirectional Pipe - FILTERED events return via stdout to exec_filter
  • 📤 Fluentd Outputs - Only filtered events ship to final destinations

Key Characteristics

Feature Description
🚦 Rate Limiting Filter events based on per-template rate limits
📋 Policy-Based Apply local or centralized filtering policies
💰 Cost Control Reduce log volume and costs by dropping noisy events
🔧 exec_filter Uses Fluentd's native exec_filter for stdin/stdout piping

Sidecar Relay

This module configures a Fluentd exec-filter that launches a 10x sidecar process and passes it collected events to receive using a local/centralized policy. The sidecar relays received events back to the Fluentd filter to ship to outputs (e.g., Splunk, S3).

Install

See the Log10x Receiver Fluentd run instructions

Deploy to k8s via Helm

See the Log10x Receiver Fluentd deployment instructions

Config Files

To configure the Fluentd Receiver module, Edit these files.

Below is the default configuration from: receive/config.yaml.

Edit Online

Edit config.yaml Locally 

# 🔟❎ 'run' Fluentd receiver configuration

# Configure an input that reads events from a Fluentd forwarder and
# an output that writes received TenXObjects back to Fluentd.
# When 'receiverOptimize' is enabled, events are encoded for volume reduction.

# This config runs in conjunction with the instructions provided in:
# https://doc.log10x.com/run/input/forwarder/fluentd/receive/#install

# Set the 10x pipeline to 'run'
tenx: run

# =============================== Dependencies ================================

include:

  # Parent module declaring shared forwarder options (outputEncodeField, outputWriteTemplates, encodeObjects)
  - run/modules/input/forwarder/module.yaml

  - run/modules/input/forwarder/fluentd/receive/module.yaml

  # Include the stdin input from which to read events from the forwarder
  - run/modules/input/forwarder/fluentd/input

  # Include Unix domain socket/stdout to write events into the forwarder
  - '$="run/modules/input/forwarder/fluentd/output/" + (TenXString.includes(TenXEnv.get("os.name"), "Windows") ? "stdio" : "unix")'

# =============================== Mode Options ================================

# See fluentbit/receive/config.yaml for the full explanation of this dispatch.
# Summary: receiverOptimize=true → compact-all; compactReceiverLookupFile=<p> → compact-lookup.

outputEncodeField: '$=yield TenXEnv.get("compactReceiverLookupFile") ? "encoded=shouldEncode() ? encode() : fullText" : (TenXEnv.get("receiverOptimize") ? "encoded=encode()" : "fullText")'

outputWriteTemplates: $=yield TenXEnv.get("receiverOptimize", false) || TenXEnv.get("compactReceiverLookupFile", false)

# Legacy flag retained for fluentd/stdio (which emits TSV with joinFields and
# can't cleanly use outputEncodeField). fluentd/unix uses outputEncodeField.
encodeObjects: $=yield TenXEnv.get("receiverOptimize", false) || TenXEnv.get("compactReceiverLookupFile", false)

# ============================== Fluentd Options ==============================

fluentd:

  # 'outputFields' specifies TenXObject field names to emit alongside the main event.
  #  When set, enrichment fields are included in the sidecar output for forwarder post-processing.
  #  Example: outputFields: [level, group, symbolMessage]
  outputFields: []

  # 'outputEncodeType' controls the output format when outputFields is set.
  #  Possible values: 'json' or 'delimited'
  outputEncodeType: delimited

Options

Specify the options below to configure the Fluentd Receiver:

Name Description
fluentdOutputFields List of TenXObject field names to include alongside the main event in output
fluentdOutputEncodeType Output format when outputFields are set. Possible values: [json, delimited]

fluentdOutputFields

List of TenXObject field names to include alongside the main event in output.

Type Default
List []

Specifies TenXObject field names to emit alongside the main event text when writing back to the Fluentd forwarder. When empty (default), only the main event field is written. Example fields: level, group, symbolMessage.

fluentdOutputEncodeType

Output format when outputFields are set. Possible values: [json, delimited].

Type Default
String delimited

Specifies how the combined output (main event field plus outputFields) is encoded when writing back to the Fluentd forwarder. Possible values:

  • json: formats all fields as a JSON object
  • delimited: formats field values separated by the output delimiter Only takes effect when fluentdOutputFields is set.


This module is defined in receive/module.yaml.