Skip to content

Regulate

Read events from a Fluentd forwarder to transform them into typed TenXObjects to filter using local/centralized regulator policy. This module is a component of the Edge Regulator app.

Architecture

graph LR
    A["<div style='font-size: 14px;'>📂 Fluentd</div><div style='font-size: 10px;'>sources</div>"] --> B["<div style='font-size: 14px;'>🔧 exec_filter</div><div style='font-size: 10px;'>plugin</div>"]
    B --> C["<div style='font-size: 14px;'>⚡ 10x Regulator</div><div style='font-size: 10px;'>filter policy</div>"]
    C --> B
    B --> D["<div style='font-size: 14px;'>📤 Fluentd</div><div style='font-size: 10px;'>outputs</div>"]

    classDef input fill:#2563eb,stroke:#1d4ed8,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef filter fill:#ea580c,stroke:#c2410c,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef engine fill:#7c3aed,stroke:#6d28d9,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef output fill:#16a34a,stroke:#15803d,color:#ffffff,stroke-width:2px,rx:8,ry:8

    class A input
    class B filter
    class C engine
    class D output

Data Flow

  • 📂 Fluentd Sources - Collect logs from files, TCP, HTTP, or other sources
  • 🔧 exec_filter Plugin - Pipes ALL events to 10x sidecar via stdin
  • ⚡ 10x Regulator - Applies rate/policy-based filtering, drops noisy events
  • 🔄 Bidirectional Pipe - FILTERED events return via stdout to exec_filter
  • 📤 Fluentd Outputs - Only filtered events ship to final destinations

Key Characteristics

Feature Description
🚦 Rate Limiting Filter events based on per-template rate limits
📋 Policy-Based Apply local or centralized filtering policies
💰 Cost Control Reduce log volume and costs by dropping noisy events
🔧 exec_filter Uses Fluentd's native exec_filter for stdin/stdout piping

Sidecar Relay

This module configures a Fluentd exec-filter that launches a 10x sidecar process and passes it collected events to regulate using a local/centralized policy. The sidecar relays regulated events back to the Fluentd filter to ship to outputs (e.g., Splunk, S3).

Install

See the Log10x Edge Regulator Fluentd run instructions

Deploy to k8s via Helm

See the Log10x Edge Regulator Fluentd deployment instructions

Configuration

To configure the Fluentd regulator module, Edit these settings.

Below is the default configuration from: regulate/config.yaml.

Edit Online

Edit config.yaml Locally 

# 🔟❎ 'run' Fluentd regulator configuration

# Configure an input that reads events from a Fluentd forwarder and
# an output that writes filtered TenXObjects back to Fluentd.

# This config runs in conjunction with the instructions provided in:
# https://doc.log10x.com/run/input/forwarder/fluentd/regulate/#install

# Set the 10x pipeline to 'run'
tenx: run

# =============================== Dependencies ================================

include:

  # Include output encoding format options
  - run/modules/input/forwarder/fluentd/output/module.yaml

  # Include the stdin input from which to read events from the forwarder
  - run/modules/input/forwarder/fluentd/input

  # Include Unix domain socket/stdout to write events into the forwarder
  - '$="run/modules/input/forwarder/fluentd/output/" + (TenXString.includes(TenXEnv.get("os.name"), "Windows") ? "stdio" : "unix")'

# ============================== Fluentd Options ==============================

fluentd:
  # 'encodeObjects' specifies the target output (Unix/stdout) will write filtered
  #  TenXObjects in their original form via the intrinsic 'fullText' field.
  encodeObjects: false


This module is defined in regulate/module.yaml.