Skip to content

Regulate

Read events from a Fluent Bit forwarder to transform into typed TenXObjects and filter using local/centralized regulator policies. This module is a component of the Edge Regulator app.

Architecture

graph LR
    A["<div style='font-size: 14px;'>📂 Fluent Bit</div><div style='font-size: 10px;'>inputs</div>"] --> B["<div style='font-size: 14px;'>🔧 Lua Filter</div><div style='font-size: 10px;'>tenx.lua</div>"]
    B --> C["<div style='font-size: 14px;'>⚡ 10x Regulator</div><div style='font-size: 10px;'>filter policy</div>"]
    C --> D["<div style='font-size: 14px;'>🔌 Unix/TCP</div><div style='font-size: 10px;'>input</div>"]
    D --> E["<div style='font-size: 14px;'>📤 Fluent Bit</div><div style='font-size: 10px;'>outputs</div>"]

    classDef input fill:#2563eb,stroke:#1d4ed8,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef filter fill:#ea580c,stroke:#c2410c,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef engine fill:#7c3aed,stroke:#6d28d9,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef socket fill:#0891b2,stroke:#0e7490,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef output fill:#16a34a,stroke:#15803d,color:#ffffff,stroke-width:2px,rx:8,ry:8

    class A input
    class B filter
    class C engine
    class D socket
    class E output

Data Flow

  • 📂 Fluent Bit Inputs - Collect logs from files, containers, or other sources
  • 🔧 Lua Filter - Intercepts ALL events and pipes them to 10x sidecar
  • ⚡ 10x Regulator - Applies rate/policy-based filtering, drops noisy events
  • 🔌 Unix/TCP Input - Receives FILTERED events back from the sidecar
  • 📤 Fluent Bit Outputs - Only filtered events ship to final destinations

Key Characteristics

Feature Description
🚦 Rate Limiting Filter events based on per-template rate limits
📋 Policy-Based Apply local or centralized filtering policies
💰 Cost Control Reduce log volume and costs by dropping noisy events
🔧 Lua Filter Uses Fluent Bit's native Lua scripting for sidecar launch

Sidecar Relay

This module configures a Fluent Bit Lua filter and Unix/TCP input. The Lua filter launches a 10x sidecar process and passes it collected events to regulate. The sidecar relays regulated events back to Fluent Bit via the configured Unix/TCP input to ship to output(s) (e.g., Splunk, S3).

Install

See the Log10x Edge Regulator Fluent Bit run instructions

Deploy to k8s via Helm

See the Log10x Edge Regulator Fluent Bit deployment instructions

Configuration

To configure the Fluent Bit regulator module, Edit these settings.

Below is the default configuration from: regulate/config.yaml.

Edit Online

Edit config.yaml Locally 

# 🔟❎ 'run' Fluent Bit regulator configuration

# Configure an input that reads events from a Fluent Bit forwarder and
# an output that writes filtered TenXObjects back to Fluent Bit.

# This config file operates in conjunction with the instructions provided in:
# https://doc.log10x.com/run/input/forwarder/fluentbit/regulate/#install

# Set the 10x pipeline to 'run'
tenx: run

# =============================== Dependencies ================================

include:

  # Include output encoding format options
  - run/modules/input/forwarder/fluentbit/output/module.yaml

  # Include the stdin input to read events from the forwarder
  - run/modules/input/forwarder/fluentbit/input

  # Include Unix domain/TCP socket output to write events into the forwarder
  - '$="run/modules/input/forwarder/fluentbit/output/" + (TenXString.includes(TenXEnv.get("os.name"), "Windows") ? "tcp" : "unix")'

# ============================= Fluent Bit Options ============================

fluentbit:
  # 'encodeObjects' specifies the target output (Unix domain/TCP) will write
  #  TenXObjects in a plain-text format using the intrinsic 'fullText' field.
  encodeObjects: false


This module is defined in regulate/module.yaml.