Skip to content

Regulate

Read events from a Filebeat forwarder to transform them into typed TenXObjects to filter using local/centralized regulator policy. This module is a component of the Edge Regulator app.

Architecture

graph LR
    A["<div style='font-size: 14px;'>📂 Filebeat</div><div style='font-size: 10px;'>inputs</div>"] --> B["<div style='font-size: 14px;'>🔧 Script</div><div style='font-size: 10px;'>processor</div>"]
    B --> C["<div style='font-size: 14px;'>⚡ 10x Regulator</div><div style='font-size: 10px;'>filter policy</div>"]
    C --> D["<div style='font-size: 14px;'>🔌 Unix/TCP</div><div style='font-size: 10px;'>input</div>"]
    D --> E["<div style='font-size: 14px;'>📤 Filebeat</div><div style='font-size: 10px;'>outputs</div>"]

    classDef input fill:#2563eb,stroke:#1d4ed8,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef filter fill:#ea580c,stroke:#c2410c,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef engine fill:#7c3aed,stroke:#6d28d9,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef socket fill:#0891b2,stroke:#0e7490,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef output fill:#16a34a,stroke:#15803d,color:#ffffff,stroke-width:2px,rx:8,ry:8

    class A input
    class B filter
    class C engine
    class D socket
    class E output

Data Flow

  • 📂 Filebeat Inputs - Collect logs from files, containers, or other sources
  • 🔧 Script Processor - Intercepts ALL events and pipes them to 10x sidecar
  • ⚡ 10x Regulator - Applies rate/policy-based filtering, drops noisy events
  • 🔌 Unix/TCP Input - Receives FILTERED events back from the sidecar
  • 📤 Filebeat Outputs - Only filtered events ship to final destinations

Key Characteristics

Feature Description
🚦 Rate Limiting Filter events based on per-template rate limits
📋 Policy-Based Apply local or centralized filtering policies
💰 Cost Control Reduce log volume and costs by dropping noisy events
🔧 Script Processor Uses Filebeat's native JavaScript processor for sidecar launch

Sidecar Relay

This module configures a Filebeat script processor and TCP/Unix input. The script processor launches a 10x sidecar process and pipes collected events to it to regulate. The sidecar relays regulated events back to Filebeat via the configured Unix/TCP input to ship to outputs (e.g., ElasticSearch).

Install

See the Log10x Edge Regulator Filebeat run instructions

Deploy to k8s via Helm

See the Log10x Edge Regulator Filebeat deployment instructions

Configuration

To configure the Filebeat regulator module, Edit these settings.

Below is the default configuration from: regulate/config.yaml.

Edit Online

Edit config.yaml Locally 

# 🔟❎ 'run' Filebeat regulator configuration

# Configure an input that reads events from a Filebeat forwarder and
# an output that writes filtered TenXObjects back to Filebeat.

# This config file operates in conjunction with the instructions provided in:
# https://doc.log10x.com/run/input/forwarder/filebeat/regulate/#install

# Set the 10x pipeline to 'run' 
tenx: run

# =============================== Dependencies ================================

include:

  # Input module to read events from Filebeat into 10x pipeline
  - run/modules/input/forwarder/filebeat/input

  # Output module to write filtered events from 10x pipeline into Filebeat
  - run/modules/input/forwarder/filebeat/output

# ============================== Filebeat Options ==============================

filebeat:
  # 'encodeObjects' is false to designate the target output (Unix/stdout)
  #  to emit filtered TenXObjects in their full-body form.
  encodeObjects: false


This module is defined in regulate/module.yaml.