Regulate

Read events from application logs to transform into typed TenXObjects and filter using local/centralized regulator policies, before Splunk UF ships them to indexers. This module is a component of the Edge Regulator app.

Architecture

graph LR
    A["<div style='font-size: 14px;'>📝 App Logs</div><div style='font-size: 10px;'>Folder A</div>"] --> B["<div style='font-size: 14px;'>🔧 Fluent Bit</div><div style='font-size: 10px;'>tail input</div>"]
    B --> C["<div style='font-size: 14px;'>⚡ 10x Regulator</div><div style='font-size: 10px;'>filter policy</div>"]
    C --> D["<div style='font-size: 14px;'>📂 Processed</div><div style='font-size: 10px;'>Folder B</div>"]
    D --> E["<div style='font-size: 14px;'>📤 Splunk UF</div><div style='font-size: 10px;'>monitors B</div>"]
    E --> F["<div style='font-size: 14px;'>🔍 Splunk</div><div style='font-size: 10px;'>Indexers</div>"]

    classDef logs fill:#2563eb,stroke:#1d4ed8,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef fluentbit fill:#ea580c,stroke:#c2410c,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef engine fill:#7c3aed,stroke:#6d28d9,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef processed fill:#0891b2,stroke:#0e7490,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef uf fill:#64748b,stroke:#475569,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef splunk fill:#16a34a,stroke:#15803d,color:#ffffff,stroke-width:2px,rx:8,ry:8

    class A logs
    class B fluentbit
    class C engine
    class D processed
    class E uf
    class F splunk

Data Flow

• 📝 App Logs (Folder A) - Application writes ALL logs to original location
• 🔧 Fluent Bit - Reads from Folder A, passes events to 10x sidecar
• ⚡ 10x Regulator - Applies rate/policy-based filtering, drops noisy events
• 📂 Processed (Folder B) - Only FILTERED events written to new location
• 📤 Splunk UF - Monitors Folder B, forwards reduced volume to Splunk
• 🔍 Splunk Indexers - Receives filtered events (reduced volume)

Key Characteristics

Feature	Description
🚦 Rate Limiting	Filter events based on per-template rate limits
📋 Policy-Based	Apply local or centralized filtering policies
💰 Cost Control	Reduce Splunk ingestion volume and costs
📤 UF Handles Delivery	Splunk UF manages buffering, retries, timeouts

File Relay Pattern

This module configures a file relay where Fluent Bit reads application logs, passes them through a 10x sidecar process for regulation, then writes only the filtered events to a folder that Splunk UF monitors. Noisy/repetitive events are dropped before reaching Splunk.

Install

Nix/OSX

See the Quickstart below or the Log10x Edge Regulator run instructions

Quickstart

Step 1: Set Environment Variables

export TENX_MODULES=/etc/tenx/modules
export FOLDER_A=/var/log/app
export FOLDER_B=/var/log/processed

Step 2: Configure Fluent Bit

fluent-bit.conf

[SERVICE]
    Flush        1
    Log_Level    info

[INPUT]
    Name         tail
    Path         ${FOLDER_A}/*.log
    Tag          app.logs

# Include 10x regulator
@INCLUDE ${TENX_MODULES}/pipelines/run/modules/input/forwarder/fluentbit/conf/tenx-regulate.conf

# Write filtered events to Folder B for Splunk UF
[OUTPUT]
    Name         file
    Match        *
    Path         ${FOLDER_B}
    Format       plain

Step 3: Configure Splunk UF

inputs.conf

[monitor://${FOLDER_B}]
index = main
sourcetype = app_logs

Step 4: Run

fluent-bit -c fluent-bit.conf

Compare the line counts in Folder A vs Folder B to see regulation in action.

This module is defined in regulate/module.yaml.