Regulate

Read events from application logs to transform into typed TenXObjects and filter using local/centralized regulator policies, before the Datadog Agent ships them to Datadog. This module is a component of the Edge Regulator app.

Architecture

graph LR
    A["<div style='font-size: 14px;'>📝 App Logs</div><div style='font-size: 10px;'>Folder A</div>"] --> B["<div style='font-size: 14px;'>🔧 Fluent Bit</div><div style='font-size: 10px;'>tail input</div>"]
    B --> C["<div style='font-size: 14px;'>⚡ 10x Regulator</div><div style='font-size: 10px;'>filter policy</div>"]
    C --> D["<div style='font-size: 14px;'>📂 Processed</div><div style='font-size: 10px;'>Folder B</div>"]
    D --> E["<div style='font-size: 14px;'>📤 DD Agent</div><div style='font-size: 10px;'>monitors B</div>"]
    E --> F["<div style='font-size: 14px;'>🐶 Datadog</div><div style='font-size: 10px;'>Intake API</div>"]

    classDef logs fill:#2563eb,stroke:#1d4ed8,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef fluentbit fill:#ea580c,stroke:#c2410c,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef engine fill:#7c3aed,stroke:#6d28d9,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef processed fill:#0891b2,stroke:#0e7490,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef agent fill:#632CA6,stroke:#4F2684,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef datadog fill:#16a34a,stroke:#15803d,color:#ffffff,stroke-width:2px,rx:8,ry:8

    class A logs
    class B fluentbit
    class C engine
    class D processed
    class E agent
    class F datadog

Data Flow

• 📝 App Logs (Folder A) - Application writes ALL logs to original location
• 🔧 Fluent Bit - Reads from Folder A, passes events to 10x sidecar
• ⚡ 10x Regulator - Applies rate/policy-based filtering, drops noisy events
• 📂 Processed (Folder B) - Only FILTERED events written to new location
• 📤 DD Agent - Monitors Folder B, forwards reduced volume to Datadog
• 🐶 Datadog - Receives filtered events (reduced volume)

Key Characteristics

Feature	Description
🚦 Rate Limiting	Filter events based on per-template rate limits
📋 Policy-Based	Apply local or centralized filtering policies
💰 Cost Control	Reduce Datadog ingestion volume and costs
📤 Agent Handles Delivery	Datadog Agent manages buffering, retries, metadata enrichment

File Relay Pattern

This module configures a file relay where Fluent Bit reads application logs, passes them through a 10x sidecar process for regulation, then writes only the filtered events to a folder that the Datadog Agent monitors. Noisy/repetitive events are dropped before reaching Datadog.

Install

Nix/OSX

See the Quickstart below or the Log10x Edge Regulator run instructions

Quickstart

Step 1: Set Environment Variables

export TENX_MODULES=/etc/tenx/modules
export FOLDER_A=/var/log/app
export FOLDER_B=/var/log/processed

Step 2: Configure Fluent Bit

fluent-bit.conf

[SERVICE]
    Flush        1
    Log_Level    info

[INPUT]
    Name         tail
    Path         ${FOLDER_A}/*.log
    Tag          app.logs

# Include 10x regulator
@INCLUDE ${TENX_MODULES}/pipelines/run/modules/input/forwarder/fluentbit/conf/tenx-regulate.conf

# Write filtered events to Folder B for Datadog Agent
[OUTPUT]
    Name         file
    Match        *
    Path         ${FOLDER_B}
    Format       plain

Step 3: Configure Datadog Agent

datadog.yaml (logs_config section)

logs_enabled: true

conf.d/tenx.d/conf.yaml

logs:
  - type: file
    path: ${FOLDER_B}/*.log
    service: myapp
    source: myapp

Step 4: Run

fluent-bit -c fluent-bit.conf

Compare the line counts in Folder A vs Folder B to see regulation in action.

This module is defined in regulate/module.yaml.