Skip to content

Edge Regulator

The Edge Regulator app filters noisy events from log forwarders, reducing storage and log analytics costs and preventing over-billing.

Regulators ensure predictable costs and free budgets to focus on analyzing meaningful events.

Edge Regulator Console

Regulate Costs

Filter out spiking or anomalous events using automatic event symbol identity to prevent unexpected costs and over-billing, while allowing valuable events to flow through log forwarders.

Observe More For Less

Avoid excessive billing and free up resources for capturing and analyzing the most valuable, insight-rich events. Aggregate and publish high-volume, 'noisy' events as lightweight metrics.

Central Cost Control

Control costs at the forwarder level, ensuring no log event type exceeds a target cost threshold. For multi-app environments (Kubernetes), regulate per-app budgets across all pods—scaling replicas doesn't bypass limits. Use environment-wide GitOps policies, driven by the reporter app's cost insight metrics, managing 'noisy' telemetry and preventing over-billing.

Workflow

The Edge Regulator app processes events from a variety of log forwarders, such as Fluentd, Fluent Bit, Filebeat, and Logstash. Configure the app to process all or a subset of the events, allowing for targeted analysis and event regulation.

graph LR
    A["<div style='font-size: 14px;'>🚙 Forwarder</div><div style='font-size: 10px; text-align: center;'>Sidecar Process</div>"] --> B["<div style='font-size: 14px;'>📡 Receive</div><div style='font-size: 10px; text-align: center;'>Stream Events</div>"]
    B --> C["<div style='font-size: 14px;'>🔄 Transform</div><div style='font-size: 10px; text-align: center;'>into TenXObjects</div>"]
    C --> D["<div style='font-size: 14px;'>🎁 Enrich</div><div style='font-size: 10px; text-align: center;'>Add Context</div>"]
    D --> E["<div style='font-size: 14px;'>🚦 Regulate</div><div style='font-size: 10px; text-align: center;'>Filter Events</div>"]
    E --> F["<div style='font-size: 14px;'>📤 Output</div><div style='font-size: 10px; text-align: center;'>Write to Forwarder</div>"]

    classDef deploy fill:#7c3aed88,stroke:#6d28d9,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef receive fill:#9333ea88,stroke:#7c3aed,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef transform fill:#2563eb88,stroke:#1d4ed8,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef enrich fill:#059669,stroke:#047857,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef regulate fill:#dc2626,stroke:#b91c1c,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef output fill:#ea580c88,stroke:#c2410c,color:#ffffff,stroke-width:2px,rx:8,ry:8

    class A deploy
    class B receive
    class C transform
    class D enrich
    class E regulate
    class F output

🚙 Forwarder: Runs 10x as a sidecar process to log forwarders for real-time event analysis

📡 Receive: Read events continuously from log forwarders via IPC

🔄 Transform: Structures log events into well-defined TenXObjects

🎁 Enrich: Applies enrichment rules to augment TenXObjects with intelligent context

📈 Report: Publishes cost insight metrics for visualization and alerting

🚦 Regulate: Filters events using local or environment policies to prevent over-billing

📤 Output: Writes regulated events back to forwarder to ship to destination analyzers

Architecture

Edge regulators execute as a forwarder sidecar to filter ‘noisy’ events before they ship to a log analyzer.

Without 10x, forwarders ship 'noisy' log events to target analyzers, consuming disproportionate resources and causing over-billing.

Image title
❌ Forwarders ship log events with no active control over costs.

Local regulators filter 'noisy' events before forwarding, using symbol identities to limit event type volumes (e.g., 10Mb/10sec) and prevent over-billing.

Image title
✅ Edge Regulators use cost thresholds to prevent over-billing.

Policy tasks query cost metrics generated by Edge Reporters to update policy lookups on GitHub. Edge regulators pull these policies to prevent over-billing from environment-wide 'noisy' events.

Image title
✅ Policy Regulators use cost insights to prevent over-billing.

Configuration

Concrete rate regulator configurations for common scenarios. Edit these in your config.yaml.

Cap any single event type at 20% of total spend. Use the Level Classifier to enrich events with severity levels, so ERROR events have a higher minimum retention floor than DEBUG and critical events survive throttling.

rateRegulator:
  fieldNames:
    - symbolMessage
  budgetPerHour: 1.50
  ingestionCostPerGB: 1.5          # Splunk Cloud
  maxSharePerFieldSet: 0.2         # No event type exceeds 20%
  minRetentionThreshold: 0.1       # At least 10% retained when over budget
  levelBoost:
    - TRACE=0.25
    - DEBUG=0.5
    - INFO=1
    - WARN=1.5
    - ERROR=2
    - FATAL=3

With minRetentionThreshold: 0.1 and levelBoost, the minimum retention floor when over budget is: DEBUG = 5% (0.1 × 0.5), INFO = 10% (0.1 × 1), ERROR = 20% (0.1 × 2). Under budget, all events pass through — boost only affects the floor.

Prevent any single app from exceeding 20% of the budget across all its pods. Uses k8s container name for stable aggregation across replicas.

rateRegulator:
  fieldNames:
    - symbolMessage
    - container                    # Same name across all pod replicas
  budgetPerHour: 1.50
  ingestionCostPerGB: 2.50         # Datadog
  maxSharePerFieldSet: 0.2

Each (event type × app) combination gets its own 20% cap. Scaling from 1 to 10 pods doesn't bypass limits because container name is stable across replicas.

Use cluster-wide spend data from the Edge Policy app for coordinated throttling across all nodes.

rateRegulator:
  fieldNames:
    - symbolMessage
  budgetPerHour: 1.50
  ingestionCostPerGB: 1.5
  maxSharePerFieldSet: 0.2
  lookup:
    file: /etc/log10x/config/data/sample/policy/policy.csv
    retain: 300000                 # 5 minutes — fall back to local if stale

The policy module periodically queries Prometheus for cluster-wide event costs and writes a lookup CSV. Each node reads this file for coordinated budget decisions. If the file becomes stale, the regulator falls back to local mode.

Safety & Reliability

The Edge Regulator runs as a sidecar alongside your log forwarder with fail-open design — if the regulator crashes or stops, your logs continue flowing normally at full volume to your analyzer.

Key topics:

See the Edge FAQ for complete operational details, capacity planning, and deployment guidance.

Next: Run