Skip to content

Edge Regulator

The Edge Regulator app filters noisy events from log forwarders, reducing storage and log analytics costs and preventing over-billing.

Regulators ensure predictable costs and free budgets to focus on analyzing meaningful events.

Edge architecture — forwarders route events through the 10x sidecar before reaching your analyzer

10x Sidecar — Reporter, Regulator, and Optimizer savings per app

Per-pattern filtering — the Regulator filtered 54% of events before reaching the analyzer, saving $310K

Regulate Costs

Filter out spiking or anomalous events using automatic event symbol identity to prevent unexpected costs and over-billing, while allowing valuable events to flow through log forwarders.

Observe More For Less

Avoid excessive billing and free up resources for capturing and analyzing the most valuable, insight-rich events. Aggregate and publish high-volume, 'noisy' events as lightweight metrics.

Central Cost Control

Control costs at the forwarder level, ensuring no log event type exceeds a target cost threshold. For multi-app environments (Kubernetes), regulate per-app budgets across all pods—scaling replicas doesn't bypass limits. Use environment-wide GitOps policies, driven by the reporter app's cost insight metrics, managing 'noisy' telemetry and preventing over-billing.

Pair with Storage Streamer

For environments that require full event retention alongside cost control, archive all events to S3 before regulation. The regulator filters what reaches your SIEM, while Storage Streamer keeps everything queryable in S3. See per-forwarder archival configuration for setup.

Workflow

The Edge Regulator app processes events from a variety of log forwarders, such as Fluentd, Fluent Bit, Filebeat, and Logstash. Configure the app to process all or a subset of the events, allowing for targeted analysis and event regulation.

graph LR
    A["<div style='font-size: 14px;'>🚙 Forwarder</div><div style='font-size: 10px; text-align: center;'>Sidecar Process</div>"] --> B["<div style='font-size: 14px;'>📡 Receive</div><div style='font-size: 10px; text-align: center;'>Stream Events</div>"]
    B --> C["<div style='font-size: 14px;'>🔄 Transform</div><div style='font-size: 10px; text-align: center;'>into TenXObjects</div>"]
    C --> D["<div style='font-size: 14px;'>🎁 Enrich</div><div style='font-size: 10px; text-align: center;'>Add Context</div>"]
    D --> E["<div style='font-size: 14px;'>🚦 Regulate</div><div style='font-size: 10px; text-align: center;'>Filter Events</div>"]
    E --> F["<div style='font-size: 14px;'>📤 Output</div><div style='font-size: 10px; text-align: center;'>Write to Forwarder</div>"]

    classDef deploy fill:#7c3aed88,stroke:#6d28d9,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef receive fill:#9333ea88,stroke:#7c3aed,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef transform fill:#2563eb88,stroke:#1d4ed8,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef enrich fill:#059669,stroke:#047857,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef regulate fill:#dc2626,stroke:#b91c1c,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef output fill:#ea580c88,stroke:#c2410c,color:#ffffff,stroke-width:2px,rx:8,ry:8

    class A deploy
    class B receive
    class C transform
    class D enrich
    class E regulate
    class F output

🚙 Forwarder: Runs 10x as a sidecar process to log forwarders for real-time event analysis

📡 Receive: Read events continuously from log forwarders via IPC

🔄 Transform: Structures log events into well-defined TenXObjects

🎁 Enrich: Applies enrichment rules to augment TenXObjects with intelligent context

📈 Report: Publishes cost insight metrics for visualization and alerting

🚦 Regulate: Filters events using local or environment policies to prevent over-billing

📤 Output: Writes regulated events back to forwarder to ship to destination analyzers

Architecture

Edge regulators execute as a forwarder sidecar to filter ‘noisy’ events before they ship to a log analyzer.

Without 10x, forwarders ship 'noisy' log events to target analyzers, consuming disproportionate resources and causing over-billing.

Architecture diagram: Log forwarders ship all events to log analyzers with no active cost control
❌ Forwarders ship log events with no active control over costs.

Local regulators filter 'noisy' events before forwarding, using symbol identities to limit event type volumes (e.g., 10Mb/10sec) and prevent over-billing.

Architecture diagram: Edge Regulator sidecar applies per-event-type rate limits to filter noisy events before forwarding to log analyzers
✅ Edge Regulators use cost thresholds to prevent over-billing.

Policy tasks query cost metrics generated by Edge Reporters to update policy lookups on GitHub. Edge regulators pull these policies to prevent over-billing from environment-wide 'noisy' events.

Architecture diagram: Policy tasks query cost metrics from Edge Reporters to update regulation policies on GitHub, which Edge Regulators pull to prevent environment-wide over-billing
✅ Policy Regulators use cost insights to prevent over-billing.

Safety & Reliability

The Edge Regulator runs as a sidecar alongside your log forwarder with fail-open design — if the regulator crashes or stops, your logs continue flowing normally at full volume to your analyzer.

Topic Detail
Fail-open design Logs continue flowing if 10x goes down
Backpressure handling Disk buffering prevents data loss during spikes
Resource requirements 512MB heap + 2 threads handles 100+ GB/day
Rollback helm uninstall takes ~1 minute, no data loss

See the Edge FAQ for complete operational details, capacity planning, and deployment guidance.

Next: Test