Regulator

The Regulator app is the execution arm of the 10x pipeline. Two modes, one app:

Filter (lossy): drop events matching a rule — up to 80% volume reduction. Safe defaults are deny; explicit allow required. See per-node budget mode and mute file mode.
Compact mode (lossless): replace events with a compact wire-form that the downstream SIEM plugin expands at query time. 50–80% reduction (64% on K8s OTel logs) with no dashboard or query changes. Requires the expand plugin installed in Splunk or Elasticsearch. No expand plugin = no value.

Regulators ensure predictable costs and free budgets to focus on analyzing meaningful events.

MCP can propose Regulator configs for you

Ask the MCP Server "cap the top cost driver" and it generates a Filter-mode mute file entry (or Compact-mode config if your SIEM supports the expand plugin) scoped to the specific pattern. Review the diff, commit to git, the Regulator picks it up.

When to use which mode

	Filter	Compact
Loss	Lossy — events matching the rule are dropped	Lossless — every event preserved in compact form
Downstream requirements	None	Expand plugin installed in Splunk / Elasticsearch
Volume reduction	Up to 80%	50–80% (64% typical on K8s OTel logs)
Risk profile	Higher — dropped events are gone; safe defaults = deny	Lower — survives full round-trip, queryable as normal
Typical trigger	A single pattern is over-budget; cap it at a sample rate	Shipping volume is the bottleneck; shrink the wire format
Pair with	Reporter (to identify what to filter)	Storage Streamer (to archive to S3 in compact form)

Compact mode (was Optimizer)

Compact mode was previously a separate app (@apps/edge/optimizer). It is now a feature of the regulator: both are the execution arm, both are commanded via GitOps, both operate on stable pattern identity. The former optimizer's deploy/run content is merged into the regulator's deploy and run pages. For SIEM-side plugin install, see the Splunk and Elasticsearch pages.

ArchitectureSidecarPer-Pattern Savings

Test Deploy (Helm) Explore Live Demo

Regulate Costs

Filter out spiking or anomalous events using automatic event symbol identity to prevent unexpected costs and over-billing, while allowing valuable events to flow through log forwarders.

Observe More For Less

Avoid excessive billing and free up resources for capturing and analyzing the most valuable, insight-rich events. Aggregate and publish high-volume, 'noisy' events as lightweight metrics.

Central Cost Control

Control costs at the forwarder level, ensuring no log event type exceeds a target cost threshold. For multi-app environments (Kubernetes), regulate per-app budgets across all pods—scaling replicas doesn't bypass limits. Use environment-wide GitOps policies, driven by the reporter app's cost insight metrics, managing 'noisy' telemetry and preventing over-billing.

Pair with Storage Streamer

For environments that require full event retention alongside cost control, archive all events to S3 before regulation. The regulator filters what reaches your SIEM, while Storage Streamer keeps everything queryable in S3. See per-forwarder archival configuration for setup.

Workflow

The Regulator app processes events from a variety of log forwarders, such as Fluentd, Fluent Bit, Filebeat, and Logstash. Configure the app to process all or a subset of the events, allowing for targeted analysis and event regulation.

graph LR
    A["<div style='font-size: 14px;'>🚙 Forwarder</div><div style='font-size: 10px; text-align: center;'>Sidecar Process</div>"] --> B["<div style='font-size: 14px;'>📡 Receive</div><div style='font-size: 10px; text-align: center;'>Stream Events</div>"]
    B --> C["<div style='font-size: 14px;'>🔄 Transform</div><div style='font-size: 10px; text-align: center;'>into TenXObjects</div>"]
    C --> D["<div style='font-size: 14px;'>🎁 Enrich</div><div style='font-size: 10px; text-align: center;'>Add Context</div>"]
    D --> E["<div style='font-size: 14px;'>🚦 Regulate</div><div style='font-size: 10px; text-align: center;'>Filter Events</div>"]
    E --> F["<div style='font-size: 14px;'>📤 Output</div><div style='font-size: 10px; text-align: center;'>Write to Forwarder</div>"]

    classDef deploy fill:#7c3aed88,stroke:#6d28d9,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef receive fill:#9333ea88,stroke:#7c3aed,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef transform fill:#2563eb88,stroke:#1d4ed8,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef enrich fill:#059669,stroke:#047857,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef regulate fill:#dc2626,stroke:#b91c1c,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef output fill:#ea580c88,stroke:#c2410c,color:#ffffff,stroke-width:2px,rx:8,ry:8

    class A deploy
    class B receive
    class C transform
    class D enrich
    class E regulate
    class F output

🚙 Forwarder: Runs 10x as a sidecar process to log forwarders for real-time event analysis

📡 Receive: Read events continuously from log forwarders via IPC

🔄 Transform: Structures log events into well-defined TenXObjects

🎁 Enrich: Applies enrichment rules to augment TenXObjects with intelligent context

📈 Report: Publishes cost insight metrics for visualization and alerting

🚦 Regulate: Filters events via per-node budget sampling or a declarative field-set mute file to prevent over-billing

📤 Output: Writes regulated events back to forwarder to ship to destination analyzers

Architecture

The Regulator executes as a forwarder sidecar to filter ‘noisy’ events or losslessly compact survivors before they ship to a log analyzer.

No 10x Per-Node Budget Mute File (GitOps)

Without 10x, forwarders ship 'noisy' log events to target analyzers, consuming disproportionate resources and causing over-billing.

Architecture diagram: Log forwarders ship all events to log analyzers with no active cost control — Forwarders ship log events with **no active control over costs**.

Forwarders ship log events with **no active control over costs**.

Regulators filter 'noisy' events before forwarding, using symbol identities to track per-event-type spend against a local hourly budget and probabilistically shed events that push any one pattern over its share. Simple, autonomous, no coordination. See per-node budget mode.

Architecture diagram: Regulator sidecar applies per-event-type rate limits to filter noisy events before forwarding to log analyzers — **Regulators** use per-node budgets to prevent over-billing.

**Regulators** use per-node budgets to prevent over-billing.

A declarative file keyed by the joined rateRegulatorFieldNames values (e.g. symbolMessage, container) caps specific patterns with an explicit sample rate and expiry. Operators (or an AI assistant via the Log10x MCP) append entries based on Reporter cost attribution, commit to git, and every regulator pulls the file on its next reload. Each mute is diff-reviewed, self-expires, and maps 1:1 to the field-sets the Reporter attributes cost to. See mute file mode.

Safety & Reliability

The Regulator runs as a sidecar alongside your log forwarder with fail-open design — if the regulator crashes or stops, your logs continue flowing normally at full volume to your analyzer.

Topic	Detail
Fail-open design	Logs continue flowing if 10x goes down
Backpressure handling	Disk buffering prevents data loss during spikes
Resource requirements	512MB heap + 2 threads handles 100+ GB/day
Rollback	`helm uninstall` takes ~1 minute, no data loss

See the Regulator FAQ for complete operational details, capacity planning, and deployment guidance.

Next: Test