Skip to content

Receiver

The Receiver is the execution arm of the 10x pipeline. The rate regulator finds which patterns are over the byte cap set for their service, and an action decides what happens to that excess. The keep-everything actions (compact, offload, tier_down) cut cost without losing data; sample and drop, the Receiver's Filter modes, are lossy: they apply only to the share of a pattern over a cap the operator sets, and that share is dropped unless the operator chooses another action; pass leaves the pattern untouched. An AI agent picks the action per service through the log10x MCP (the configure_engine tool); the engine enforces it, and the decision travels as a config change through the GitOps repo. The Receiver runs sidecar or embedded depending on the forwarder: a separate sidecar container with Fluentd, Fluent Bit, Logstash, OTel Collector, and Vector; embedded in Filebeat (image swap); a file relay with Splunk Universal Forwarder and Datadog Agent.

Overview

What is the Receiver and how does it control costs

The Receiver decides an action per service, applied to the events a pattern spends above its cap:

  • pass: forward unchanged.
  • sample: forward a rate-limited share against a per-pattern budget. Uses automatic message enrichment to identify event types by their symbol identity and calculate ingestion cost per event (bytes × $/GB). When an event type exceeds its share of the budget, the Receiver samples it down, in real time at the source, before expensive data reaches your analyzer. A severity floor keeps a minimum share of ERROR and WARN events flowing even when their pattern is over its cap.
  • compact: replace repeated lines with an encoded form the destination expands at query time. In Splunk, classic dashboards keep their searches, alerts are compiled at save time, and a search in the search bar or a Dashboard Studio panel starts with the app's tx command. Lossless only on Splunk (up to 256 KB per event) and self-hosted Elasticsearch/OpenSearch with the expand plugin installed (Splunk, Elasticsearch). It is a no-op on managed/SaaS destinations (Datadog, CloudWatch, Azure, GCP, Sumo, managed Elasticsearch).
  • tier_down: tag the pattern for a cheaper tier the destination enforces (Datadog Flex, CloudWatch IA, Azure Monitor Basic/Auxiliary).
  • offload: route the pattern to customer-owned object storage (S3 or any S3-compatible bucket) instead of the destination.
  • drop: stop forwarding the pattern.

An AI agent picks the action per service through the log10x MCP (the configure_engine tool), which turns a target percent or budget into per-service caps and actions carried as cap and action CSV files. The files land in the config repo through a GitOps PR and hot-reload on the next pull. Per-pattern rates come from the mute file, a CSV keyed by pattern, committed to git and pulled the same way.

Log10x is normally driven by an AI agent (Claude, or a model the customer brings) through the log10x MCP server, which installs, configures, and queries via MCP tools. The manual steps in these docs are the same operations without an agent.

How do budget policies work

Configuration is a YAML file, written by hand or generated by the MCP configure_engine tool and delivered through GitOps. The rate receiver caps the bytes each pattern (symbol identity) may spend in its container over a rolling window. You configure:

  • A byte cap per pattern per container, fleet-wide or per container from a cap file; the MCP derives it from a monthly dollar budget
  • A share guard, a pattern below a minimum share of its container's volume (default 5%) is left alone
  • Severity floors, an over-cap pattern keeps at least a set share of each level (by default ERROR 50%, WARN 30%, INFO 10%)
  • An action per service for the excess: pass (the default), offload, tier_down, compact, sample or drop

Cost is reported per pattern; the cap itself is in bytes. Caps are scoped by the container name, so scaling replicas doesn't bypass them.

What happens to logs when budget limits are reached (sample action)

The rate receiver decides per event, on bytes:

  • At or below the pattern's cap, every event flows through
  • Over the cap, each level keeps its floor share and the rest takes the container's action (pass when none is set); nothing targets the excess back down to the cap line
  • Severity floors, ERROR events keep a higher share of the over-cap slice than DEBUG noise

The result: a noisy pattern is throttled while its errors keep flowing. The Receiver exports metrics tracking what each pattern's action did, so policies can be tuned over time. Routed-away events carry the label routeState (for example routeState="drop").

Configuration

Can I set different budgets for different event types

Yes. The rate receiver tracks spend per event type using configurable field sets. You can scope budgets by:

  • Message pattern, each distinct event type gets its own budget share
  • K8s container name, cap total spend per application, regardless of pod replica count
  • Both combined, cap per event type per application for fine-grained control

The severity floor keeps a minimum share of higher-severity events (ERROR, WARN) flowing when sampling kicks in, without requiring separate budget tiers.

How does the Receiver protect critical events

The rate receiver uses two mechanisms that naturally protect critical events:

  • Severity floors, ERROR and WARN events keep a higher minimum share of the over-cap slice. Even when their pattern is over its cap, critical-severity events keep flowing at a higher rate than DEBUG noise
  • Per-pattern caps, the regulator engages only on a pattern over its byte cap and above the share guard (default 5% of its container's volume). Low-volume patterns, which security and authentication logs typically are, stay under their cap and pass through unaffected

The Receiver targets noisy, high-volume patterns, not broad categories. Security events that stay under their cap flow through untouched.

How do I monitor budget consumption

The Receiver publishes cost metrics per event type, volume, spend rate, and per-action counts. Three ways to consume them:

  • Cost dashboards, the metrics feed your time-series backend (Prometheus, Datadog, Elastic, and others), where budget consumption, processed volume, and cost trends per event type are charted in your own tooling
  • Prometheus Metrics API, standard REST endpoints for querying all cost and action metrics programmatically via /query, /query_range, and /series
  • Native metric outputs, export to Prometheus, Datadog, CloudWatch, or any compatible platform for custom dashboards and alerts
How do I configure priority tiers to always forward critical events

Use severityFloors to set a minimum retention per severity level. The floor is an absolute fraction, not a multiplier, and it beats the cap: a pattern over its byte cap still keeps at least this share of each level, so high-severity events are never fully suppressed.

How it works: the rate regulator engages only once a pattern has spent more bytes than the cap set for its container. From then on, each event of that pattern is kept with the probability its level's floor gives, and whatever the floor does not keep takes the container's action, which is drop unless an action file names another. A level with no entry uses minRetentionThreshold (default 0.1). The same floor applies to mute-file entries, so a 0 mute never silences ERROR or FATAL.

Floor keys must match the level vocabulary the Level Classifier emits.

Config example:

# config.yaml for the Receiver
rateReceiver:
  fieldNames:
    - message_pattern
  containerField: container
  absoluteCap: 10485760        # 10 MB per pattern per container per window
  minRetentionThreshold: 0.1   # floor for any level not listed below
  severityFloors:
    - TRACE=0.05
    - DEBUG=0.05
    - INFO=0.1
    - WARN=0.3
    - ERROR=0.5                # half of an over-cap error pattern keeps flowing
    - CRITICAL=0.5

No cap ships by default, so absoluteCap or a per-container cap file is what turns the regulator on. For Kubernetes, the same settings are set through the Helm chart.

Real-world scenarios:

  1. API Gateway (errors matter most): - Cap: 10 MB per pattern per container per window - ERROR floor 0.5, DEBUG floor 0.05, so verbose request logs are thinned hardest and errors keep flowing

  2. Background Job Queue (low signal, high volume): - Cap: 2 MB per pattern per container per window - INFO floor 0.1 for job completion messages, CRITICAL floor 0.5 for job-queue-down events

  3. Kubernetes cluster (health checks vs. pod crashes): - Cap: 5 MB per pattern per container per window - DEBUG floor 0.05 for health check spam, ERROR floor 0.5 for pod failures and crashes

What the floor does not do:

  • It does not rank levels against each other. Each level's floor is read on its own, so raising ERROR does not thin DEBUG.
  • It applies only to a pattern already over its cap. Under the cap every event is kept whatever its level.
  • It sets a minimum, not a maximum. The floor decides what share of the excess survives; the container's action decides what happens to the rest.
When should I use the sample action vs the compact action

sample and compact are two of the actions, each applied to the events a pattern spends above its cap.

sample compact
Loss Lossy, the over-budget share of a pattern is dropped Lossless on supported destinations, each event replaced with a compact wire-form the expand plugin restores at query time
Downstream requirements None Self-hosted Splunk or Elasticsearch/OpenSearch with the expand plugin installed
Volume reduction Set by the per-pattern sample rate Set by how repetitive the pattern is, where the destination supports it (Splunk, self-hosted Elasticsearch/OpenSearch); a no-op on managed/SaaS destinations
Risk profile Higher, dropped events are gone; safe defaults = deny Lower, events survive full round-trip, queryable as normal
Typical trigger A single pattern is over-budget; cap it at a sample rate Shipping volume is the bottleneck; shrink the wire format
What happens to events that the Receiver filters

Filtered events are not forwarded to your log analyzer. If you want a copy of events you can fetch back later, configure your forwarder to also route a copy of events to your own S3 bucket before the Receiver acts. Each forwarder has a native mechanism for duplicating the event stream, see per-forwarder S3 configuration.

Use Retriever to fetch the exact offloaded events from your own S3 bucket on demand; it returns the stored events without rehydration or re-ingest, at your S3 storage cost.

Can I use the Receiver with Retriever

Yes. Configure your forwarder to duplicate events, one copy goes to your own S3 bucket, the other goes through the Receiver (filtered or compact events to your analyzer). Hard-dropped events never reach S3. Retriever fetches selected offloaded events from your S3 bucket on demand.

This pairs log analyzer cost control with on-demand fetch of the offloaded copies from your own S3 bucket. Events you route to S3 can be fetched back through Retriever on demand. Compact events stored in S3 are also queryable, Retriever expands them on the way out.

See the per-forwarder configuration for Fluent Bit, Fluentd, OTel Collector, and Logstash recipes.

Can I preview the impact before enabling actions

Yes. The Reporter app runs the same enrichment pipeline as the Receiver, severity classification, symbol identity, cost metrics, but does not filter or compact events. Deploy the Reporter first to see which event types incur the highest costs and what the actions would target. When the metrics confirm the expected behavior, enable the Receiver.

Integration & Deployment

Which log forwarders does the Receiver support

The Receiver integrates with all major log forwarders. Six are native, the install wizard automates each:

Splunk Universal Forwarder and Datadog Agent integrate through a Fluent Bit file-system handoff: Fluent Bit reads files from disk, the engine processes them, and the UF or agent tails the output folder.

Deployment: runs in-path with your forwarder, sidecar or embedded depending on the forwarder, so it can act on events before they ship to the analyzer. Kubernetes deployment via Helm chart; the workload type follows the forwarder chart (DaemonSet for most, StatefulSet for Logstash). Setup time: ~30 minutes.

Resource requirements: set the heap explicitly and the container memory limit above it, for example a 512m heap in a 768Mi limit. See Node counting for sizing details and Kubernetes resource specs, and the deployment guide for per-forwarder configuration.

Can the Receiver reduce Kubernetes health check volume

Yes. Health checks, liveness probes, and pod lifecycle events are highly repetitive, so a per-pattern cap holds each one to its budget.

How it works:

  1. Set the sample action with a budget cap for health check event types using rate sampling
  2. For the surviving events, set the compact action to losslessly compact them via templates
  3. Result: the health check patterns stay within their budget, and every pod metadata field (namespace, pod name, container name, labels) is preserved

Failed health checks are retained at top priority and very likely to pass even when the event type is over budget. Works with Splunk Connect for Kubernetes, Fluentd, Fluent Bit, OpenTelemetry Collector.

How does the Receiver handle failures

With sidecar forwarders (Fluentd, Fluent Bit, Logstash, OTel Collector, Vector), Kubernetes runs 10x as a separate container next to the forwarder and manages its lifecycle. If the 10x process exits, crash, OOM kill, panic, the container exits and the kubelet restarts it automatically via the pod's default restartPolicy. The forwarder keeps running and buffers in-flight events while the sidecar restarts, so events are not dropped during the restart and unsampled traffic is held back. With the embedded forwarder (Filebeat), 10x runs inside the forwarder's own container, so the two share one lifecycle and restart together.

Scenario Behavior
10x crash or OOM The 10x process exits and Kubernetes restarts the sidecar container automatically. Forwarder buffers events while the sidecar is down and reconnects on restart.
Volume exceeds 10x capacity Forwarder buffers locally; 10x catches up. Backpressure signals the forwarder to slow down.
Forwarder crashes Kubernetes restarts the forwarder container; the pod stays up and the sidecar keeps running.
Network interruption No effect, both containers share the pod's loopback network.
Downstream analyzer slow or unreachable Forwarder handles retries to the destination, 10x unaffected.
Rollback Single helm uninstall, forwarder continues unchanged, no data migration.