Skip to content

FAQ

The Receiver is the execution arm of the 10x pipeline. It decides a per-pattern action for every pattern it sees: pass, sample, compact, tier_down, offload, or drop. An AI agent picks the action per service through the log10x MCP (the configure_engine tool); the engine enforces it, and the decision travels as a config change through the GitOps repo. The Receiver runs as a sidecar alongside your forwarder.

Overview

What is the Receiver and how does it control costs

The Receiver decides a per-pattern action for every pattern it sees:

  • pass: forward unchanged.
  • sample: forward a rate-limited share against a per-pattern budget. Uses automatic message enrichment to identify event types by their symbol identity and calculate ingestion cost per event (bytes × $/GB). When an event type exceeds its share of the budget, the Receiver samples it down, in real time at the source, before expensive data reaches your SIEM. A severity boost keeps ERROR and WARN events more likely to be retained than DEBUG noise.
  • compact: replace repeated lines with an encoded form the destination expands at query time, with no dashboard or query changes. Lossless only on Splunk, self-hosted Elasticsearch/OpenSearch, and ClickHouse with the expand plugin installed (Splunk, Elasticsearch). It is a no-op on managed/SaaS destinations (Datadog, CloudWatch, Azure, GCP, Sumo, managed Elasticsearch).
  • tier_down: tag the pattern for a cheaper storage tier the destination enforces (Datadog Flex, CloudWatch IA).
  • offload: route the pattern to customer-owned object storage (S3, GCS, Azure Blob) instead of the destination.
  • drop: stop forwarding the pattern.

An AI agent picks the action per service through the log10x MCP (the configure_engine tool), which turns a target percent or budget into a per-pattern action set carried as a cap/action CSV. The CSV lands in the config repo through a GitOps PR and hot-reloads on the next pull. The sample action runs in per-node budget mode or mute file mode, a declarative field-set keyed cap file committed to git and pulled via GitOps.

Log10x is normally driven by an AI agent (Claude, or a model the customer brings) through the log10x MCP server, which installs, configures, and queries via MCP tools. The manual steps in these docs are the same operations without an agent.

How do budget policies work

Configuration via YAML file or Console UI. The rate receiver tracks per-event-type spend using automatic symbol identity enrichment. You configure:

  • Budget per hour, target ingestion cost rate (e.g., $1.50/hour)
  • Max share per event type, prevents any single event type from dominating the budget (e.g., 20%)
  • Severity boost, ERROR events get higher retention probability than DEBUG noise

Example policy: cap total ingestion at $500/hour, cap any single event type at 20% of the budget, prioritize ERROR over INFO.

When an event type exceeds its max share, the Receiver samples it down proportionally. For multi-app environments (Kubernetes), cap per-app budgets using the container name field, scaling replicas doesn't bypass limits.

What happens to logs when budget limits are reached (sample action)

The rate receiver applies cost-based sampling. Each event is either retained or dropped based on how much its event type has spent relative to the budget:

  • Under budget, all events flow through normally
  • Event type over its max share, that type gets sampled down proportionally (e.g., an event type consuming 60% of the budget with a 20% cap gets sampled to ~33%)
  • Severity boost, ERROR events receive a higher retention multiplier, making them far more likely to survive sampling than DEBUG noise

The result: noisy event types are automatically throttled while critical events are preserved. The Receiver exports metrics tracking what each pattern's action did, so policies can be tuned over time. Routed-away events carry the label routeState (for example routeState="drop").

Configuration

Can I set different budgets for different event types

Yes. The rate receiver tracks spend per event type using configurable field sets. You can scope budgets by:

  • Message pattern, each distinct event type gets its own budget share
  • K8s container name, cap total spend per application, regardless of pod replica count
  • Both combined, cap per event type per application for fine-grained control

The severity boost ensures higher-severity events (ERROR, WARN) are more likely to be retained when sampling kicks in, without requiring separate budget tiers.

How does the Receiver protect critical events

The rate receiver uses two mechanisms that naturally protect critical events:

  • Severity boost, ERROR and WARN events receive a higher retention multiplier during sampling. Even when their event type is over budget, critical-severity events are far more likely to be retained than DEBUG noise
  • Max share targeting, sampling only kicks in when a specific event type exceeds its configured share of the budget (e.g., 20%). Low-volume event types, which security and authentication logs typically are, stay well under their share and pass through unaffected

The Receiver targets noisy, high-volume event types that dominate your budget, not broad categories. Security events that don't spike beyond their budget share flow through without sampling.

How do I monitor budget consumption

The Receiver publishes cost metrics per event type, volume, spend rate, and per-action counts. Three ways to consume them:

  • Cost dashboards, the metrics feed your time-series backend (Prometheus, Datadog, Elastic, and others), where budget consumption, processed volume, and cost trends per event type are charted in your own tooling
  • Prometheus Metrics API, standard REST endpoints for querying all cost and action metrics programmatically via /query, /query_range, and /series
  • Native metric outputs, export to Prometheus, Datadog, CloudWatch, SignalFx, or any compatible platform for custom dashboards and alerts
How do I configure priority tiers to always forward critical events

Use the levelBoost configuration to force minimum retention floors per severity level. This creates a priority tier system where critical events are very likely to pass through even when their event type is over budget.

How it works: The Receiver calculates a retention threshold based on budget spend. With levelBoost, if any event's severity level multiplier forces a higher minimum threshold, that event is retained at top priority. For example, ERROR: 100 means ERROR events require random() < (minThreshold * 100) to be dropped, making ERROR drops extremely unlikely at typical minThreshold values.

Helm configuration example:

# values.yaml for Receiver Helm chart
receiver:
  config:
    rateReceiverBudgetPerHour: "$1.50"
    rateReceiverMaxSharePerFieldSet: "0.20"
    # Severity boost: guarantee higher priority for critical events
    rateReceiverLevelBoost:
      TRACE: 0.25
      DEBUG: 0.5
      INFO: 1.0
      WARN: 1.5
      ERROR: 10.0    # High boost ensures ERRORs pass through
      CRITICAL: 50.0  # Highest priority for security/outage alerts

YAML config example (non-Helm):

# config.yaml for the Receiver
tenx-receiver:
  mode: local
  threshold:
    budgetPerHour: "$1.50"
    maxSharePerFieldSet: 0.20
    levelBoost:
      TRACE: 0.25      # Debug noise: aggressive sampling
      DEBUG: 0.5       # Development logs: light sampling
      INFO: 1.0        # Normal events: baseline threshold
      WARN: 1.5        # Warnings: higher priority
      ERROR: 10.0      # Production errors: very high retention priority
      CRITICAL: 50.0   # System critical: top retention priority

Real-world scenarios:

  1. API Gateway (all errors must be captured): - Budget: $2/hour across 50 microservices - ERROR boost: 100 (a high ERROR boost makes API errors extremely likely to pass) - DEBUG boost: 0.1 (aggressive sampling on verbose request logs) - Result: ~95% reduction on debug, errors retained at top priority

  2. Background Job Queue (low signal, high volume): - Budget: $0.50/hour - INFO boost: 0.25 (job completion messages sampled 75%) - ERROR boost: 20.0 (failed jobs retained at top priority) - CRITICAL boost: 100.0 (job-queue-down events kept at top priority) - Result: Cost control on routine ops, full visibility on failures

  3. Kubernetes cluster (health checks vs. pod crashes): - Budget: $5/hour per node - DEBUG boost: 0.1 (health check events: 90% dropped) - WARN boost: 2.0 (pod warnings: light sampling) - ERROR boost: 100.0 (pod failures and crashes: highest retention priority) - Result: 95% reduction on health checks, pod issues retained at top priority

Configuration interaction with budget:

The Receiver applies: retentionThreshold = max(calculatedThreshold, minThreshold * boost)

This means: - Events with high boost values are always more likely to pass than events with low boost values, regardless of budget pressure - Budget still applies as the upper limit, you won't exceed your per-hour cap - Lower boost values get sampled first when over budget; higher boost values get sampled last - A boost: 100 on ERROR doesn't mean "ignore budget", it means "ERROR events compete at 100x priority vs. DEBUG events at 0.25x priority"

When should I use the sample action vs the compact action

sample and compact are two of the per-pattern actions, applied independently per pattern.

sample compact
Loss Lossy, the over-budget share of a pattern is dropped Lossless on supported destinations, each event replaced with a compact wire-form the expand plugin restores at query time
Downstream requirements None Self-hosted Splunk, Elasticsearch/OpenSearch, or ClickHouse with the expand plugin installed
Volume reduction Set by the per-pattern sample rate Typically 50-80% on supported destinations (Splunk, self-hosted Elasticsearch, ClickHouse), varying with how repetitive the pattern is; a no-op on managed/SaaS destinations
Risk profile Higher, dropped events are gone; safe defaults = deny Lower, events survive full round-trip, queryable as normal
Typical trigger A single pattern is over-budget; cap it at a sample rate Shipping volume is the bottleneck; shrink the wire format
What happens to events that the Receiver filters

Filtered events are not forwarded to your log analyzer. If you want a copy of events you can fetch back later, configure your forwarder to also route a copy of events to your own S3 bucket before the Receiver acts. Each forwarder has a native mechanism for duplicating the event stream, see per-forwarder S3 configuration.

Use Retriever to fetch the exact offloaded events from your own S3 bucket on demand; it returns the stored events without rehydration or SIEM re-ingest, at your S3 storage cost.

Can I use the Receiver with Retriever

Yes. Configure your forwarder to duplicate events, one copy goes to your own S3 bucket, the other goes through the Receiver (filtered or compact events to your SIEM). Hard-dropped events never reach S3. Retriever fetches selected offloaded events from your S3 bucket on demand.

This pairs SIEM cost control with on-demand fetch of the offloaded copies from your own S3 bucket. Events you route to S3 can be fetched back through Retriever on demand. Compact events stored in S3 are also queryable, Retriever expands them on the way out.

See the per-forwarder configuration for Fluent Bit, Fluentd, OTel Collector, and Logstash recipes.

Can I preview the impact before enabling actions

Yes. The Reporter app runs the same enrichment pipeline as the Receiver, severity classification, symbol identity, cost metrics, but does not filter or compact events. Deploy the Reporter first to see which event types incur the highest costs and what the actions would target. When the metrics confirm the expected behavior, enable the Receiver.

Integration & Deployment

Which log forwarders does the Receiver support

The Receiver integrates with all major log forwarders. Six are native, the install wizard automates each:

Splunk Universal Forwarder and Datadog Agent integrate through a Fluent Bit file-system handoff: Fluent Bit reads files from disk, the engine processes them, and the UF or agent tails the output folder.

Deployment: runs as a sidecar process alongside your forwarder so it can act on events in the critical log path before they ship. Kubernetes deployment via Helm chart (DaemonSet). Setup time: ~30 minutes.

Resource requirements: 512 MB heap + 2 threads handles 100+ GB/day per node. See Performance FAQ for sizing details and Kubernetes resource specs, and the deployment guide for per-forwarder configuration.

Can the Receiver reduce Kubernetes health check volume

Yes. Health checks, liveness probes, and pod lifecycle events are highly repetitive and can represent 20-40% of total log volume in K8s environments. 95% reduction is common for these event types.

How it works:

  1. Set the sample action with a budget cap for health check event types using rate sampling
  2. For the surviving events, set the compact action to losslessly compact them via templates
  3. Result: 95% reduction on health checks while preserving all pod metadata (namespace, pod name, container name, labels)

Failed health checks are retained at top priority and very likely to pass even when the event type is over budget. Works with Splunk Connect for Kubernetes, Fluentd, Fluent Bit, OpenTelemetry Collector.

How does the Receiver handle failures

Kubernetes runs 10x as a sidecar container next to the forwarder and manages its lifecycle. If the 10x process exits, crash, OOM kill, panic, the container exits and the kubelet restarts it automatically via the pod's default restartPolicy. The forwarder keeps running and buffers in-flight events while the sidecar restarts, so events are not dropped during the restart and unsampled traffic is held back.

Scenario Behavior
10x crash or OOM The 10x process exits and Kubernetes restarts the sidecar container automatically. Forwarder buffers events while the sidecar is down and reconnects on restart.
Volume exceeds 10x capacity Forwarder buffers locally; 10x catches up. Backpressure signals the forwarder to slow down.
Forwarder crashes Kubernetes restarts the forwarder container; the pod stays up and the sidecar keeps running.
Network interruption No effect, both containers share the pod's loopback network.
Downstream SIEM slow or unreachable Forwarder handles retries to the destination, 10x unaffected.
Rollback Single helm uninstall, forwarder continues unchanged, no data migration.
Next: Retriever