Skip to content

Compact Receiver

Compact specific patterns' events into a template+values tuple, without redeploying the engine.

The compact receiver makes a per-event decision whether to emit via encode() (the pattern's template hash plus extracted variable values, typically 20–40× smaller than the original line) or preserve fullText. The decision is keyed by pattern identity (the same symbolMessage the Reporter attributes cost to), so an operator targets compaction at the same patterns surfaced in cost analysis.

Entries live in a CSV cap-file, typically committed to a git repo and edited by PR. The file is hot-reloaded on in-place writes; a merged PR takes effect within ~10 seconds, no pod restart.

Per-pattern caps

A cap-file declares, for each pattern, whether its events are compacted or preserved. Patterns not listed fall back to compactReceiverDefault.

File format, CSV with a header row, keyed by the fields named in compactReceiverFieldNames joined with _ (default: [symbolMessage]):

fieldSet,value
<fieldSet>,<true|false>[:<untilEpochSec>][:<reason>]
  • fieldSet, the event fields named by compactReceiverFieldNames joined with _. With the default [symbolMessage] this is the symbolMessage value for the pattern.
  • value, true compacts via encode(); false explicitly preserves fullText for this pattern (beats the default).
  • untilEpochSec, optional Unix-epoch (seconds) expiry. Past it the entry self-heals and the pattern falls back to compactReceiverDefault.
  • reason, optional free-text for audit. Must not contain commas (would break CSV parsing).

Example (with compactReceiverDefault: false):

fieldSet,value
payment_retry_gateway_timeout,true:1745856000:OPS-5123 spike
auth_audit_trail,false:1745856000:compliance keep verbose

The engine hot-reloads on in-place file writes (the gitops pattern); Kubernetes ConfigMap mounts don't reload because the CM swap is a symlink rename, not an in-place write.

Default policy

compactReceiverDefault sets the fallback decision when no cap-file entry matches:

  • false (default), preserve fullText. Cap-file entries opt specific patterns into compaction. Right when most traffic is already high-signal.
  • true, compact via encode(). Cap-file entries opt specific patterns out of compaction (e.g. audit/compliance patterns that must stay verbose). Right when most traffic is low-signal machinery and only a few patterns need full-text fidelity.

Flipping the default is a policy decision that affects every event and requires a pod rollout. Cap-file edits handle per-pattern exceptions without restart.

Wiring

compactReceiver:
  fieldNames: [symbolMessage]        # fields joined to form the lookup key
  default: false                     # fallback when no entry matches
  lookup:
    file: $=path("data/caps") + "/compact-cap.csv"
    retain: $=parseDuration("10m")
  • Setting lookup.file enables the per-pattern compaction path; leaving it unset preserves the pre-compact behavior (receive-only emits fullText; receiverOptimize=true compacts every event).

Tune these values in this config block, not via container environment variables. Any compactReceiver: key set here resolves to a launch argument at engine init and shadows a same-named env var, so env-only overrides are silently ignored. Edit the config (via a gitops PR) to change a value at runtime.

Config Files

To configure the compact receiver module, Edit these files.

Below is the default configuration from: compact/config.yaml.

Edit Online

Edit config.yaml Locally 

# 🔟❎ 'run' compact receiver configuration

# Per-pattern compaction decision via a declarative cap-file.
# The forwarder output stream emits encode() for listed patterns and
# fullText for everyone else (or vice versa via per-pattern opt-out).
# To learn more see https://doc.log10x.com/run/receive/compact/

# Set the 10x pipeline to 'run'
tenx: run

# =============================== Dependencies ================================

include: run/modules/receive/compact

# ============================ Compact Options ================================

compactReceiver:

  # 'fieldNames' defines the TenXObject fields joined with underscores to form
  # each event's lookup key. Defaults to [symbolMessage], matching the rate
  # receiver's rateReceiverFieldNames so MCP-authored entries address the same
  # pattern_hash the Reporter attributes cost to.
  fieldNames:
    - symbolMessage

  # 'default' is the fallback when no cap-file entry matches the event's
  # pattern. `false` (default) preserves fullText; `true` compacts via
  # encode(). The cap-file's per-pattern decision overrides this per event.
  default: false

  lookup:

    # 'file' specifies the per-pattern cap-file path. Hot-reloaded on in-place
    #  writes (the gitops pattern); Kubernetes ConfigMap mounts won't reload.
    #  Comment out to disable the cap-file (every event follows 'default' above).
    # file: $=path("data/sample/compact") + "/compact-cap.csv"

    # 'retain' specifies the period before the file is marked as stale.
    retain: $=parseDuration("10m")

Options

Specify the options below to configure the compact receiver:

Name Description
compactReceiverLookupFile Path or URL to a per-pattern compact decisions cap-file
compactReceiverLookupRetain Retention period for the per-pattern cap-file before it is considered stale
compactReceiverFieldNames Event fields joined to form each event's cap-file lookup key
compactReceiverDefault Fallback compaction decision when no cap-file entry applies

compactReceiverLookupFile

Path or URL to a per-pattern compact decisions cap-file.

Type Default
String ""

Points the engine at a cap-file whose entries decide whether each pattern's events are compacted via encode() or preserved as fullText. Patterns not listed fall back to compactReceiverDefault.

The lookup key is the event fields named by compactReceiverFieldNames joined with _ (defaults to [symbolMessage]), matching the same pattern identity the Reporter attributes cost to.

File format (CSV; header row + one comma-separated entry per row):

fieldSet,value
<fieldSet>,<true|false>[:<untilEpochSec>][:<reason>]
  • fieldSet, the event fields named by compactReceiverFieldNames joined with _. With the default [symbolMessage] this is the symbolMessage value for the pattern.
  • value, true (compact via encode()) or false (explicitly preserve fullText for this pattern, beating the default).
  • untilEpochSec, optional Unix-epoch (seconds) expiry. Past it the entry becomes a no-op and the pattern falls back to compactReceiverDefault.
  • reason, optional free-text for audit. Must not contain commas (would break CSV parsing).

Example:

fieldSet,value
payment_retry_gateway_timeout,true:1745856000:OPS-5123 spike
auth_audit_trail,false:1745856000:compliance keep verbose

The file is hot-reloaded on in-place writes (the gitops pattern); a Kubernetes ConfigMap mount won't reload because the CM swap is a symlink rename, not an in-place write.

compactReceiverLookupRetain

Retention period for the per-pattern cap-file before it is considered stale.

Type Default
Number 300000

Defines the retention period (in milliseconds) for the cap-file (compactReceiverLookupFile). If the file's last modified time is older than this period, a warning is logged. The cap entries continue to apply; the staleness check is advisory so operators notice when an automated tooling pipeline has stopped updating the file.

Validation: must be greater than 60000ms (1 minute).

compactReceiverFieldNames

Event fields joined to form each event's cap-file lookup key.

Type Default
List [symbolMessage]

Defines the list of TenXObject field names joined with _ to form each event's lookup key into the cap-file. Matches the rate receiver's rateReceiverFieldNames, so a cap-file entry targets the same pattern identity the Reporter attributes cost to.

Default: [symbolMessage], the symbolMessage field, which is the stable pattern identity produced by the engine's classifier.

Multi-dimensional compaction (e.g., compact a pattern only for a specific tenant):

compactReceiverFieldNames:
  - symbolMessage
  - tenantId

Changing this field set requires an engine restart.

compactReceiverDefault

Fallback compaction decision when no cap-file entry applies.

Type Default
Boolean false

Controls the fallback decision when no cap-file is loaded, no entry matches an event's pattern, or a matching entry has expired.

  • false (default), preserve fullText. Cap-file entries opt specific patterns INTO compaction.
  • true, compact via encode(). Cap-file entries opt specific patterns OUT (e.g., audit/compliance patterns that must stay verbose).

Flipping this value is a policy-level decision (affects every event), not a cap-file edit. Changing it requires a pod rollout. The cap-file handles per-pattern overrides without restart.


This module is defined in compact/module.yaml.