Compact Receiver

Ship every high-value event as full text and every low-signal event as a compact template+values tuple — without redeploying the engine.

The compact receiver makes a per-event decision whether to emit via encode() (the pattern's template hash plus extracted variable values — typically 20–40× smaller than the original line) or preserve fullText. The decision is keyed by the same field-set identity the rate receiver uses, so the same symbolMessage value a Reporter attributes cost to is the key an operator targets to reduce that cost.

Entries are declared in a CSV lookup file, typically committed to a git repo and edited by PR. The file is re-read on forwarder pod restart — so the surface area of a policy change is a diff, a review, a merge, and a rolling restart.

Lookup entry format

Standard CSV with a key,value header:

key,value
<fieldSet>,true
<fieldSet>,false

• <fieldSet> — the joined values of compactReceiverFieldNames on the event. With the default [symbolMessage] it's just the symbolMessage value (e.g. payment_retry_gateway_timeout); with [symbolMessage, container] it becomes <symbolMessage>_<container>.
• value — true to compact via encode(), false to preserve fullText. Entries are deviations from compactReceiverDefault.

Example (with compactReceiverFieldNames: [symbolMessage] and compactReceiverDefault: false):

key,value
payment_retry_gateway_timeout,true
auth_audit_trail,false

For time-bounded overrides, remove the entry via PR once no longer needed — the receiver falls back to compactReceiverDefault for any unlisted field-set.

Default policy

compactReceiverDefault sets the fallback decision when no entry matches:

• false (default) — preserve fullText. Entries opt specific patterns into compaction. Right when most traffic is already high-signal.
• true — compact via encode(). Entries opt specific patterns out of compaction (e.g. audit/compliance patterns that must stay verbose). Right when most traffic is low-signal machinery and only a few patterns need full-text fidelity.

Flipping the default is a policy decision that affects every event and requires a pod rollout. Lookup edits handle pattern-level exceptions without restart.

Reload

The lookup file is read once at pod startup. Pushing a new entry requires a rolling restart of the forwarder daemonset for it to take effect.

compactReceiverLookupRetain controls the staleness warning logged at startup when the file's mtime is older than the interval (default 5m). It does not currently trigger a mid-run reload — reload-on-file-change is a planned enhancement.

Wiring

• Set compactReceiverLookupFile to the CSV path — that's the single gate that loads the module (both CompactInput and CompactObject check it in shouldLoad).
• The forwarder output streams then branch on a single ternary field expression: encoded=shouldEncode() ? encode() : fullText. No field mutation — the decision lives in the stream expression, not on the event.
• When compactReceiverLookupFile is not set, the pre-compact path is preserved unchanged (receive-only emits fullText; receiverOptimize=true emits encoded=encode() for every event).

Config Files

To configure the compact receiver module, Edit these files.

Below is the default configuration from: compact/config.yaml.

Edit Online Edit Locally

Edit config.yaml Locally

Download and save to:

• Linux / Docker / macOS
• Custom directory
  $TENX_CONFIG/run/receive/compact/config.yaml
• Within cloned repo
  ./pipelines/run/receive/compact/config.yaml

Download

# 🔟❎ 'run' compact receiver configuration

# Per-event compaction decision via a declarative lookup file.
# Each event is tagged with `_tenxEncode: "true"|"false"`; forwarder
# output streams branch on that field to emit encode() vs fullText.
# To learn more see https://doc.log10x.com/run/receive/compact/

# Set the 10x pipeline to 'run'
tenx: run

# =============================== Dependencies ================================

include: run/modules/receive/compact

# ============================ Compact Options ================================

compactReceiver:

  # 'lookupFile' — path or URL to the compact-decisions file.
  # Entry format: <fieldSet>=<encode>:<untilEpochSec>[:<reason>]
  # Hot-reloaded every `lookupRetain` ms without engine restart.
  # lookupFile: /etc/tenx/compact-lookup.csv

  # 'fieldNames' — how each event's lookup key is built.
  # Matches rate receiver's default so compact + rate entries
  # address the same pattern identity.
  fieldNames:
    - $=yield TenXEnv.get("symbolMessageField")

  # 'lookupRetain' — poll interval in milliseconds.
  lookupRetain: $=parseDuration("5m")

  # 'default' — fallback when no lookup entry applies.
  #   false (default) — preserve fullText; entries opt IN to compaction
  #   true            — compact via encode(); entries opt OUT
  default: false

Options

Specify the options below to configure the compact receiver:

Name	Description
compactReceiverLookupFile	Path or URL to a compact decisions lookup file
compactReceiverFieldNames	List of TenXObject fields used to key compact lookup entries
compactReceiverLookupRetain	Lookup file poll interval in milliseconds
compactReceiverDefault	Default compaction decision when no lookup entry applies

compactReceiverLookupFile

Path or URL to a compact decisions lookup file.

Type	Default
String	""

Points the engine at a lookup file whose entries express per-pattern compaction decisions.

Entry format:

<fieldSet>=<encode>:<untilEpochSec>[:<reason>]

encode is true (compact via encode()) or false (preserve fullText). untilEpochSec is a Unix epoch timestamp after which the entry self-heals to compactReceiverDefault.

Example (with compactReceiverFieldNames: [symbolMessage]):

payment_retry_gateway_timeout=true:1745856000:OPS-5123 spike mitigation
auth_audit_trail=false:1745856000:compliance — keep verbose

Entries express deviations from compactReceiverDefault. An expired entry falls back to the default exactly like a rate-receiver mute does.

The engine polls the file on the interval set by compactReceiverLookupRetain — changes take effect without a restart, so this plays cleanly with a GitOps workflow where the MCP authors entries via PR.

compactReceiverFieldNames

List of TenXObject fields used to key compact lookup entries.

Type	Default
List	[symbolMessage]

Defines the list of TenXObject field names joined with underscores to form each event's lookup key. The key format matches the rate receiver's rateReceiverFieldNames, so an entry in the compact lookup targets the same pattern identity the Reporter attributes cost to.

Default: [symbolMessage] (same as the rate receiver's default).

Multi-dimensional compaction (e.g., "compact this pattern only for tenants X, Y"):

compactReceiverFieldNames:
  - symbolMessage
  - tenantId

Changing this field set currently requires an engine restart — a future enhancement will allow hot-swapping via a bootstrap key inside the lookup file.

compactReceiverLookupRetain

Lookup file poll interval in milliseconds.

Type	Default
Number	300000

Defines how often the engine re-reads compactReceiverLookupFile to pick up new/changed entries. Default of 5 minutes (300000ms) aligns with the rate receiver's rateReceiverLookupRetain.

Shorter values make GitOps changes take effect faster at the cost of more file I/O. Longer values reduce I/O but delay effect.

Validation: must be at least 60000ms (1 minute).

compactReceiverDefault

Default compaction decision when no lookup entry applies.

Type	Default
Boolean	false

Controls the fallback decision when no lookup file is loaded, no entry matches an event's field-set, or a matching entry has expired.

• false (default) — preserve fullText. Lookup entries opt specific patterns INTO compaction.
• true — compact via encode(). Lookup entries opt specific patterns OUT (e.g., audit/compliance patterns that must stay verbose).

Flipping this value is a policy-level decision (affects every event), not a lookup edit. Changing it requires a pod rollout. The lookup handles pattern-level overrides without restart.

This module is defined in compact/module.yaml.