Skip to content

Rate

Hold each log pattern, and each service, to a budget per window, on the forwarder, before that volume is billed downstream. Lines over a budget take the service's action; error-class lines keep a fixed count per pattern, and a pattern an operator lists in the mute file follows its own entry instead of the cap.

The rate regulator watches each container's recent volume and acts on any one pattern that spends more than the cap set for that container. That pattern is the same value, in the field symbolMessageField names (message_pattern in the shipped configuration), that a Reporter attributes cost to, so a top spender maps straight to what gets regulated. No cap ships by default, so nothing is regulated until an operator sets one.

The cap

The cap is a budget for one pattern in one container over one window (resetIntervalMs, default 4 minutes). The regulator counts it in unit: bytes, the UTF-8 size of each event as the forwarder hands it to the Receiver, the whole record and not only the message, or events, one per line, for analyzers that bill per event such as Datadog. The cap and total files are written in the same unit. The cap resolves per event: the container's row in the cap file, then the file's * row, then the fleet-wide absoluteCap. All three default to none, and a resolved cap of 0 means keep everything, so protection is opt-in.

Windows start at each multiple of resetIntervalMs since the Unix epoch, so every node's windows line up with the clock: with the default of 4 minutes they run from :00 to :04, :04 to :08 and on through the hour, the same windows for all patterns and services. The first window after the engine starts is partial and ends at the next multiple. A policy file reload does not start a new window.

At or below the cap every line is kept. Above it a line is held and takes the container's action, with three exceptions: the first baselineCount lines of every pattern each window (default 5), the error-class lines in keepCounts, and the optional share guard.

With the shipped settings, the most one pattern in one container passes per window is the larger of its cap and the 5-line sample, plus 50 lines for an error-class pattern. The window is recent rather than all-time, so a pattern that spikes during a deploy and then goes quiet stops being held on its own.

The share guard is off by default (minSharePercent: 0). Set above 0, it keeps a pattern over its cap while the pattern is below that share of its container's volume, (pattern units + event) / (container units + event) over the window.

Service totals

An optional total file bounds what each service passes per window on this node, across all its patterns. New patterns from a deploy each bring a full per-pattern cap, so a service without a total can send many times its budget until its caps are rewritten. Two rules apply once a service has a total:

  • Largest first. When the service's caps alone let more than its total through in the previous window, every pattern's cap scales by total / that amount for the whole window, or by what they let through so far this window once that is larger. The service's largest patterns reach the scaled cap first and are held, and its small ones keep passing. The scale follows the previous window, so the window after a storm is scaled by the storm for its whole length and recovers the window after. The first window after a quiet one fills its total in arrival order before the scale applies. A service whose first lines of this window and of the last window it sent in are two window lengths or more apart scales by total / its units so far this window instead.
  • Hard bound. Once the lines the service passed reach its total, every further line takes the service's action.

The baseline sample counts toward the total like any line. Once the total is spent, the service passes at most baselineMaxOverTotal more sample lines in the window (default 50, counted in lines whatever the unit), and the rest of its samples take the service's action. The error-class lines in keepCounts count toward the total the same way, and once it is spent the service passes at most keepMaxOverTotal more of them in the window (default 500, counted in lines), so with the shipped settings a burst of new error patterns exceeds the total by at most 550 lines. The lines a mute entry passes always pass and count toward the total, so the service's other patterns give way first. Each regulator instance holds its own total; the MCP's budget run (log10x_setup_recurring) divides a fleet budget across instances, and a node added between runs enforces its own total until the next run.

File format, CSV with a header row, keyed by the containerField value, read through totalLookupFile (TOTAL_LOOKUP_FILE):

container,total
<container>,<total>[:<untilEpochSec>][:<reason>]
*,<total>[:<untilEpochSec>][:<reason>]
  • total is the service's total per window on this instance, in unit. 0 exempts the service.
  • * sets the total for every service without a row of its own; a * row of 0 gives them none.
  • untilEpochSec expires the entry, which then self-heals to a no-op.
  • reason is free text for audit. Must not contain commas (would break CSV parsing).

A missing total file loads as an empty table, so no service has a total until a row names it. Totals apply to containers with a cap, in the regulators that read the cap file.

Error-class lines

Error-class lines are never held to zero. keepCounts passes a fixed number of lines per pattern per window for each listed level, once the pattern is over its cap or its service is over its total, and under an active mute entry. The shipped list is ERROR=50, CRITICAL=50 and FATAL=50; the lines past the count take the container's action. The count is per pattern, per container, per node, per window, so it bounds what an error storm adds to the bill. With a cap file, keepCountSlices (default 4) releases the count through the window: the allowance grows by a quarter each minute of a 4-minute window, 12, 25, 37 and 50 lines in all for ERROR=50, so a sustained storm passes 12 or 13 lines every minute. Lines a minute leaves unused carry to the next, so a burst late in the window passes more than one early in it, never more than the count, and lines under a mute entry share the same count.

ERROR, CRITICAL and FATAL always keep a count, 50 unless keepCounts sets another: the module sets the three itself and entries merge by level, so changing one means setting it. Any other level not in keepCounts, and a level set to -1 there, follows severityFloors, the fraction of a held pattern's lines kept at random. The shipped floors are 0 for every level, and any level with no floor of its own takes minRetentionThreshold (0). Both lists read the severity the level enrichment produced, so they apply whatever field the original log used. A config without keepCounts therefore still keeps them; severityFloors must stay set, since an empty or missing list stops the launch.

Warmup

An optional warmupMs leaves a container unregulated for that long after this regulator instance first sees its events. The default is 0, so a container is capped from its first line. Set it for slow-ramping JVMs or workloads with long init phases whose first minutes should not be capped.

"First sees" is per regulator instance, not per container birth. A container that has been running for hours but whose events have only just started flowing through this regulator, after a regulator restart or a forwarder reconnect, restarts the warmup window. With a warmup set, a daemonset rolling restart therefore disables the cap for warmupMs per node as it cycles.

Protection list

An optional mute file overrides the regulator for patterns an operator has declared. A listed, active pattern is decided by its entry, so the human declaration wins and the regulator is skipped for it; every other pattern is handled by the cap. The two run together rather than as separate modes.

File format, CSV with a header row, keyed by the joined fieldNames:

fieldSet,value
<fieldSetKey>,<sampleRate>:<untilEpochSec>[:<reason>]
  • sampleRate retains that fraction: 1 is never sampled, 0 is a full mute.
  • untilEpochSec expires the entry, which then self-heals to a no-op.
  • reason is free text for audit.

The error-class levels in keepCounts still pass their count per pattern per window before the sample rate applies, so a 0 mute never silences ERROR, CRITICAL or FATAL. The file is typically committed to a repo and pulled via gitops, or read directly from a Kubernetes ConfigMap via the @kubernetes launch macro, so each change carries a diff, a review, and a merge. Both lanes hot-reload via atomic rename; a plain volume-mounted ConfigMap does not, because the swap is a symlink rename.

Per-container caps

An optional cap file sets a cap for a specific container in priority over the fleet-wide absoluteCap. Listed containers get their own row; unlisted containers get the * row, and with no * row they fall back to absoluteCap (or to no cap, when absoluteCap is 0).

File format, CSV with a header row, keyed by the containerField value (k8s container name by default):

container,cap
<container>,<cap>[:<untilEpochSec>][:<reason>]
*,<cap>[:<untilEpochSec>][:<reason>]
  • cap is the per-pattern per-window cap for the container, in unit. 0 exempts the container from every cap, the * row and absoluteCap included.
  • * sets the cap for every container without a row of its own; a * row of 0 exempts them all, absoluteCap included.
  • untilEpochSec expires the entry, which then self-heals to a no-op.
  • reason is free text for audit. Must not contain commas (would break CSV parsing).

The cap value changes; the baseline sample, the keep counts and the service total still apply. The Log10x MCP writes the file: its budget run (log10x_setup_recurring) writes it to the policy ConfigMap or commits it to the policy branch, and a log10x_configure_engine call derives caps from a target percent or a monthly dollar budget and delivers them as its delivery says: a PR against the gitops repo, a write to the policy ConfigMap, or no write (stdout_only).

Same hot-reload rule as the mute file: both launch-macro lanes reload, a plain volume-mounted ConfigMap does not.

From a policy repo, the GitHub lane in pipelines/gitops/config.yaml pulls log10x/policy/*.csv and mirrors it into ${GH_DEST} (default ${java.io.tmpdir}/tenx/github). With GH_ENABLED=true, GH_REPO, GH_BRANCH and GH_TOKEN set, the cap, action, total and config-generation files default to those copies, and each sync rewrites them in place. The folder must exist in the repo and hold caps.csv when the engine starts, and a sync refreshes only the files the first pull found. With the ConfigMap lane (K8S_ENABLED=true) also on, the GitHub copies win for these four files, so enable one lane.

Actions

An optional action file decides what happens to the lines the cap or the service total holds. Without it, or for a container with neither its own row nor a * row, the held lines take rateReceiverDefaultAction, pass unless set.

File format, CSV with a header row, keyed by the same containerField value as the cap file:

container,action
<container>,<action>[:<untilEpochSec>][:<reason>]
*,<action>[:<untilEpochSec>][:<reason>]
  • action is one of pass, offload, tier_down, compact, sample, or drop. * sets the action for every container without a row of its own; a container with neither takes rateReceiverDefaultAction, pass unless set.
  • untilEpochSec expires the entry, which then self-heals to a no-op.
  • reason is free text for audit. Must not contain commas (would break CSV parsing).

The action is set per service and applies to every held line in that service. Here a service is a container. Per-pattern thinning comes from the mute file, not from this file.

The event keeps flowing either way. The action lands on the event as its routeState, and the output streams and the forwarder recipe act on that, so offload and tier_down reach their destinations instead of being discarded at the regulator. The cap and the total decide which lines are held; the action only decides where they go.

The action and total files are siblings of the cap file in the same ConfigMap, and hot-reload on the same rule.

Containers

Share is scoped per container, named by containerField (default the k8s container name). That name is stable across replicas, so scaling from one pod to ten does not bypass the cap, and a sidecar never spends the application container's share. Use container, never pod.

Outside Kubernetes, or when no container field is present, the regulator falls back to a single node-wide bucket and caps each pattern across the node.

Savings

Held lines are measured, not estimated. The receive-stage aggregators tally every event by pattern and container both before and after regulation, so the saving for a pattern is the volume seen minus the volume emitted. A dropped event still counts as seen, so the figure reflects exactly what the regulator removed.

Config-version stamp

When the receiver runs the MCP closed loop, the control plane (log10x_configure_engine) writes a config-generation.csv next to the cap file in the same ConfigMap, a one-row key,value CSV whose generation is a hash of the cap policy. Point configGeneration.file at it (or set the CONFIG_GENERATION_FILE env) and the receiver stamps that value on every event as tenx_config_version and publishes it as one series per generation, config_generation_summaryBytes_total{tenx_config_version=...}, so the running engine advertises which config generation it loaded. The per-pattern counters do not carry the label, so a policy write does not restart their series. The MCP then confirms the policy it wrote is actually live by comparing the label to the hash of the current caps, the config-generation closed loop.

It is opt-in and decoupled from caps: unset by default, so a regulator managed by GitOps without the MCP emits no tenx_config_version label and never depends on the file existing. Hot-reloaded like the cap file, so a new generation goes live without an engine restart (this is the stale → live transition the verifier observes).

Wiring

rateReceiver:
  fieldNames:
    - message_pattern        # the pattern identity, the field symbolMessageField names
  containerField: container  # scopes the cap denominator
  unit: bytes                # or events, for analyzers that bill per event
  absoluteCap: 10485760      # 10 MB per pattern per container per window (optional; 0 = no fleet-wide cap)
  minSharePercent: 0         # share guard, off
  keepCounts:                # error-class lines kept per pattern per window once held
    - ERROR=50
    - CRITICAL=50
    - FATAL=50
  keepCountSlices: 4         # slices of the window through which the keep count accumulates; 1 = all at the window start
  severityFloors:            # random share kept for the other levels once held
    - TRACE=0
    - DEBUG=0
    - INFO=0
    - WARN=0
    - ERROR=0
    - CRITICAL=0
    - FATAL=0
  warmupMs: 0                # per-instance grace before capping; 0 caps from the first line
  baselineCount: 5
  baselineMaxOverTotal: 50   # sample lines a service passes per window once over its total; 0 = no bound
  keepMaxOverTotal: 500      # keep-count lines a service passes per window once over its total; 0 = no bound
  capLookup:
    # file: $=path("data/caps") + "/caps.csv"   # optional per-container caps, with a * row
    retain: $=parseDuration("10m")
  # actionLookupFile: $=path("data/caps") + "/actions.csv"  # optional per-container action for held lines; default pass
  # totalLookupFile: $=path("data/caps") + "/totals.csv"    # optional per-service total per window
  configGeneration:
    # file: $=TenXEnv.get("CONFIG_GENERATION_FILE", "")   # MCP config-version stamp; opt-in, sibling of caps.csv

Tune these values in this config block, not via container environment variables. Any rateReceiver: key set here resolves to a launch argument at engine init and shadows a same-named env var, so env-only overrides are silently ignored. Edit the config (via a gitops PR) to change a value at runtime.

Config Files

To configure the rate receiver module, Edit these files.

Below is the default configuration from: rate/config.yaml.

Edit Online

Edit config.yaml Locally

# 🔟❎ 'run' rate regulator configuration

# The rate regulator holds each log pattern in a container to a cap per window, and
# each container to an optional total per window, counted in bytes or in events.
# Lines over a cap or a total take the container's action; error-class levels keep
# a fixed count of lines per pattern per window. An optional mute file overrides the
# regulator for explicitly declared patterns.
# To learn more see https://doc.log10x.com/run/receive/rate/

# Set the 10x pipeline to 'run'
tenx: run

# =============================== Dependencies ================================

include: run/modules/receive/rate

# ============================== Rate Options =================================

rateReceiver:

  # 'fieldNames' identifies the log PATTERN (the numerator bucket): the field the
  #  message initializer writes, named by symbolMessageField (message_pattern). The container is configured
  #  separately via 'containerField' below -- it is the denominator and is keyed
  #  on its own, so a pattern's share is measured per container.
  fieldNames:
    - $=yield TenXEnv.get("symbolMessageField")

  # 'containerField' names the field whose value scopes the share denominator
  #  (the per-container total). Defaults to the k8s container name, which is
  #  stable across pod replicas (never the pod). When the field is absent on an
  #  event (non-k8s input), the regulator falls back to a single node-wide bucket.
  containerField: $=yield TenXEnv.get("k8sContainerNameField")

  # 'resetIntervalMs' is the window over which recent share is measured.
  #  Windows start at each multiple of the interval since the Unix epoch, so
  #  every node's windows line up with the clock, the same windows for all
  #  patterns and services. The first window after a start is partial, and a
  #  policy file reload does not start a new window.
  resetIntervalMs: $=parseDuration("4m")

  # 'unit' is what the caps, the totals and the share guard count: 'bytes' (each
  #  event counts the UTF-8 size of the whole record the forwarder sends, not only
  #  the message) or 'events' (each line counts 1). Write the cap
  #  and total files in the same unit. Any other value stops the launch.
  unit: bytes

  # 'absoluteCap' is the fleet-wide cap: the budget, in 'unit', for a single
  #  pattern per container within the current window. Past it, lines take the
  #  container's action, except the 'baselineCount' sample and, for the levels
  #  in 'keepCounts', that many lines per pattern per window.
  #
  #  Default 0 = disabled. Three customer configurations are first-class:
  #    - Per-container only: leave 'absoluteCap' unset, set 'capLookup.file' below.
  #      Listed containers get the file's cap; unlisted containers are
  #      unregulated by the absolute cap. Most selective.
  #    - Fleet-wide only: set 'absoluteCap' to a positive integer (e.g., 10485760
  #      for 10 MB). Applies to every container the regulator sees.
  #    - Both: per-container entries in 'capLookup.file' override the fleet-wide
  #      cap; unlisted containers fall back to the fleet-wide cap (safety floor).
  # absoluteCap: 10485760   # 10 MB

  # 'minSharePercent' is an optional guard for legitimately high-volume
  #  containers. If a pattern is over its cap BUT below this share of its
  #  container's volume, it is left alone. 0 turns the guard off, so a cap holds
  #  every pattern it is set for.
  minSharePercent: 0

  # 'defaultAction' is the action the held lines of a container with neither its
  #  own row nor a '*' row in 'actionLookupFile' take: 'pass', 'offload',
  #  'tier_down', 'compact', 'sample' or 'drop'. 'pass' leaves them in place, so
  #  only the containers the file covers act on theirs. A value outside the six
  #  logs a warning at load and 'pass' applies. Read by the regulators that
  #  consult 'actionLookupFile'.
  defaultAction: pass

  # 'keepCounts' is the number of lines per pattern per window a level passes once
  #  the pattern is over its cap or its service over its total, also under a mute
  #  entry, in place of its 'severityFloors' draw. ERROR, CRITICAL and FATAL keep
  #  a count of 50 even when removed here; -1 gives a level its floor instead. In
  #  a service with a total, the lines count toward it and 'keepMaxOverTotal'
  #  bounds them past it.
  keepCounts:
    - ERROR=50
    - CRITICAL=50    # the level enrichment maps fatal/critical -> CRITICAL
    - FATAL=50       # kept for forwarders whose level vocabulary emits FATAL

  # 'keepCountSlices' is how many equal slices of the window release each
  #  pattern's 'keepCounts' lines. The allowance accumulates: by the end of
  #  slice i of S the pattern may have passed the count x i / S, rounded down,
  #  so with ERROR=50 and the 4-minute window, 4 slices pass 12, 13, 12 and 13
  #  lines of a sustained storm in its four minutes. Lines a slice leaves unused
  #  carry to the next, so a burst late in the window passes more than one
  #  early in it, never more than the count, and lines under a mute entry share
  #  the same count. 1 passes the whole count at the window start; below 1 or
  #  not a number acts as 1, and a fraction is rounded down. Read by the
  #  regulators that consult 'capLookup.file'.
  keepCountSlices: 4

  # 'severityFloors' is the fraction of each level kept at random once a pattern is
  #  over its cap, for levels not in 'keepCounts'. 0 keeps none, so the cap holds.
  #  Floor-map keys must match the level vocabulary emitted by the level enrichment.
  severityFloors:
    - TRACE=0
    - DEBUG=0
    - INFO=0
    - WARN=0
    - ERROR=0        # 'keepCounts' governs ERROR, CRITICAL and FATAL
    - CRITICAL=0
    - FATAL=0

  # 'minRetentionThreshold' is the floor used for any level not present in
  #  'severityFloors'.
  minRetentionThreshold: 0

  # 'warmupMs' is an optional per-instance grace period before the regulator starts
  #  capping a newly-seen container. Scope is per regulator instance, so a regulator
  #  restart restarts the window. 0 caps a container from its first line. See
  #  https://doc.log10x.com/run/receive/rate/#warmup for the operator guide.
  warmupMs: 0

  # 'baselineCount' is the number of events of each pattern kept per window
  #  regardless of the cap and the share, so even a heavily-capped pattern leaves
  #  a forensic sample. In a service with a total, the sample counts toward it.
  baselineCount: 5

  # 'baselineMaxOverTotal' is the most sample lines a service passes per window
  #  once it has passed its total; the rest take the regular path (the keep
  #  count, then the service's action). It bounds a service that prints many new
  #  patterns in one window. Counted in lines whatever 'unit' is. 0 removes the
  #  bound, so every sample line passes.
  baselineMaxOverTotal: 50

  # 'keepMaxOverTotal' is the most 'keepCounts' lines a service passes per window
  #  once it has passed its total; the rest take the service's action. Within the
  #  total they pass like any line. It bounds a service that prints many new
  #  error patterns in one window. Counted in lines whatever 'unit' is. 0 removes
  #  the bound, so every keep-count line passes.
  keepMaxOverTotal: 500

  # 'ingestionCostPerGB' is used only to render savings metrics in dollars; it does
  #  not affect the keep/drop decision (the decision is share-based).
  ingestionCostPerGB: 1.5

  # ---------------------------- Mute File Options ----------------------------

  # Optional declarative mute file keyed by the joined 'fieldNames'. The mute file
  # composes with the regulator: a listed, active pattern is decided by its file
  # entry (the human declaration wins), and everything else is handled by the
  # regulator.
  #
  # File format (CSV; header row + one comma-separated entry per row):
  #   fieldSet,value
  #   <fieldSetKey>,<sampleRate>:<untilEpochSec>[:<reason>]
  #
  # sampleRate 1.0 = never sampled; <1.0 = retain at that rate. Entries self-heal
  # past 'untilEpochSec'. The 'keepCounts' levels still pass their count per
  # pattern per window, so a 0.0 mute never silences ERROR/CRITICAL/FATAL, and in
  # a capped container the lines a mute passes count toward the service's total.
  #
  # Periodically pulling the mute file to keep it fresh is done via the gitops
  # configuration -- see https://doc.log10x.com/config/github/#config
  lookup:

    # 'file' specifies the mute file path. Hot-reloaded on in-place writes (the
    #  gitops pattern); Kubernetes ConfigMap mounts won't reload. Comment out to
    #  run the regulator alone (the default).
    # file: $=path("data/sample/mutes") + "/mutes.csv"

    # 'retain' specifies the period before the file is marked as stale.
    retain: $=parseDuration("10m")

  # ---------------------------- Cap File Options -----------------------------

  # Optional declarative per-container cap file keyed by the value of
  # 'containerField' (the k8s container name by default). Each entry assigns a
  # cap, in 'unit', to a specific container; the file wins per-event over
  # 'absoluteCap' above. A row keyed '*' applies to every container without a
  # row of its own. Unlisted containers with no '*' row fall back to
  # 'absoluteCap' (or to no cap, when 'absoluteCap' is unset/0). A row of 0
  # exempts its container; a '*' row of 0 exempts every container without a row
  # of its own, so 'absoluteCap' then applies to none of them.
  #
  # File format (CSV; header row + one comma-separated entry per row):
  #   container,cap
  #   <container>,<cap>[:<untilEpochSec>][:<reason>]
  #   *,<cap>[:<untilEpochSec>][:<reason>]
  #
  # Written by the Log10x MCP: its budget run (log10x_setup_recurring) writes the
  # file to the policy ConfigMap or commits it to the policy branch. A
  # log10x_configure_engine call derives caps from a target percent or a monthly
  # dollar budget and delivers them as its 'delivery' says: a PR against the
  # gitops repo, a write to the policy ConfigMap, or no write (stdout_only).
  capLookup:

    # 'file' specifies the per-container cap file path. Hot-reloaded on in-place
    #  writes (the gitops pattern) and when the file is replaced via ATOMIC_MOVE
    #  (the Kubernetes ConfigMap pull driver, whose copy is at
    #  ${java.io.tmpdir}/tenx/kubernetes/<namespace>/<configMap>/caps.csv).
    #
    # CAP_LOOKUP_FILE sets the path. Unset, no cap table loads and every container
    # falls back to 'absoluteCap', so a Receiver launches before its policy files
    # arrive.
    #
    # With GH_ENABLED=true (the policy lane in pipelines/gitops/config.yaml), the
    # default is the pulled copy of the policy repo's log10x/policy/caps.csv, in
    # the lane's mirror folder: ${GH_DEST:-${java.io.tmpdir}/tenx/github}. The
    # policy folder must then hold caps.csv when the engine starts. GH_ENABLED is
    # read as the lane reads it: true, True or TRUE turn it on (any other value
    # but false/False/FALSE stops the launch). It wins over the ConfigMap path
    # for all four policy files: enable one lane, not both.
    file: '$=TenXEnv.get("CAP_LOOKUP_FILE", (TenXEnv.get("GH_ENABLED", "false") == "true" || TenXEnv.get("GH_ENABLED", "false") == "True" || TenXEnv.get("GH_ENABLED", "false") == "TRUE") ? TenXEnv.get("GH_DEST", TenXEnv.get("java.io.tmpdir", "/tmp") + "/tenx/github") + "/log10x/policy/caps.csv" : TenXEnv.get("CAP_LOOKUP_FILE"))'

    # 'retain' specifies the period before the file is marked as stale.
    retain: $=parseDuration("10m")

  # =========================== Config-Version Stamp ============================
  #
  # 'configGeneration.file' points at the config-generation.csv the MCP control
  # plane (log10x_configure_engine) writes as a SIBLING of the cap file in the
  # cap ConfigMap. The receiver reads its `generation` value (a hash of the cap
  # policy) and stamps it on every event as `tenx_config_version`. The
  # `config_generation` aggregator publishes it as one series per generation
  # (`config_generation_summaryBytes_total{tenx_config_version=...}`); the per-pattern
  # counters do not carry it, so a policy write does not restart their series.
  # The running engine advertises which config generation it loaded and the MCP
  # can verify the policy it wrote is live (the config-generation closed loop).
  # Hot-reloaded like the cap file.
  #
  # OPT-IN, decoupled from caps: unset by default, so the stamp is emitted ONLY
  # when this points at a file. A receiver running the regulator WITHOUT the MCP
  # closed loop (GitOps-managed caps) leaves it unset and never loads the file --
  # no dependency, no init failure on a missing file. Set CONFIG_GENERATION_FILE
  # (or this) to the sibling of caps.csv, e.g.:
  #   ${java.io.tmpdir}/tenx/kubernetes/<namespace>/<configMap>/config-generation.csv
  # With GH_ENABLED=true the default is the GitHub policy lane's pulled copy of
  # log10x/policy/config-generation.csv.
  configGeneration:
    file: '$=TenXEnv.get("CONFIG_GENERATION_FILE", (TenXEnv.get("GH_ENABLED", "false") == "true" || TenXEnv.get("GH_ENABLED", "false") == "True" || TenXEnv.get("GH_ENABLED", "false") == "TRUE") ? TenXEnv.get("GH_DEST", TenXEnv.get("java.io.tmpdir", "/tmp") + "/tenx/github") + "/log10x/policy/config-generation.csv" : "")'

  # --------------------------- Action File Options ---------------------------

  # Optional declarative per-SERVICE action file, keyed by the value of
  # 'containerField' (the k8s container name = the service). Each entry assigns
  # the disposition of that service's held lines: 'pass' (default), 'offload'
  # (to customer S3), 'tier_down' (SIEM cheap tier), 'compact' (lossless
  # encode), 'sample', or 'drop'. The cap and the total decide which lines are
  # held; the action only decides what happens to them. A row keyed '*' applies
  # to every service without a row of its own; a service with neither takes
  # 'defaultAction' above ('pass' unless set), and a cap file with no actions
  # file loads an empty table.
  #
  # File format (CSV; header row + one comma-separated entry per row):
  #   container,action
  #   <container>,<action>[:<untilEpochSec>][:<reason>]
  #   *,<action>[:<untilEpochSec>][:<reason>]
  #
  # Written by the Log10x MCP beside the cap file, the same way (see 'capLookup').
  # 'actionLookupFile' is the per-service action file path. Declared as a FLAT
  #  key (like 'fieldNames' / 'minSharePercent'): the schema generator nests a
  #  group only when two or more options share its prefix, and this option is
  #  the only one under 'actionLookup'. Always set (default below, sibling to the
  #  cap file) so the cap-variant regulator can read it: the pulled copy of
  #  log10x/policy/actions.csv with GH_ENABLED=true, else the ConfigMap path.
  #  ACTION_LOOKUP_FILE overrides for non-k8s / fixed-path deployments.
  actionLookupFile: '$=TenXEnv.get("ACTION_LOOKUP_FILE", (TenXEnv.get("GH_ENABLED", "false") == "true" || TenXEnv.get("GH_ENABLED", "false") == "True" || TenXEnv.get("GH_ENABLED", "false") == "TRUE") ? TenXEnv.get("GH_DEST", TenXEnv.get("java.io.tmpdir", "/tmp") + "/tenx/github") + "/log10x/policy/actions.csv" : TenXEnv.get("java.io.tmpdir", "/tmp") + "/tenx/kubernetes/" + TenXEnv.get("K8S_NAMESPACE", "demo") + "/" + TenXEnv.get("K8S_CONFIGMAP", "log10x-action-intent") + "/actions.csv")'

  # ---------------------------- Total File Options ---------------------------

  # Optional declarative per-SERVICE total file, keyed by the value of
  # 'containerField'. Each entry bounds what the service passes per window on
  # this node, in 'unit'. When the service's caps alone let more than its total
  # through in the previous window, every pattern's cap scales by total / that
  # amount for the whole window, or by what they let through so far this window
  # once that is larger, so its largest patterns are held first. The scale
  # follows the previous window, so the window after a storm is scaled by the
  # storm for its whole length and recovers the window after. The first window
  # after a quiet one fills its total in arrival order before the scale applies.
  # A service whose first lines of this window and of the last window it sent in
  # are two window lengths or more apart scales by total / its units so far this
  # window instead. Once the lines it passed reach the total, every further line
  # takes the service's action. The 'baselineCount' sample counts toward the
  # total like any line, and once the total is spent the service passes at most
  # 'baselineMaxOverTotal' more sample lines in the window. The 'keepCounts'
  # lines count toward the total the same way, and once it is spent the service
  # passes at most 'keepMaxOverTotal' more of them in the window. The lines a
  # mute entry passes still pass and count toward the total, so a muted pattern
  # can exceed it by that much. Read by the cap-file regulators, for containers
  # with a cap. A row keyed '*' applies to every service without a row of its
  # own; a row of 0 exempts its service, and a '*' row of 0 gives every unlisted
  # service no total. A missing file loads as an empty table.
  #
  # File format (CSV; header row + one comma-separated entry per row):
  #   container,total
  #   <container>,<total>[:<untilEpochSec>][:<reason>]
  #   *,<total>[:<untilEpochSec>][:<reason>]
  #
  # 'totalLookupFile' is the per-service total file path, a FLAT key like
  #  'actionLookupFile', defaulting to the sibling of the cap file: the pulled
  #  copy of log10x/policy/totals.csv with GH_ENABLED=true, else the ConfigMap
  #  path. TOTAL_LOOKUP_FILE overrides for non-k8s / fixed-path deployments.
  totalLookupFile: '$=TenXEnv.get("TOTAL_LOOKUP_FILE", (TenXEnv.get("GH_ENABLED", "false") == "true" || TenXEnv.get("GH_ENABLED", "false") == "True" || TenXEnv.get("GH_ENABLED", "false") == "TRUE") ? TenXEnv.get("GH_DEST", TenXEnv.get("java.io.tmpdir", "/tmp") + "/tenx/github") + "/log10x/policy/totals.csv" : TenXEnv.get("java.io.tmpdir", "/tmp") + "/tenx/kubernetes/" + TenXEnv.get("K8S_NAMESPACE", "demo") + "/" + TenXEnv.get("K8S_CONFIGMAP", "log10x-action-intent") + "/totals.csv")'

Options

Specify the options below to configure the rate receiver:

Name Description
rateReceiverFieldNames List of TenXObject fields to identify rate counter buckets
rateReceiverContainerField Field whose value scopes the per-container share denominator
rateReceiverResetIntervalMs Reset interval for rate counters in milliseconds
rateReceiverUnit What the caps, the totals and the share guard count, `bytes` or `events`
rateReceiverMinRetentionThreshold Default retention floor for severity levels not listed in severityFloors
rateReceiverSeverityFloors Fraction of each severity level kept at random once a pattern is held
rateReceiverKeepCounts Lines per pattern per window that pass for a level once the pattern is held
rateReceiverKeepCountSlices Equal slices of the window through which each pattern's keep count is released
rateReceiverLookupFile Declarative mute file keyed by field-set
rateReceiverLookupRetain Retention period for the lookup file containing global event type rates
rateReceiverActionLookupFile Declarative per-service action file, keyed by the `rateReceiverContainerField` value; decides the disposition of each service's regulator excess
rateReceiverDefaultAction The action a container with no `rateReceiverActionLookupFile` row takes for its over-cap excess
rateReceiverCapLookupFile Declarative per-container cap file, keyed by the `rateReceiverContainerField` value
rateReceiverCapLookupRetain Retention period for the per-container cap file before it is considered stale
rateReceiverTotalLookupFile Declarative per-service total file, keyed by the `rateReceiverContainerField` value; bounds what each capped service passes per window
rateReceiverConfigGenerationFile Optional config-generation file whose `generation` value is stamped as the `tenx_config_version` metric label
rateReceiverWarmupMs Per-container grace period before this regulator instance starts capping
rateReceiverBaselineCount Events per pattern kept each window regardless of share
rateReceiverBaselineMaxOverTotal Sample lines a service passes per window once it has passed its total
rateReceiverKeepMaxOverTotal Keep-count lines a service passes per window once it has passed its total
rateReceiverAbsoluteCap Fleet-wide cap per pattern per container per window; `0` (default) disables the cap unless overridden per-container via `rateReceiverCapLookupFile`
rateReceiverMinSharePercent Minimum share of container volume below which a pattern is exempt from sampling
rateReceiverIngestionCostPerGB Vendor ingestion cost per GB in USD, for rendering savings in dollars

rateReceiverFieldNames

List of TenXObject fields to identify rate counter buckets.

Type Default
List [symbolMessage]

Defines the list of TenXObject field names extracted to identify which rate counter bucket an event belongs to. The list usually contains the pattern field the message enrichment module writes, named by symbolMessageField: message_pattern in the shipped configuration, symbolMessage at that option's default. It can include additional fields like GeoIP, HTTP code, k8s container name, or custom enrichments for multi-dimensional rate tracking.

Common Use Cases:

Single-app receiving (per event type):

rateReceiverFieldNames:
  - message_pattern

Multi-dimensional tracking (event type + geography + HTTP status):

rateReceiverFieldNames:
  - message_pattern
  - country
  - httpCode

Multi-app receiving in Kubernetes:

Option A: Cap total spend per app (all event types combined):

rateReceiverFieldNames:
  - container  # Aggregates all event types for each app

Each app's total spend (across all event types and pods) gets one cap. Simple but loses event-type intelligence.

Option B: Cap spend per event type per app:

rateReceiverFieldNames:
  - message_pattern  # Event type
  - container      # App identifier (same across all pods)

Each (event type × app) combo gets its own cap. Provides fairness within apps but allows apps with many event types to potentially exceed one total cap.

The container is configured separately via rateReceiverContainerField (it scopes the share denominator), so fieldNames identifies only the log pattern itself.

rateReceiverContainerField

Field whose value scopes the per-container share denominator.

Type Default
String

Names the field whose value scopes the share denominator (the per-container total). A pattern's share is measured against the total volume of the container it belongs to, so the cap is enforced per (pattern, container).

Defaults to the k8s container name, which is stable across pod replicas, so pods of one container on the same node share the cap, and a sidecar never consumes the application container's allowance. Each regulator instance counts its own, so pods spread across ten nodes get ten caps. Use container, never pod.

When the field is absent on an event (non-k8s input, or no k8s extractor configured), the regulator falls back to a single node-wide bucket and enforces the cap per pattern across the node.

rateReceiverResetIntervalMs

Reset interval for rate counters in milliseconds.

Type Default
Number 240000

Defines the window in milliseconds over which a pattern's recent share is measured. The counters reset every interval, so share reflects recent volume rather than all-time totals. Windows start at each multiple of the interval since the Unix epoch, so every node's windows line up with the clock, the same windows for all patterns and services. The first window after a start is partial, and a policy file reload does not start a new window.

Shorter windows (e.g., 1 minute) are more reactive but noisier; longer windows are smoother but slower to adapt. The default of 4 minutes balances the two.

Validation: Must be at least 60000 milliseconds (1 minute).

rateReceiverUnit

What the caps, the totals and the share guard count, `bytes` or `events`.

Type Default
String bytes

Defines what the regulator counts against rateReceiverAbsoluteCap, the cap file (rateReceiverCapLookupFile), the total file (rateReceiverTotalLookupFile) and the share guard: bytes counts the UTF-8 size of each event as the forwarder sends it, the whole record and not only the message, events counts each line as 1.

Use events where the analyzer bills per event (Datadog indexes logs per event) and bytes where it bills per GB. Write the cap and total files in the same unit.

Validation: must be bytes or events; any other value stops the launch.

rateReceiverMinRetentionThreshold

Default retention floor for severity levels not listed in severityFloors.

Type Default
Number 0

Defines the retention floor (0.0 to 1.0) used for any severity level not present in rateReceiverSeverityFloors: the fraction of a held pattern's lines of that level kept at random. 0 keeps none, so the cap holds.

rateReceiverSeverityFloors

Fraction of each severity level kept at random once a pattern is held.

Type Default
List []

defines a map of severity levels to the fraction (0.0 to 1.0) of a held pattern's lines kept at random, for levels not listed in rateReceiverKeepCounts. The shipped configuration sets every level to 0, so a cap holds and error-class levels pass by count instead.

The floor also applies under a mute-file entry: an active entry keeps the larger of its sample rate and the floor.

Floor-map keys must match the level vocabulary emitted by the level enrichment. Any level not listed uses rateReceiverMinRetentionThreshold.

For example:

severityFloors:
  - TRACE=0
  - DEBUG=0
  - INFO=0
  - WARN=0
  - ERROR=0
  - CRITICAL=0
  - FATAL=0

rateReceiverKeepCounts

Lines per pattern per window that pass for a level once the pattern is held.

Type Default
List []

defines a map of severity levels to a count: the number of lines of each pattern of that level that pass per window once the pattern is over its cap or its container is over its total (rateReceiverTotalLookupFile), and under an active mute-file entry. Lines past the count take the container's action, or the mute's sample rate under a mute. A listed level skips its rateReceiverSeverityFloors draw.

The count is per pattern, per container, per regulator instance (one per node), per window (rateReceiverResetIntervalMs). The lines it passes count toward the container's total, and once the total is spent the container passes at most rateReceiverKeepMaxOverTotal more of them in the window.

Keys must match the level vocabulary emitted by the level enrichment. The module's settings set ERROR=50, CRITICAL=50 and FATAL=50, and entries merge by level, so those three levels always keep a count (50 unless set); an entry for one of them in the config or on the command line comes later and wins, and -1 gives it its floor instead. Any other level not listed follows its floor.

For example:

keepCounts:
  - ERROR=50
  - CRITICAL=50
  - FATAL=50

rateReceiverKeepCountSlices

Equal slices of the window through which each pattern's keep count is released.

Type Default
Number 4

Defines how many equal slices of the window (rateReceiverResetIntervalMs) release the rateReceiverKeepCounts lines of a pattern once it is over its cap or its container is over its total. The allowance accumulates through the window: by the end of the first of S slices the pattern may have passed the count x 1 / S lines, by the end of the second the count x 2 / S, rounded down, and the whole count in the last, so with ERROR=50, 4 slices and the 4-minute window a sustained storm passes 12, 13, 12 and 13 lines in the window's four minutes and reaches the destination in every one of them. Lines a slice leaves unused carry to the next, so a burst late in the window passes more than one early in it, never more than the count. Lines an active mute-file entry passes and the lines released here share one count per window. Slices start on the clock like windows, so with the defaults each slice is one clock minute. 1 releases the whole count at the start of the window; a value below 1 or not a number acts as 1, and a fraction is rounded down. Past the container's total every keep-count line counts toward rateReceiverKeepMaxOverTotal. Read by the cap-file regulators.

rateReceiverLookupFile

Declarative mute file keyed by field-set.

Type Default
String

Defines the path to a declarative mute file that caps specific log patterns by the joined field-set defined in rateReceiverFieldNames (e.g. message_pattern, container, httpCode). The key format is the same one the local receiver uses for per-node counters.

When this option is set, the mute file composes with the regulator rather than replacing it: a listed, active pattern is decided by its file entry (the human declaration wins, and the regulator is skipped for that event), while every other pattern is handled by the regulator's cap.

File format (CSV; header row + one comma-separated entry per row):

fieldSet,value
<fieldSetKey>,<sampleRate>:<untilEpochSec>[:<reason>]
  • fieldSetKey, the joined values of the fields named in rateReceiverFieldNames, separated by _. With rateReceiverFieldNames: [message_pattern] the key is just the pattern (e.g. Error_syncing_pod). With [message_pattern, container] the key is message_pattern_container (e.g. heartbeat_debug_frontend).
  • sampleRate, probability in [0, 1.0] that a matching event is retained. 0 = full mute; 0.1 = keep 10%; 1.0 = no-op.
  • untilEpochSec, mute expires at this Unix epoch (seconds). Past that, the entry becomes a no-op until someone edits or removes it. Self-healing by design.
  • reason, optional free-text string for audit. Not used at runtime. Must not contain commas (would break CSV parsing).

Example (with rateReceiverFieldNames: [message_pattern]):

fieldSet,value
Error_syncing_pod,0.10:1744848000:pod error spam OPS-4821
heartbeat_debug,0:1744416000:k8s liveness 200s
jwt_validated,0.25:1744502400:auth flood after deploy

The file is hot-reloaded on in-place writes (the gitops pattern); a Kubernetes ConfigMap mount won't reload because the CM swap is a symlink rename, not an in-place write.

Why this shape:

  • Field-set keyed, not regex keyed → mute keys are the exact identifiers the Reporter attributes cost to (same rateReceiverFieldNames on both sides), so a "top spender" in the Reporter maps 1:1 to a mute entry.
  • Git-friendly: the file is a human-readable declaration that can live in a config repo, reviewed via PR, git blame-d for who/why.
  • Self-healing: every mute has an explicit expiry, so forgotten entries eventually stop filtering instead of silently dropping production data forever.
  • AI-editable: an operator can ask an assistant (e.g., Claude Code via the Log10x MCP) to "mute Error_syncing_pod for 24 hours at 10%", the assistant reads the Reporter's cost attribution, appends the entry, and opens a PR or pushes the change, as the repo is set up. The mute file is the interface.

Error-class lines still pass by count. Under an active entry, the levels in rateReceiverKeepCounts pass up to their count per pattern per window, released across the window, before the sample rate applies, so a 0.0 mute never silences ERROR/CRITICAL/FATAL traffic.

Lines a mute passes still count. In a container with a cap file row, the lines an active entry passes count toward the service's total (rateReceiverTotalLookupFile), so the service's other patterns are held first once the total is reached.

When rateReceiverLookupFile is unset, the regulator runs alone on every pattern.

rateReceiverLookupRetain

Retention period for the lookup file containing global event type rates.

Type Default
Number 300000

Defines the retention period for the lookup file containing global event type frequency data.

If the file's last modified time is older than this period, the lookup is considered stale, and local counter rates are used. Used to make sampling decisions based on cluster-wide event patterns.

Validation: Must be greater than 60000 milliseconds.

rateReceiverActionLookupFile

Declarative per-service action file, keyed by the `rateReceiverContainerField` value; decides the disposition of each service's regulator excess.

Type Default
String ""

Defines the path to a declarative per-service action file (sibling to the cap file). Each entry assigns the disposition of a service's held lines: pass (default), offload (to customer S3), tier_down (analyzer cheap tier), compact (lossless encode), sample, or drop. The cap (rateReceiverCapLookupFile) and the total (rateReceiverTotalLookupFile) decide which lines are held; the action only decides what happens to them. A row keyed * applies to every service without a row of its own; a service with neither takes rateReceiverDefaultAction (pass unless set), and a cap file with no actions file loads an empty table. Read by the cap-variant regulator classes, which call route(action) at the over-budget decision point.

File format (CSV; header row + one comma-separated entry per row):

container,action
<container>,<action>[:<untilEpochSec>][:<reason>]
*,<action>[:<untilEpochSec>][:<reason>]
  • container, value of rateReceiverContainerField (the k8s container = the service), or * for every service without a row of its own.
  • action, one of drop / offload / tier_down / compact / sample / pass.
  • untilEpochSec, optional Unix-epoch (seconds) expiry; past it the entry is a no-op.
  • reason, optional free-text for audit. Must not contain commas (would break CSV parsing).

Intended to be managed via GitOps + the log10x_configure_engine MCP tool, which derives per-service actions from the per-pattern action plan it already computes.

From a policy repo. With GH_ENABLED=true, the default is the GitHub policy lane's mirrored copy of the repo's log10x/policy/actions.csv: ${GH_DEST}/log10x/policy/actions.csv, with GH_DEST defaulting to ${java.io.tmpdir}/tenx/github (pipelines/gitops/config.yaml). Each sync rewrites the copy in place, so the lookup hot-reloads. GH_ENABLED turns on with true, True or TRUE, as the lane reads it, and with the Kubernetes ConfigMap lane also on the GitHub copy wins: enable one lane.

rateReceiverDefaultAction

The action a container with no `rateReceiverActionLookupFile` row takes for its over-cap excess.

Type Default
String pass

Defines what the held lines of a container with neither its own row nor a * row in rateReceiverActionLookupFile do: one of pass, offload, tier_down, compact, sample or drop. pass leaves them where they were, so only the containers the actions file covers act on theirs. An explicit row, and then the * row, still win.

Set it beside rateReceiverAbsoluteCap so the fleet-wide cap acts on services nobody listed. Read by the cap-file regulator classes, the ones that consult the actions file. A value outside the six logs a warning at load and the regulator uses pass.

rateReceiverCapLookupFile

Declarative per-container cap file, keyed by the `rateReceiverContainerField` value.

Type Default
String

Defines the path to a declarative per-container cap file. Each entry assigns a cap, in rateReceiverUnit, to a specific container (the value of rateReceiverContainerField, which defaults to the k8s container name). The file's cap wins per-event over rateReceiverAbsoluteCap: a listed container gets its own row, an unlisted one gets the row keyed *, and a container with neither falls back to the fleet-wide cap (or to no cap, if rateReceiverAbsoluteCap is 0).

File format (CSV; header row + one comma-separated entry per row):

container,cap
<container>,<cap>[:<untilEpochSec>][:<reason>]
*,<cap>[:<untilEpochSec>][:<reason>]
  • container, value of rateReceiverContainerField for the events to cap. In k8s this is the k8s container name, stable across pod replicas. * caps every container without a row of its own.
  • cap, the cap per pattern per container per window, in rateReceiverUnit. 0 exempts the container from every cap, the * row and rateReceiverAbsoluteCap included (the over-cap branch is skipped for it). A * row of 0 exempts every container without a row of its own, so rateReceiverAbsoluteCap then applies to none of them.
  • untilEpochSec, optional Unix-epoch (seconds) expiry. Past that, the entry becomes a no-op and the fallback cap applies.
  • reason, optional free-text string for audit. Not used at runtime. Must not contain commas (would break CSV parsing).

Example:

container,cap
payment-service,4194304:1735689600:annual budget protection PAY-101
auth-service,2097152
istio-proxy,0:1735689600:exempt platform sidecar PLAT-42

Why this shape:

  • Container-keyed, matching the regulator's container axis. The file's key is the same value the regulator's counters are bucketed by, so listed caps map 1:1 to counters without translation.
  • Git-friendly: a human-readable declaration that lives in a config repo, reviewed via PR, git blame-d for who/why.
  • Self-healing: every entry can carry an explicit expiry, so forgotten caps eventually stop filtering instead of silently dropping production data forever.
  • AI-editable: the Log10x MCP writes it. Its budget run (log10x_setup_recurring) writes the file to the policy ConfigMap or commits it to the policy branch. A log10x_configure_engine call derives caps from a target percent or a monthly dollar budget and delivers them as its delivery says: a PR against the gitops repo, a write to the policy ConfigMap, or no write (stdout_only). The cap file is the interface.

Error-class lines still pass by count. Past the cap, the levels in rateReceiverKeepCounts pass that many lines per pattern per window, so a tight cap never fully silences ERROR/CRITICAL traffic.

When rateReceiverCapLookupFile is unset, every container falls back to rateReceiverAbsoluteCap (which itself defaults to 0 = disabled).

From a policy repo. With GH_ENABLED=true, the default is the GitHub policy lane's mirrored copy of the repo's log10x/policy/caps.csv: ${GH_DEST}/log10x/policy/caps.csv, with GH_DEST defaulting to ${java.io.tmpdir}/tenx/github (pipelines/gitops/config.yaml). Each sync rewrites the copy in place, so the lookup hot-reloads. GH_ENABLED turns on with true, True or TRUE, as the lane reads it, and with the Kubernetes ConfigMap lane also on the GitHub copy wins: enable one lane. The policy folder must hold caps.csv when the engine starts.

The file is hot-reloaded on in-place writes (the gitops pattern); a Kubernetes ConfigMap mount won't reload because the CM swap is a symlink rename, not an in-place write.

rateReceiverCapLookupRetain

Retention period for the per-container cap file before it is considered stale.

Type Default
Number 300000

Defines the retention period (in milliseconds) for the cap file (rateReceiverCapLookupFile). If the file's last modified time is older than this period, a warning is logged. The cap entries continue to apply; the staleness check is advisory so operators notice when an automated tooling pipeline has stopped updating the file.

Validation: Must be greater than 60000 milliseconds.

rateReceiverTotalLookupFile

Declarative per-service total file, keyed by the `rateReceiverContainerField` value; bounds what each capped service passes per window.

Type Default
String

Defines the path to a declarative per-service total file (sibling to the cap file). Each entry bounds what a service passes per window (rateReceiverResetIntervalMs) on this regulator instance, in rateReceiverUnit. Read by the cap-file regulator classes, for containers with a resolved cap.

Two rules apply once a service has a total:

  • Largest first. When the service's caps alone let more than its total through in the previous window, every pattern's cap scales by total / that amount for the whole window, or by what they let through so far this window once that is larger, so the service's largest patterns are held first and its small ones keep passing. The scale follows the previous window, so the window after a storm is scaled by the storm for its whole length and recovers the window after. The first window after a quiet one fills its total in arrival order before the scale applies. A service whose first lines of this window and of the last window it sent in are two window lengths or more apart scales by total / its units so far this window instead.
  • Hard bound. Once the lines the service passed reach its total, every further line takes the service's action (rateReceiverActionLookupFile).

The rateReceiverBaselineCount sample counts toward the total like any line, and once the total is spent the service passes at most rateReceiverBaselineMaxOverTotal more sample lines in the window. The rateReceiverKeepCounts lines count toward the total the same way, and once it is spent the service passes at most rateReceiverKeepMaxOverTotal more of them in the window. The lines an active mute entry passes always pass and count toward the total, so a muted pattern can exceed it by that much.

File format (CSV; header row + one comma-separated entry per row):

container,total
<container>,<total>[:<untilEpochSec>][:<reason>]
*,<total>[:<untilEpochSec>][:<reason>]
  • container, value of rateReceiverContainerField (the k8s container = the service), or * for every service without a row of its own.
  • total, the service's total per window on this instance, in rateReceiverUnit. 0 exempts the service; a * row of 0 gives every service without a row of its own no total.
  • untilEpochSec, optional Unix-epoch (seconds) expiry; past it the entry is a no-op.
  • reason, optional free-text for audit. Must not contain commas (would break CSV parsing).

A missing file loads as an empty table, so no service has a total until a row names it. Each regulator instance holds its own total; the MCP's budget run (log10x_setup_recurring) divides a fleet total across instances.

From a policy repo. With GH_ENABLED=true, the default is the GitHub policy lane's mirrored copy of the repo's log10x/policy/totals.csv: ${GH_DEST}/log10x/policy/totals.csv, with GH_DEST defaulting to ${java.io.tmpdir}/tenx/github (pipelines/gitops/config.yaml). Each sync rewrites the copy in place, so the lookup hot-reloads. GH_ENABLED turns on with true, True or TRUE, as the lane reads it, and with the Kubernetes ConfigMap lane also on the GitHub copy wins: enable one lane.

rateReceiverConfigGenerationFile

Optional config-generation file whose `generation` value is stamped as the `tenx_config_version` metric label.

Type Default
String

Defines the path to the config-generation file the MCP control plane (log10x_configure_engine) writes as a SIBLING of the cap file in the cap ConfigMap. It is a two-column CSV (key,value) with one row, generation,<hash-of-caps>, where the generation is a deterministic hash of the cap policy. The receiver's config-generation object reads that value and stamps it on every event as tenx_config_version (declared in this module's metricTagFields). The receiver's config_generation aggregator publishes it as one series per generation, config_generation_summaryBytes_total{tenx_config_version=...}, so the running engine advertises which config generation it loaded; the per-pattern counters do not carry it, so a policy write does not restart their series. The MCP then verifies the policy it wrote is actually live by comparing the label to hash(current caps), the config-generation closed loop.

Opt-in, and decoupled from caps. Unset by default (""); the tenx_config_version stamp is emitted only when this points at a file. A receiver running the rate regulator WITHOUT the MCP closed loop (e.g. GitOps-managed caps) leaves it unset and emits no tenx_config_version label. It never depends on the file existing, so it can't fail init on a missing file. Set it only where the MCP writes the file (via the CONFIG_GENERATION_FILE env or configGeneration.file in config.yaml).

File format (CSV; header row + one row):

key,value
generation,<12-char hash of caps.csv>

Hot-reloaded like the cap file, so a new generation written by configure_engine goes live without an engine restart. This is exactly what the verifier observes as the stale -> live transition.

From a policy repo. With GH_ENABLED=true, the default is the GitHub policy lane's mirrored copy of the repo's log10x/policy/config-generation.csv: ${GH_DEST}/log10x/policy/config-generation.csv, with GH_DEST defaulting to ${java.io.tmpdir}/tenx/github (pipelines/gitops/config.yaml). Each sync rewrites the copy in place, so the lookup hot-reloads. GH_ENABLED turns on with true, True or TRUE, as the lane reads it, and with the Kubernetes ConfigMap lane also on the GitHub copy wins: enable one lane.

rateReceiverWarmupMs

Per-container grace period before this regulator instance starts capping.

Type Default
Number 0

Defines an optional per-container grace period in milliseconds. A container is left unregulated for this long after this regulator instance first sees its events. The default is 0: a container is capped from its first line, which is what a cap meant as a budget needs.

A positive value defers cap decisions while the per-pattern counts fill. This is a delay only; no learned baseline is built, and the only state that survives is the container's first-seen timestamp.

"First sees" is per regulator instance, not per container birth. A container that has been running for hours but whose events have only just started flowing through this regulator (e.g. after a regulator restart or forwarder reconnect) restarts the warmup window. On a daemonset rolling restart the cap is therefore disabled for warmupMs per node as it cycles.

Set it for slow-ramping JVMs or workloads with long init phases whose first minutes should not be capped. Patterns that are legitimately dominant in steady state belong on the mute file, not in a warmup.

rateReceiverBaselineCount

Events per pattern kept each window regardless of share.

Type Default
Number 5

Defines how many events of each (pattern, container) are kept per window regardless of the cap and the share, so even a heavily-capped pattern leaves a small forensic sample to inspect during an incident. In a service with a total (rateReceiverTotalLookupFile) the sample counts toward it, and once the total is spent the service passes at most rateReceiverBaselineMaxOverTotal more sample lines in the window.

rateReceiverBaselineMaxOverTotal

Sample lines a service passes per window once it has passed its total.

Type Default
Number 50

Defines how many rateReceiverBaselineCount sample lines a service passes per window on this regulator instance once it has passed its total (rateReceiverTotalLookupFile). Within the total the sample passes like any line. Past it, sample lines pass up to this count, and the rest take the regular path: the keep count for the levels in rateReceiverKeepCounts, then the service's action. This bounds a service that prints many new patterns in one window, each of which would otherwise pass its sample over the total.

The count is in lines whatever rateReceiverUnit is, like the sample and the keep counts, so under bytes a service can exceed its total by this many of its own lines. 0 removes the bound, so every sample line passes. Read by the cap-file regulators, for services with a total.

rateReceiverKeepMaxOverTotal

Keep-count lines a service passes per window once it has passed its total.

Type Default
Number 500

Defines how many rateReceiverKeepCounts lines a service passes per window on this regulator instance once it has passed its total (rateReceiverTotalLookupFile). Within the total a keep-count line passes like any line. Past it, keep-count lines pass up to this count, and the rest take the service's action. This bounds a service that prints many new error patterns in one window, each of which would otherwise pass its keep count over the total.

The count is in lines whatever rateReceiverUnit is, like the keep counts, so under bytes a service can exceed its total by this many of its own lines. 0 removes the bound, so every keep-count line passes. The lines an active mute entry passes follow the entry and are not counted here. Read by the cap-file regulators, for services with a total.

rateReceiverAbsoluteCap

Fleet-wide cap per pattern per container per window; `0` (default) disables the cap unless overridden per-container via `rateReceiverCapLookupFile`.

Type Default
Number 0

Defines a fleet-wide cap, in rateReceiverUnit: the budget for a single pattern (a unique combination of rateReceiverFieldNames values) per container within the current window (rateReceiverResetIntervalMs). Applies to every container the regulator sees, unless that container has its own entry, or the * entry, in rateReceiverCapLookupFile, which wins per-event.

At or below the resolved cap the pattern is kept untouched. Above it, a line takes the container's action, except the rateReceiverBaselineCount sample, the levels in rateReceiverKeepCounts (that many lines per pattern per window), a pattern below rateReceiverMinSharePercent of its container, and the rateReceiverSeverityFloors share of the other levels.

0 (the default) disables the fleet-wide cap. With no fleet-wide cap and no per-container entry in rateReceiverCapLookupFile, the regulator's over-cap branch never engages for that container. Protection is strictly opt-in. Three configurations are first-class:

  • Per-container only: set rateReceiverCapLookupFile. Listed containers get their cap; unlisted containers are unregulated by the absolute cap. Most selective.
  • Fleet-wide only: set rateReceiverAbsoluteCap to a positive integer. Applies to every container.
  • Both: per-container entries override the fleet-wide cap; unlisted containers fall back to the fleet-wide cap (safety floor).

Example: 10485760 (10 MB, with rateReceiverUnit bytes) means a pattern is held once it has spent 10 MB in a container within the window.

With the shipped floors at 0 and the share guard off, the most one pattern in one container passes per window is the larger of the cap and the baseline sample, plus the keep count for an error-class pattern.

rateReceiverMinSharePercent

Minimum share of container volume below which a pattern is exempt from sampling.

Type Default
Number 0

Defines an optional share-of-container guard (0.0 to 1.0): if a pattern is above its cap BUT below this share of its container's volume, the regulator keeps it. The default is 0, which turns the guard off, so a cap holds every pattern it is set for.

Share is (pattern units + event) / (container units + event) over the current window, in rateReceiverUnit.

Example: 0.05 keeps a pattern above its cap while it represents less than 5% of its container's volume, for containers whose traffic is spread across many patterns.

rateReceiverIngestionCostPerGB

Vendor ingestion cost per GB in USD, for rendering savings in dollars.

Type Default
Number 1.5

Defines the cost per GB charged by your observability vendor for log ingestion. Used only to render savings metrics in dollars (dropped bytes × cost per GB); it does not affect the keep/drop decision, which is driven by the byte cap.

Common vendor pricing (2025):

  • Splunk Cloud: ~$1.50/GB (varies by contract, SKU)
  • Datadog Logs: ~$0.10-$0.25/GB (depends on tier: standard, flex, online archives)
  • Elastic Cloud: ~$0.109/GB (standard logging tier)
  • New Relic: ~$0.30/GB (Data Plus)
  • Sumo Logic: ~$1.50/GB (depends on plan)
  • AWS CloudWatch Logs: ~$0.50/GB ingestion + $0.03/GB storage
  • Azure Monitor Logs: ~$2.30/GB Analytics, $0.50/GB Basic, $0.05/GB Auxiliary (+ per-GB query fee).


This module is defined in rate/module.yaml.