Skip to content

Output Regulator

Filters noisy telemetry before shipping to output (Splunk, Elastic) using node-specific and environment-wide thresholds to control costs.

Output Filters

Filters provides a JavaScript-based mechanism for controlling which TenXObjects to write to output stream(s):

  • Custom constructors provide a programmatic method for dropping instances.

  • The outputFilters option provides a global hook into which regulator modules can install logic (e.g., thresholds, anomaly detection).

  • Each output stream provides an individual outputFilter setting to allow for granular control over which instances it should emit.

Filter Expressions

JavaScript expressions can be set inline within the YAML config to control which TenXObject instances to encode. For example:

outputFile:
  path: encoded.log
  filter: isObject # encode TenXObject instances only (exclude TenXSummaries and templates)
  ...

The following expressions provide examples for different types of filters:

Expression Description
isObject Checks if the instance TenXObject (exclude templates and summaries).
isEncoded Checks if the instance has been encoded.
isTemplate Checks if the instance is a TenXTemplate instance.
isSummary Identifies if the object is a TenXSummary instances.
timestamped Checks if the instance has timestamps.
ipAddress == TenXEnv.get("host") Matches the object's IP address with a host env variable
inputName == "myElastic" Filters objects with the input name "myElastic".
startsWith("ERROR") Filters objects starting with "ERROR".
myFilter() Applies a custom filter function.

Modules

  • Rate Regulator

    Filter TenXObjects from output based on cost-aware sampling using local or global budget policies.

    More info

  • Policy Regulator Lookup Inputs

    Generate a regulator event rate policy lookup file.

    More info

Options

Specify the options below to configure the Output regulator:

Name Description
outputFilters JavaScript expressions an TenXObject must evaluate as truthy against to write to output(s)

outputFilters

JavaScript expressions an TenXObject must evaluate as truthy against to write to output(s).

Type Default
List []

Provides a list of 10x JavaScript expressions that must ALL evaluate as truthy to write a target TenXObject to output.

The current TenXObject's outputType and outputPath intrinsic fields provide information about the output for which the current TenXObject is a candidate.

This option is primarily used by 10x modules regulator modules to provide control over the volume of data is written out by the pipeline to target output(s).

Most output modules that publish to an event (e.g., Splunk) or time-series (e.g., Prometheus) destination provide dedicated filter options to control which TenXObjects to write.

For example, the Datadog metric output provides the datadogFilter option to allow granular control over which TenXSummary instances to write to it.


This module is defined in regulate/unit.yaml.