Skip to content

Aggregator

Groups TenXObjects by specified field values and generates TenXSummary instances for metric outputs (Prometheus, Datadog, CloudWatch).

How It Works

Each configured aggregator maintains an in-memory table where each row correlates to a unique combination of field values extracted from input TenXObjects.

When the number of TenXObjects upserted into the table exceeds a target threshold (e.g., 1000) or a specified interval (e.g., 2s) elapses, the aggregator creates a TenXSummary instance for each of the table's rows and flushes it to output. Once summaries are flushes, in-memory table is cleared for the next set of TenXObjects to aggregate, and so on.

Workflow

The aggregation process follows these key steps:

graph LR
    A["<div style='font-size: 14px;'>📥 Receive</div><div style='font-size: 10px; text-align: center;'>TenXObjects</div>"] --> B["<div style='font-size: 14px;'>🔍 Extract</div><div style='font-size: 10px; text-align: center;'>Field Values</div>"]
    B --> C["<div style='font-size: 14px;'>🗃️ Group</div><div style='font-size: 10px; text-align: center;'>In-Memory Table</div>"]
    C --> D{"<div style='font-size: 14px;'>⏰ Trigger</div><div style='font-size: 10px; text-align: center;'>Threshold/Interval</div>"}
    D -->|Yes| E["<div style='font-size: 14px;'>📊 Generate</div><div style='font-size: 10px; text-align: center;'>TenXSummaries</div>"]
    D -->|No| C
    E --> F["<div style='font-size: 14px;'>📤 Flush</div><div style='font-size: 10px; text-align: center;'>To Output</div>"]
    F --> G["<div style='font-size: 14px;'>🧹 Clear</div><div style='font-size: 10px; text-align: center;'>Table</div>"]
    G --> A

    classDef receive fill:#7c3aed88,stroke:#6d28d9,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef extract fill:#9333ea88,stroke:#7c3aed,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef group fill:#2563eb88,stroke:#1d4ed8,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef trigger fill:#059669,stroke:#047857,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef generate fill:#dc2626,stroke:#b91c1c,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef flush fill:#ea580c88,stroke:#c2410c,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef clear fill:#7c2d12,stroke:#9a3412,color:#ffffff,stroke-width:2px,rx:8,ry:8

    class A receive
    class B extract
    class C group
    class D trigger
    class E generate
    class F flush
    class G clear

📥 Receive TenXObjects: Receives TenXObjects from the processing pipeline

🔍 Extract Field Values: Extracts configured field values from each TenXObject for grouping

🗃️ Group in Table: Maintains in-memory table where rows correspond to unique field value combinations

⏰ Check Triggers: Monitors flush threshold and flush interval conditions

📊 Generate Summaries: Creates TenXSummary instances from each table row when triggered

📤 Flush to Output: Sends TenXSummaries to configured outputs for metric publishing

🧹 Clear Table: Resets in-memory table to begin next aggregation cycle

Grouped Instances

Aggregators operate on grouped instances (e.g., stack traces) as a single logical unit allowing them to be reported collectively vs. aggregating each element. b (e.g., tallying stack traces jointly vs. generating a metric for each separate line) .

Summary Fields

Each generated TenXSummary instance provides access to the following intrinsic fields:

Field Description
summaryVolume Number of TenXObjects aggregated into this summary instance
summaryBytes Accumulative number of bytes in the text field value of TenXObjects aggregated into this instance
summaryValues Array of values by which TenXObjects were aggregated into this instance. This value commonly specifies metric tags when writing TenXSummaries to time-series outputs.
summaryTotals The sum values of the aggregatorTotalFields value of TenXObjects aggregated into this instance. This value commonly specifies metric counter values when writing TenXSummaries to time-series outputs.
summaryValuesHash Hash value of the summaryValues field, providing a concise alphanumeric hash representation of the values of TenXObjects aggregated into this TenXSummary instance.

Configuration

To configure the Aggregator unit, Edit these settings:

Dev

Dev

Below is the default configuration from: dev/config.yaml.

Edit Online

Edit config.yaml Locally 

# 🔟❎ 'run' dev app aggregator config

# This file configures aggregators for the dev app.
# To learn more see https://doc.log10x.com/run/aggregate/

# Set the 10x pipeline to 'run'
tenx: run

# ============================== Aggregator Options ===========================

# Multiple aggregators can be defined below
aggregator:

  - name: devAggregator

    # 'filter' sets a JavaScript expression that TenXObjects must evaluate as truthy to be aggregated.
    #  To learn more see https://doc.log10x.com/run/output/regulate/#filter-expressions
    filter: null

    # 'fields' set the names of intrinsic/extracted/calculated fields whose combined values
    #  identify a unique row within the aggregator's in-mem table for
    #  each object passed to it. This aggregator defaults to enrichment fields.
    #  To learn more see https://doc.log10x.com/run/initialize/#enrichmentFields
    fields:
      - $=yield TenXEnv.get("enrichmentFields")

    # 'totalFields' specifies fields to extract and tally into generated TenXSummaries
    totalFields: []

    # 'flushThreshold' sets the max number of TenXObjects whose values to upsert into
    #  the aggregator's in-mem table. Once exceeded, the aggregator instantiates TenXSummaries 
    #  from the table rows to send to output and reset the table.
    flushThreshold: 0 

    # 'flushInterval' sets a periodic interval, after which the aggregator
    #  flushes the rows of its in-mem table into the pipeline as TenXSummary instances.
    flushInterval: null

    # 'maxCardinality' sets the maximum number of rows for the aggregator's
    #  in-mem table. If an TenXObjects whose 'values' are not contained in the table
    #  is passed to the aggregator, 'defaultValue' will be used for it instead
    #  until the aggregator flushes its in-mem table after 'flushInterval' or 'flushThreshold'
    #  are exceeded.
    maxCardinality: 65536

    # 'defaultValue' sets a constant value that is used instead of 'values' when
    #  'maxCardinality' is exceeded within 'flushInterval'
    defaultValue: unknown
Report

Report

Below is the default configuration from: report/config.yaml.

Edit Online

Edit config.yaml Locally 

# 🔟❎ 'run' aggregator config for the reporter app.

# This file configures aggregators for tracking emitted (non-dropped) events.
# To learn more see https://doc.log10x.com/run/aggregate

# Set the 10x pipeline to 'run'
tenx: run

# ============================== Aggregator Options ===========================

# Multiple aggregators can be defined below
aggregator:

  # Defines an aggregator for tracking emitted (non-dropped) events
  # This enables an output metric to show an accurate (lossless) metric of the
  # data actually emitted to downstream systems.
  - name: emitted_events

    # 'filter' sets a JavaScript expression that TenXObjects must evaluate as truthy to be aggregated.
    #  To learn more see https://doc.log10x.com/run/output/regulate/#filter-expressions
    #
    # This filter excludes dropped events by checking the 'isDropped' property,
    # ensuring only emitted events are tracked.
    #
    # For more info, see https://doc.log10x.com/api/js/#TenXBaseObject+dropped
    filter: "this.isObject && !this.isDropped"

    # 'fields' set the names of intrinsic/extracted/calculated fields whose combined values
    #  identify a unique row within the aggregator's in-mem table for
    #  each object passed to it. This aggregator defaults to enrichment fields.
    #  To learn more see https://doc.log10x.com/run/initialize/#enrichmentFields
    fields:
      - $=yield TenXEnv.get("enrichmentFields")

    # 'totalFields' specifies fields to extract and tally into generated TenXSummaries
    totalFields: []

    # 'flushThreshold' sets the max number of TenXObjects whose values to upsert into
    #  the aggregator's in-mem table. Once exceeded, the aggregator instantiates TenXSummaries
    #  from the table rows to send to output and reset the table.
    flushThreshold: 10000

    # 'flushInterval' sets a periodic interval, after which the aggregator
    #  flushes the rows of its in-mem table into the pipeline as TenXSummary instances.
    flushInterval: 2s

    # 'maxCardinality' sets the maximum number of rows for the aggregator's
    #  in-mem table. If an TenXObjects whose 'values' are not contained in the table
    #  is passed to the aggregator, 'defaultValue' will be used for it instead
    #  until the aggregator flushes its in-mem table after 'flushInterval' or 'flushThreshold'
    #  are exceeded.
    maxCardinality: 65536

    # 'defaultValue' sets a constant value that is used instead of 'values' when
    #  'maxCardinality' is exceeded within 'flushInterval'
    defaultValue: unknown
Regulate

Regulate

Below is the default configuration from: regulate/config.yaml.

Edit Online

Edit config.yaml Locally 

# 🔟❎ 'run' aggregator config for the regulator app.

# This file configures aggregators for tracking emitted (non-dropped) events with
# additional metrics for tracking regulated (dropped) events for cost saving analysis.
# To learn more see https://doc.log10x.com/run/aggregate

# Set the 10x pipeline to 'run'
tenx: run

# ============================== Aggregator Options ===========================

# Multiple aggregators can be defined below
aggregator:

  # Defines an aggregator for tracking emitted (non-dropped) events
  # This enables an output metric to show an accurate (lossless) metric of the
  # data actually emitted to downstream systems.
  - name: emitted_events

    # 'filter' sets a JavaScript expression that TenXObjects must evaluate as truthy to be aggregated.
    #  To learn more see https://doc.log10x.com/run/output/regulate/#filter-expressions
    #
    # This filter excludes dropped events by checking the 'isDropped' property,
    # ensuring only emitted events are tracked.
    #
    # For more info, see https://doc.log10x.com/api/js/#TenXBaseObject+dropped
    filter: "this.isObject && !this.isDropped"

    # 'fields' set the names of intrinsic/extracted/calculated fields whose combined values
    #  identify a unique row within the aggregator's in-mem table for
    #  each object passed to it. This aggregator defaults to enrichment fields.
    #  To learn more see https://doc.log10x.com/run/initialize/#enrichmentFields
    fields:
      - $=yield TenXEnv.get("enrichmentFields")

    # 'totalFields' specifies fields to extract and tally into generated TenXSummaries
    totalFields: []

    # 'flushThreshold' sets the max number of TenXObjects whose values to upsert into
    #  the aggregator's in-mem table. Once exceeded, the aggregator instantiates TenXSummaries
    #  from the table rows to send to output and reset the table.
    flushThreshold: 10000

    # 'flushInterval' sets a periodic interval, after which the aggregator
    #  flushes the rows of its in-mem table into the pipeline as TenXSummary instances.
    flushInterval: 2s

    # 'maxCardinality' sets the maximum number of rows for the aggregator's
    #  in-mem table. If an TenXObjects whose 'values' are not contained in the table
    #  is passed to the aggregator, 'defaultValue' will be used for it instead
    #  until the aggregator flushes its in-mem table after 'flushInterval' or 'flushThreshold'
    #  are exceeded.
    maxCardinality: 65536

    # 'defaultValue' sets a constant value that is used instead of 'values' when
    #  'maxCardinality' is exceeded within 'flushInterval'
    defaultValue: unknown

  # Defines an aggregator for all events seen by this pipeline
  # This enables an output metric to show an accurate (lossless) metric of the
  # data processed by this pipeline, even if it was later dropped.
  - name: all_events

    # 'filter' sets a JavaScript expression that TenXObjects must evaluate as truthy to be aggregated.
    #  To learn more see https://doc.log10x.com/run/output/regulate/#filter-expressions
    filter: isObject

    # 'values' set the names of intrinsic/extracted/calculated fields whose combined values
    #  identify a unique row within the aggregator's in-mem table for
    #  each object passed to it.

    # By default, this configuration uses the enrichment fields specified below.
    fields:
      - $=yield TenXEnv.get("enrichmentFields")

    # 'totalFields' specifies fields to extract and tally into generated TenXSummaries
    totalFields: []

    # 'flushThreshold' sets the max number of TenXObjects whose values to upsert into
    #  the aggregator's in-mem table. Once exceeded, the aggregator instantiates TenXSummaries
    #  from the table rows to send to output and reset the table.
    flushThreshold: 10000

    # 'flushInterval' sets a periodic interval, after which the aggregator
    #  flushes the rows of its in-mem table into the pipeline as TenXSummary instances.
    flushInterval: 2s

    # 'maxCardinality' sets the maximum number of rows for the aggregator's
    #  in-mem table. If an TenXObjects whose 'values' are not contained in the table
    #  is passed to the aggregator, 'defaultValue' will be used for it instead
    #  until the aggregator flushes its in-mem table after 'flushInterval' or 'flushThreshold'
    #  are exceeded.
    maxCardinality: 65536

    # 'defaultValue' sets a constant value that is used instead of 'values' when
    #  'maxCardinality' is exceeded within 'flushInterval'
    defaultValue: unknown
Optimize

Optimize

Below is the default configuration from: optimize/config.yaml.

Edit Online

Edit config.yaml Locally 

# 🔟❎ 'run' aggregator config for the optimizer app.

# This file configures aggregators for tracking emitted (non-dropped) events with
# additional metrics for optimization analysis and regulated (dropped) events for
# cost saving analysis.
# To learn more see https://doc.log10x.com/run/aggregate

# Set the 10x pipeline to 'run'
tenx: run
# ============================== Aggregator Options ===========================

# Multiple aggregators can be defined below
aggregator:

  # Defines an aggregator for tracking optimized (emitted) events with size metrics
  # This enables output metrics to show both event counts and optimized byte sizes,
  # providing visibility into optimization effectiveness.
  - name: emitted_events

    # 'filter' sets a JavaScript expression that TenXObjects must evaluate as truthy to be aggregated.
    #  To learn more see https://doc.log10x.com/run/output/regulate/#filter-expressions
    #
    # This filter excludes dropped events, tracking only optimized events that are emitted.
    # For more info on output filters see https://doc.log10x.com/run/output/#output-filters
    #
    filter: "this.isObject && !this.isDropped"

    # 'fields' set the names of intrinsic/extracted/calculated fields whose combined values
    #  identify a unique row within the aggregator's in-mem table for
    #  each object passed to it. This aggregator defaults to enrichment fields.
    #  To learn more see https://doc.log10x.com/run/initialize/#enrichmentFields
    fields:
      - $=yield TenXEnv.get("enrichmentFields")

    # 'totalFields' specifies fields to extract and tally into generated TenXSummaries
    totalFields:
      # Calculates the utf8 byte size of the encoded/optimized event, providing metrics
      # on the size of optimized events for measuring optimization effectiveness.
      # To learn more see https://doc.log10x.com/api/js/#TenXObject+encode
      - optimized_size=this.utf8Size("encoded=encode()")

    # 'flushThreshold' sets the max number of TenXObjects whose values to upsert into
    #  the aggregator's in-mem table. Once exceeded, the aggregator instantiates TenXSummaries
    #  from the table rows to send to output and reset the table.
    flushThreshold: 10000

    # 'flushInterval' sets a periodic interval, after which the aggregator
    #  flushes the rows of its in-mem table into the pipeline as TenXSummary instances.
    flushInterval: 2s

    # 'maxCardinality' sets the maximum number of rows for the aggregator's
    #  in-mem table. If an TenXObjects whose 'values' are not contained in the table
    #  is passed to the aggregator, 'defaultValue' will be used for it instead
    #  until the aggregator flushes its in-mem table after 'flushInterval' or 'flushThreshold'
    #  are exceeded.
    maxCardinality: 65536

    # 'defaultValue' sets a constant value that is used instead of 'values' when
    #  'maxCardinality' is exceeded within 'flushInterval'
    defaultValue: unknown

  # Defines an aggregator for all events seen by this pipeline
  # This enables an output metric to show an accurate (lossless) metric of the
  # data processed by this pipeline, even if it was later dropped.
  - name: all_events

    # 'filter' sets a JavaScript expression that TenXObjects must evaluate as truthy to be aggregated.
    #  To learn more see https://doc.log10x.com/run/output/regulate/#filter-expressions
    filter: isObject

    # 'values' set the names of intrinsic/extracted/calculated fields whose combined values
    #  identify a unique row within the aggregator's in-mem table for
    #  each object passed to it.

    # By default, this configuration uses the enrichment fields specified below.
    fields:
      - $=yield TenXEnv.get("enrichmentFields")

    # 'totalFields' specifies fields to extract and tally into generated TenXSummaries
    totalFields: []

    # 'flushThreshold' sets the max number of TenXObjects whose values to upsert into
    #  the aggregator's in-mem table. Once exceeded, the aggregator instantiates TenXSummaries
    #  from the table rows to send to output and reset the table.
    flushThreshold: 10000

    # 'flushInterval' sets a periodic interval, after which the aggregator
    #  flushes the rows of its in-mem table into the pipeline as TenXSummary instances.
    flushInterval: 2s

    # 'maxCardinality' sets the maximum number of rows for the aggregator's
    #  in-mem table. If an TenXObjects whose 'values' are not contained in the table
    #  is passed to the aggregator, 'defaultValue' will be used for it instead
    #  until the aggregator flushes its in-mem table after 'flushInterval' or 'flushThreshold'
    #  are exceeded.
    maxCardinality: 65536

    # 'defaultValue' sets a constant value that is used instead of 'values' when
    #  'maxCardinality' is exceeded within 'flushInterval'
    defaultValue: unknown
Index

Index

Below is the default configuration from: index/config.yaml.

Edit Online

Edit config.yaml Locally 

# 🔟❎ 'run' index aggregator config

# This file configures the aggregators for the Storage Streamer app Index component.
# To learn more see https://doc.log10x.com/run/aggregate

# Set the 10x pipeline to 'run'
tenx: run

# ============================== Aggregator Options ===========================

# Multiple aggregators can be defined below
aggregator:

  # Defines an aggregator for all events seen by this pipeline
  # This enables an output metric to show an accurate (lossless) metric of the
  # data processed by this pipeline, even if it was later dropped.
  - name: indexed_events

    # 'filter' sets a JavaScript expression that TenXObjects must evaluate as truthy to be aggregated.
    #  To learn more see https://doc.log10x.com/run/output/regulate/#filter-expressions
    filter: isObject

    # 'fields' set the names of intrinsic/extracted/calculated fields whose combined values
    #  identify a unique row within the aggregator's in-mem table for
    #  each object passed to it. This aggregator defaults to enrichment fields.
    #  To learn more see https://doc.log10x.com/run/initialize/#enrichmentFields
    fields:
      - $=yield TenXEnv.get("enrichmentFields")

    # 'totalFields' specifies fields to extract and tally into generated TenXSummaries
    totalFields: []

    # 'flushThreshold' sets the max number of TenXObjects whose values to upsert into
    #  the aggregator's in-mem table. Once exceeded, the aggregator instantiates TenXSummaries
    #  from the table rows to send to output and reset the table.
    flushThreshold: 10000

    # 'flushInterval' sets a periodic interval, after which the aggregator
    #  flushes the rows of its in-mem table into the pipeline as TenXSummary instances.
    flushInterval: 2s

    # 'maxCardinality' sets the maximum number of rows for the aggregator's
    #  in-mem table. If an TenXObjects whose 'values' are not contained in the table
    #  is passed to the aggregator, 'defaultValue' will be used for it instead
    #  until the aggregator flushes its in-mem table after 'flushInterval' or 'flushThreshold'
    #  are exceeded.
    maxCardinality: 65536

    # 'defaultValue' sets a constant value that is used instead of 'values' when
    #  'maxCardinality' is exceeded within 'flushInterval'
    defaultValue: unknown
Stream

Stream

Below is the default configuration from: stream/config.yaml.

Edit Online

Edit config.yaml Locally 

# 🔟❎ 'run' Query Stream application settings

# This configuration defines the settings for use by the Query Stream operation of the Streamer app.
# To learn more see https://doc.log10x.com/apps/cloud/analyzer/#query

# Set the 10x pipeline to 'run'
tenx: run

# ============================== Aggregator Options ===========================

# Multiple aggregators can be defined below
aggregator:

  # Defines an aggregator for all events seen by this pipeline
  # This enables an output metric to show an accurate (lossless) metric of the
  # data processed by this pipeline, even if it was later dropped.
  - name: streamed_events

    # 'filter' sets a JavaScript expression that TenXObjects must evaluate as truthy to be aggregated.
    #  To learn more see https://doc.log10x.com/run/output/regulate/#filter-expressions
    filter: "this.isObject && !this.isDropped"

    # 'values' set the names of intrinsic/extracted/calculated fields whose combined values
    #  identify a unique row within the aggregator's in-mem table for
    #  each object passed to it.

    # By default, this configuration uses the enrichment fields specified below.
    fields:
      - $=yield TenXEnv.get("enrichmentFields")

    # 'totalFields' specifies fields to extract and tally into generated TenXSummaries
    totalFields: []

    # 'flushThreshold' sets the max number of TenXObjects whose values to upsert into
    #  the aggregator's in-mem table. Once exceeded, the aggregator instantiates TenXSummaries
    #  from the table rows to send to output and reset the table.
    flushThreshold: 10000

    # 'flushInterval' sets a periodic interval, after which the aggregator
    #  flushes the rows of its in-mem table into the pipeline as TenXSummary instances.
    flushInterval: 2s

    # 'maxCardinality' sets the maximum number of rows for the aggregator's
    #  in-mem table. If an TenXObjects whose 'values' are not contained in the table
    #  is passed to the aggregator, 'defaultValue' will be used for it instead
    #  until the aggregator flushes its in-mem table after 'flushInterval' or 'flushThreshold'
    #  are exceeded.
    maxCardinality: 65536

    # 'defaultValue' sets a constant value that is used instead of 'values' when
    #  'maxCardinality' is exceeded within 'flushInterval'
    defaultValue: unknown

Options

Specify the options below to configure multiple Aggregator:

Name Description
aggregatorName Logical name of aggregator
aggregatorEnabled Enable/disable this aggregator
aggregatorFields List of field names by which to aggregate TenXObjects
aggregatorDefaultValue A constant value to default if 'maxCardinality' is exceeded
aggregatorTotalFields List of fields whose values to tally into generated TenXSummaries
aggregatorMaxCardinality Max number of concurrent rows within the aggregator in-mem table within the flush interval
aggregatorFilter A JavaScript expression an TenXObject must evaluate as 'truthy' against to be aggregated
aggregatorFlushThreshold Maximum the number of TenXObjects to aggregate before the aggregator flushes
aggregatorFlushInterval Interval to wait before the aggregator flushes

aggregatorName

Logical name of aggregator.

Type Required
String ✔

Provides a logical name for this aggregator (e.g., 'errors'). This value is accessible at run-time via the intrinsic 'inputName' field to identify TenXSummaries originating from this aggregator.

aggregatorEnabled

Enable/disable this aggregator.

Type Default
String true

sets whether to open the input stream.

To enable this input only when an 'aggregateErrors' startup argument value is truthy:

...
aggregator:
  enabled: $=TenXEnv.get("aggregateErrors")

aggregatorFields

List of field names by which to aggregate TenXObjects.

Type Required
List ✔

Provides a list of intrinsic/extracted/calculated TenXObject field names by which to aggregate instances.

When the aggregator flushes the in-mem table to produced TenXSummaries, each generated TenXSummary instance provides access to the total values via its summaryTotals array field.

aggregatorDefaultValue

A constant value to default if 'maxCardinality' is exceeded.

Type Default
String ""

Defines a constant value to use instead of field values when the current number of rows within the aggregator's in-mem table during flushInterval exceeds maxCardinality.

This value prevents an out-of-mem error if TenXObjects whose field values are of high cardinality, which can cause the in-mem table to grow beyond the confines of the host JVM heap.

This error is possible if fields contains a calculated/extracted field whose value is alphanumeric (e.g., 'spanID') and is practically unique to each TenXObject.

aggregatorTotalFields

List of fields whose values to tally into generated TenXSummaries.

Type Default
List []

Specifies a list of intrinsic calculated/extracted TenXObject field names to tally and write into aggregated TenXSummary instances within a target flush threshold or interval.

For example, setting a value of: price into this array will tally the prices of all TenXObjects aggregated into a target output TenXSummary instance.

A matching price value containing the sum of aggregated prices will be accessible for each TenXSummary instance via its summaryTotals array.

aggregatorMaxCardinality

Max number of concurrent rows within the aggregator in-mem table within the flush interval.

Type Default
Number 0

Sets the maximum number of unique rows within the aggregator's in-mem table. If the number of rows exceeds this value during flushInterval , the 'aggregatorDefaultValue' value will be for all TenXObjects instead of field values until the aggregator is flushed.

This value is used as a backstop to ensure that the aggregator's in-mem table does not overflow when extracting and aggregating very high cardinality values from the fields of TenXObjects passed to this aggregator.

If no value is provided or set to 0, the aggregator has no cap. This may result in an out-of-mem error when very high cardinality (e.g., 'spanID', 'GUID') fields are specified.

aggregatorFilter

A JavaScript expression an TenXObject must evaluate as 'truthy' against to be aggregated.

Type Default
String ""

Provides a way to control which TenXObjects are incorporated into this aggregator. Unlike the 'aggregatorEnabled', which enables/disables this aggregator altogether, this JavaScript expression is evaluated per TenXObject to provide granular control over what information to aggregate.

For example, to only aggregate TenXObjects read from a specified input:

aggregator:
  filter: this.inputName == "elastic"

To apply more complex logic, invoke an 10x JavaScript function declared in a .js file by passing it to the 10x Engine at startup (e.g., $ tenx run @my.js).

aggregator:
  ...
  filter: myAggregatorFilter()

For an example function, see policy-regulator-object.js.

aggregatorFlushThreshold

Maximum the number of TenXObjects to aggregate before the aggregator flushes.

Type Default
Number 0

Controls how many TenXObjects to upsert into this aggregator (e.g., 10000) before the aggregator transforms the rows of its in-mem table into TenXSummaries instances and flushes them into the pipeline.

If not set, the flushInterval argument determines when to flush the aggregator. If neither argument is present, the aggregator only flushes pending TenXSummaries when the pipeline terminates.

aggregatorFlushInterval

Interval to wait before the aggregator flushes.

Type Default
String ""

Sets the period (e.g., '30s'), after which the aggregator transforms the rows of its in-mem table into TenXSummaries instances and flushes them into the pipeline.

If set, the flushThreshold argument determines when to flush the aggregator. If neither argument is present, the aggregator only flushes pending TenXSummaries when the pipeline terminates.


This unit is defined in aggregate/unit.yaml.