Group Sequencer

Groups TenXObjects for filtering, aggregation, and output as single units.

Log events read from an input stream can serve as a part of a larger logical group. A typical example of events spanning multiple sub-events are stack traces where each line within the stack trace may be logged as a separate line.

Grouping enables:

Identify groups consuming the most storage and analytics resources using aggregators. This is especially valuable when storing stack traces than span 100s of lines and consume a significant amount of resources.
Filter unnecessary groups such as 'noisy' stack traces via group filters and output regulators.
Optimize storage of multi-line events by losslessly compacting them as a composite instances to reduce storage footprint by > 75% when compared to storing individual events.

Group Heads

tenXObjects which evaluate as truthy against groupExpressions are marked as starting of a new group (i.e. a group head).

All subsequent TenXObjects read from the same input will join the current group until either:

Another group is started marked by a subsequent instance which evaluates as truthy against groupExpressions.
The number of TenXObjects in the current group exceeds groupMaxSize.
The groupFlushTimeout elapses.

At that point the group is sealed as new composite TenXObject and flushed forward for aggregation and output. Each composite TenXObject returns the number of instances grouped within it via the groupSize member.

Example #1 - timestamps + negators (default)

The TenXObject constructor below marks an instance as the head of a group if:

its text field starts with an indicator value.
its timestamped field is true.

export class GroupTemplate extends TenXTemplate {

    static get isGroup() {

        // https://doc.log10x.com/run/initialize/group/#groupindicators
        if (this.startsWith(TenXEnv.get("groupIndicators"))) {
            return true;
        }

        return this.timestamped;
    }

    // This constructor is invoked by the engine once for each unique TenXTemplate discovered
    // at runtime based on log event structures
    constructor() {
        GroupTemplate.isGroup = this.isGroup();
     }
}

Example #2 - Group ISO_8601 events

The TenXObject below marks an instance as the head of a group if it has an ISO_8601 timestamp:

export class IsoTemplate extends TenXTemplate {

  constructor() {
    IsoTemplate.isGroup = this.timestampFormat() == "yyyy-MM-dd'T'HH:mm'Z'";
  }
}

Example #3 - Group Linux Call Traces

The constructor below groups Linux Call Traces (see example) by folding lines that are part of a trace (e.g., an interrupt marker, or end with a memory address) into a logical group.

class NixTemplate extends TenXTemplate {

  constructor() {  

   // is event is call trace interrupt marker
    if (this.contains("<IRQ>")) {
        NixTemplate.isGroup = false;
    } else 

    // does event end in a '/'' + hex memory address (e.g., 0x16d0) ?
    // use the token() function to access the last + penultimate instance values
    if (this.token(-2) == "/") && (startsWith(this.token(-1), "0x")) {
        NixTemplate.isGroup = false;
    }
  }
}

Group Filters

The groupFilters option provides a mechanism for filtering TenXObject groups based on conditions that relate to the entire group (e.g., filter entire stack traces vs. individual lines).

The YAML config below places a limit of a maximum of 'SocketException' 1000 groups per minute using a cyclical counter:

group:
  filters: 'this.includes("SocketException") && (this.groupSize > 1) ? (TenXCounter.incAndGet("socketException", 1, "1m") > 1000) : true'

A filter may also be set to a function loaded from a JavaScript config file. For example:

group:
  filters: socketExceptionFilter()

Where the JavaScript file would contain:

// @loader: tenx

public class MyFilter extends TenXObject {

  function socketExceptionFilter() {

    return this.includes("SocketException") && (this.groupSize > 1) ?
      (TenXCounter.incAndGet("socketException", 1, "1m") > 1000) : 
      true
  }

}

Event Source

Input streams that read data from multiple locations (e.g., log files, pods, hosts) can utilize source patterns and fields to assign each event a logical origin (e.g., log file, host address).

This value ensure each TenXObject is grouped alongside instances from the same source and is accessible via the source member.

Configuration

To configure the Group sequencer unit, Edit these settings.

Below is the default configuration from: group/config.yaml.

Edit Online

# 🔟❎ 'run' event grouping configuration

# Group  sequences of TenXObjects to filter, aggregate and output as a single logical unit.
# To learn more see https://doc.log10x.com/run/transform/group/

# Set the 10x pipeline to 'run'
tenx: run

# =============================== Group Options ===============================

group:

  # 'filters' specify JavaScript expressions an TenXObject instance/group must 
  #  evaluate as truthy against to be written to output
  filters: []

  # 'maxSize' defines the maximum number of TenXObjects to group
  #  before the group is sealed and forwarded into the pipeline. 
  #  Subsequent TenXObjects can form a new group.
  maxSize: 20000

  # 'flushTimeout' defines the max interval (e.g., 10s) to wait for 
  #  new events to be read from an input stream before it flushes any
  #  pending TenXObjects group into the pipeline.
  #  This mechanism is designed to avoid latencies in dispatching pending event
  #  groups to output destinations.
  flushTimeout: $=parseDuration("5s")

  # 'async' specifies whether to sequence and group TenXObjects in a dedicated thread
  async: true

Options

Specify the options below to configure the Group sequencer:

Name	Description
groupMaxSize	Max number of objects to place in a group
groupMaxSize	Max number of objects to place in a group
groupFlushTimeout	Interval to flush a pending TenXObjects group
groupExpressions	JavaScript expressions TenXObject instance/group must evaluate as truthy against to be set as group head
groupFilters	JavaScript expressions TenXObject instance/group must evaluate as truthy against
groupAsync	Group TenXObjects in a dedicated thread

`groupMaxSize`

Max number of objects to place in a group.

Type	Default
Number	20000

Sets the maximum number of objects to place under one group before starting a new group. Set 0 to unlimited.

`groupMaxSize`

Max number of objects to place in a group.

Type	Default
Number	20000

Sets the maximum number of objects to place under one group before starting a new group. Set 0 to unlimited.

`groupFlushTimeout`

Interval to flush a pending TenXObjects group.

Type	Default
String	5sec

Sets the max interval after which to flush pending TenXObjects into the pipeline. This option provides a timeout period to flush the current group head and composed TenXObjects into the pipeline for aggregation, filtering, and encoding to output.

`groupExpressions`

JavaScript expressions TenXObject instance/group must evaluate as truthy against to be set as group head.

Type	Default
String	""

Specifies a list of JavaScript expressions that must all evaluate as truthy for the target TenXObject to be classified as group head.

For example, the group initializer module calculates a TenxTemplate variable to determine whether instances of a target TenXTemplate represent group heads based on a number of heuristic indicators and sets the calculated field name (e.g., GroupTemplate.isGroup) holding the resulting value into this array.

Using TenXTemplate variables allows for computing this state once for all instances of a target event.

`groupFilters`

JavaScript expressions TenXObject instance/group must evaluate as truthy against.

Type	Default
List	[]

Specifies a list of JavaScript expressions that must all evaluate as truthy for the target object/group to be part of a sequence aggregated/written to output.

To learn more see group filters.

`groupAsync`

Group TenXObjects in a dedicated thread.

Type	Default
Boolean	true

Specifies whether to perform TenXObject grouping logic in a dedicated thread.

This unit is defined in group/unit.yaml.