Pipeline Monitor

Logs diagnostics and publishes health metrics from pipeline units.

Each unit emits counters (scanned symbols, input events) and states (scan errors), periodically logged and published to the 10x Prometheus endpoint.

Configuration

To configure the Pipeline monitor unit, Edit these settings:

Run

Launch a run pipeline to report, regulate and optimize log/trace events in edge and cloud environments.

Below is the default configuration from: monitor/config.yaml.

Edit Online

# 🔟❎ 'run' monitor configuration
#
# Emit diagnostic information and metrics to the 10x console and log.
# To learn more see https://doc.log10x.com/run/monitor/

# Set the 10x pipeline to 'run'
tenx: run

# ============================== Monitor Options ================================

monitor:

  # 'pollInterval' sets the frequency by which to emit debug information
  #  for the target units specified below.
  pollInterval: 1s

  # 'metricPattern' specifies a regex pattern to match against counters generated
  #  by the specified 'units' below to publish as metrics to the 10x console
  metricPattern: null

  # 'units' specifies the run pipeline units to emit period diagnostic
  #  information to the 10x log and console. Possible values:
  units:

    # load symbol library file(s)
    # - symbolLoader

    # load TenXTemplates file(s)
    # - templateLoader

    # load hash values of known TenXTemplates from file(s)
    # - loadTemplateHashes

    # read input log/trace event from text content passes a launch argument
    # - readInline

    # read input log/trace event from a file
    # - readFile

    # read input log/trace event from a java.io.InputStream/Reader
    # - readStream

    # split transform workload across multiple cores
    # - parallelize

    # transform input events into structured well-defined TenXObjects
    # - transform

    # group TenXObjects into composite logical units
    # - group

    # aggregate TenXObjects into summary instances
    # - aggregateSingle

    # write TenXObjects to java.io.OutputStream/Writer(s)
    # - streamOutput

    # write TenXObjects to stdout/err
    # - stdOutput

    # write TenXObjects to log4j2 appenders
    # - logOutput

    # write TenXObjects to Micrometer registries
    # - metricOutput

Compile

Launch a compile pipeline to scan source code and binary files to generate symbol unit and library files.

Below is the default configuration from: monitor/config.yaml.

Edit Online

# 🔟❎ 'compile' monitor configuration

# Emit diagnostic information and metrics to the 10x console and log.
# To learn more see https://doc.log10x.com/compile/monitor/

# Set the 10x pipeline to 'compile'
tenx: compile

# ============================== Monitor Options ================================

monitor:

  # 'pollInterval' sets the frequency by which to emit debug information
  #  for the target units specified below.
  pollInterval: null

  # 'metricPattern' specifies a regex pattern to match against counters generated
  #  by the specified 'units' below to publish as metrics to the 10x console
  metricPattern: null

  # 'units' specifies the compile pipeline units to emit period diagnostic
  #  information to the 10x log and console. Possible values:
  units:

    # Push output symbol files to a GitHub repository
    - githubPushFolder

    # discover input source code/binary files to scan on disk
    - discoverSources

    # scan input files to generate output symbol unit files
    - scanToFolder

    # launch a subprocess to scan target input files
    - subprocessScan

    # link symbol unit files into a symbol library file for use at run time
    - link

Options

Specify the options below to configure the Pipeline monitor:

Name	Description
monitorPollInterval	Interval to information to log/ publish counters
monitorUnits	Pipeline unit names to log and emit debug metrics
monitorMetricPattern	Regexp pattern to match against counters to publish as metrics

`monitorPollInterval`

Interval to information to log/ publish counters.

Type	Default
String	1s

Sets the frequency to log debug information for pipeline units specified by 'monitorUnits' to log/time series backend.

`monitorUnits`

Pipeline unit names to log and emit debug metrics.

Type	Default
List	[]

Specifies which 10x pipeline units will log and emit debug metrics while executing. Each pipeline prints the names of its configured units upon startup on the console and log file. To emit information for all pipeline units, specify '*'.

`monitorMetricPattern`

Regexp pattern to match against counters to publish as metrics.

Type	Default
String	""

Defines a pattern to match against counters debug produced by units specified by monitorUnits. Each counter provides a name in the following format: <unit name>_<key0...KeyN> (e.g., input_events_0).

The pipeline will emit counters matching the pattern, tagging each counter with information about the reporting unit instance (e.g., the destination for a 'run' pipeline output).

This unit is defined in monitor/unit.yaml.