Process Output

Spawns a process and writes TenXObject instance and template field values to its stdin.

This output enables the runtime to leverage external processes (e.g., FluentBit) as a means of transmitting processed data to local and remote destinations.

Configuration

To configure the Process output module, Edit these settings.

Below is the default configuration from: process/config.yaml (* Required Fields).

Edit Online Edit Locally

Edit config.yaml Locally

Download and save to:

• Linux / Docker / macOS
• Custom directory
  $TENX_CONFIG/run/output/event/process/config.yaml
• Within cloned repo
  ./pipelines/run/output/event/process/config.yaml

Download

# 🔟❎ 'run' Fluent Bit output configuration

# Launch a Fluent Bit process into whose stdin pipe to write TenXObject instance/template field values.
# To learn more see https://doc.log10x.com/run/output/event/process/

# Set the 10x pipeline to 'run'
tenx: run

# =============================== Dependencies ================================

include: run/modules/output/event/process

# ============================ Fluent Bit Options =============================

# Multiple sub-process outputs can be defined below

procOut:

  # ----------------------------- Object Output -------------------------------

   # 'writeObjects' controls whether to write TenXObject field values to output.
  - writeObjects: false # (❗ REQUIRED)

    # -------------------------- Process Options ------------------------------

    # 'command' defines the OS-resolvable path of the sub-process to spawn
    command: /opt/fluent-bit/bin/fluent-bit

    args:
      - -i
      - stdin
      - -o
      - stdout

    # 'destroyMode' is set to 'cached' to allow a fluent-bit sub-process to
    #  to be reused across multiple 10x pipeline executions within a host JVM.
    destroyMode: cached

    # 'maxCacheSize' sets the size of the cache to used for caching fluent-bit instances
    maxCacheSize: 1

    # 'lazyLaunch' determines whether the fluent-bit instance will launch immediately
    # during the pipeline initialization, or only on-demand.
    lazyLaunch: true

    # 'startupWaitMs' and 'startupWaitPattern' controls the wait for the fluent-bit instance
    # to be ready for receiving events, before starting to emit to it.
    startupWaitMs: $=parseDuration("5s")
    startupWaitPattern: \[\s*sp\s*]\s+stream\s+processor\s+started\b

    # ------------------------------ Encoding ---------------------------------

    # 'filter' specifies an expression that any TenXObject passed to this output must
    #  evaluate as truthy against to write its instance/template values to stdout.
    #  For example, 'filter: this.timestamp' will only emit timestamped objects.
    #  To learn more see: https://doc.log10x.com/api/js/#TenXBaseObject+timestamped
    filter: null

    # 'fields' specifies a list of 10x intrinsic/calculated/extracted fields
    #  to extract from each target object passed to the output
    #  and written to stdout in the format specified by 'encodeType'
    #  If no fields are specified, encode/decode the target TenXObject.

    #  This example corresponds to the stdin format defined by fluent-bit in:
    #  https://docs.fluentbit.io/manual/pipeline/inputs/standard-input
    #  The 'text' argument below could be replaced by any list of intrinsic/calculated/extracted fields.
    #  The "" delimiter instructs 'joinFields' to produce JSON output.
    fields:  #
      - open="["
      - timestamp=dateFormat("+%s.%N")
      - separator=","
      - fields=joinFields("", "text")
      - close="]"

    # 'encodeType' specifies how 'fields' are encoded to stdout. Possible values:
    #    - 'json': formats names and values for the current TenXObject as: {"field":"str","field2":1, "field3": true}
    #    - 'delimited': formats values for the current TenXObject as: "str",1,true
    #                   The value separator (e.g., ',') is set by: 'encodeDelimiter'
    encodeType: delimited

    # 'encodeDelimiter' sets the char delimiter used to separate TenXObject
    #  field values when 'encodeType' is set to: 'delimited'
    encodeDelimiter: ' '

  # ----------------------------- Template Output -----------------------------

    # 'writeTemplates' controls whether to write new templates for TenXObjects written to
    #  this output. To learn more https://doc.log10x.com/run/template
  - writeTemplates: false  # (❗ REQUIRED)

    # -------------------------- Process Options ------------------------------

    # The output below writes the template values of TenXObjects
    # passed to this location. To learn more about TenXTemplates, see:
    # https://doc.log10x.com/run/template

    # 'command' defines the OS-resolvable path of the sub-process to spawn
    command: /opt/fluent-bit/bin/fluent-bit

    args:
      - -i
      - stdin
      - -o
      - stdout

    # 'destroyMode' is set to 'cached' to allow a fluent-bit sub-process to
    #  to be reused across multiple 10x pipeline executions within a host JVM.
    destroyMode: cached

    # 'maxCacheSize' sets the size of the cache to used for caching fluent-bit instances
    maxCacheSize: 1

    # 'lazyLaunch' determines whether the fluent-bit instance will launch immediately
    # during the pipeline initialization, or only on-demand.
    lazyLaunch: true

    # 'startupWaitMs' and 'startupWaitPattern' controls the wait for the fluent-bit instance
    # to be ready for receiving events, before starting to emit to it.
    startupWaitMs: $=parseDuration("5s")
    startupWaitPattern: \[\s*sp\s*]\s+stream\s+processor\s+started\b

Options

Specify the options below to configure multiple Process output:

Name	Description	Category
procOutCommand	Command to launch	Process
procOutArgs	Arguments to pass to the spawned sub-process	Process
procOutLazyLaunch	Sets whether to launch the process on-demand	Process
procOutStartupWaitMs	Wait time for process to startup	Process
procOutStartupWaitPattern	Wait for an output from the process at startup	Process
procOutDestroyMode	Control how to terminate the sub-process on output close	Lifecycle
procOutDestroyWait	Duration to wait for spawned sub-process to terminate	Lifecycle
procOutMaxCacheSize	Max number of sub-processes to keep alive across subsequent 10x pipeline executions	Lifecycle
procOutAllowMultipleStreams	Allow multiple concurrent outputs to share a subprocess	Lifecycle
procOutFields	List of TenXObject field names to encode	Encoding
procOutFilter	A JavaScript expression an TenXObject must evaluate as 'truthy' against to be written to this output	Encoding
procOutWriteObjects	If true, TenXObject field values specified by [procOutFields](https://doc.log10x.com/run/output/event/process/#procoutfields) are encoded to this output	Encoding
procOutWriteTemplates	If true, write new TenXTemplates to this output	Encoding
procOutEncodeType	The output format in which to encode TenXObject values. Possible values:[json, delimited]	Encoding
procOutEncodedLinePrefix	The line prefix prepended to the encoded event if 'procOutEncodeType' is 'delimited'	Encoding
procOutEncodeDelimiter	The delimiter placed between key and values pairs if 'procOutEncodeType' is 'delimited'	Encoding

Process

procOutCommand

Command to launch.

Type	Required	Category
String	✔	Process

Specifies the command to execute to launch the sub-process. This command can specify a shipper (e.g., Fluent Bit, Filebeat) or OS util (e.g., nc) process that will forward the output object values to a remote destination.

procOutArgs

Arguments to pass to the spawned sub-process.

Type	Default	Category
List	[]	Process

List of values to pass as startup arguments to the spawned sub-process.

procOutLazyLaunch

Sets whether to launch the process on-demand.

Type	Default	Category
Boolean	false	Process

Controls whether to spawn the target sub-process upon the creation of the output or on-demand. If true, launch the sub-process when the first TenXObject instance is written to its stdin.

procOutStartupWaitMs

Wait time for process to startup.

Type	Default	Category
Number	0	Process

Sets a startup delay wait time in milliseconds before the process is considered available for events. if provided, will add a wait as part of starting the process before emitting events into it. max value for this argument is 10,000 (ie 10 seconds). if procOutStartupWaitPattern is also provided, the process will be considered a failure if the wait time has passed before a line matching 'procOutStartupWaitPattern' was emitted.

procOutStartupWaitPattern

Wait for an output from the process at startup.

Type	Default	Category
String	""	Process

Sets a regex pattern to wait for the process to emit for it to be considered available for events. if provided, will wait until the process has emitted a line to it's output that matches the regex before emitting events into it. if procOutStartupWaitMs is also provided, will only wait for the specified wait time, after which if no line was emitted to match the provided regex the process will be considered a failure.

Lifecycle

procOutDestroyMode

Control how to terminate the sub-process on output close.

Type	Default	Category
String	""	Lifecycle

Controls how to treat spawned sub-process when this output closes. Possible values:

Value	Description
graceful (default)	The sub-process is destroyed gracefully with a timeout period of procOutDestroyWait.
forcible	Terminates the sub-process immediately.
cached	The sub-process remains alive for future pipelines that launch the same command and argument combination within the host JVM. A maximum of procOutMaxCacheSize can be kept alive. Useful for launching shippers (e.g., Filebeat, Fluent Bit) across multiple 10x pipeline runs in a long-living server environment.

procOutDestroyWait

Duration to wait for spawned sub-process to terminate.

Type	Default	Category
String	""	Lifecycle

Controls the time duration that the output stream will wait for the sub-process to terminate gracefully (e.g., '5s'). If the process timeouts, the output will forcibly terminate it. If set to 0, the output stream will wait indefinitely for the sub-process to terminate. This value only applies when procOutDestroyMode is set to graceful.

procOutMaxCacheSize

Max number of sub-processes to keep alive across subsequent 10x pipeline executions.

Type	Default	Category
Number	0	Lifecycle

Sets the global maximum number of processes that can retained globally across 10x pipeline executions within the host JVM. This value is useful when launching shippers (e.g., Filebeat, Fluent Bit) for use across multiple 10x pipeline executions within a server environment. This value only applies when procOutDestroyMode is set to cached. If procOutMaxCacheSize is set to 0, no caching is performed, and sub-processes are destroyed gracefully.

procOutAllowMultipleStreams

Allow multiple concurrent outputs to share a subprocess.

Type	Default	Category
Boolean	false	Lifecycle

If this value is true, multiple output streams sharing an identical command and arguments will emit their values to a single subprocess. Write access to the subprocess' stdin device will be synchronized between concurrent output streams.

For example, if the target subprocess is a log forwarder (e.g., Fluentd/Bit) it can be configured to route TenXObjects from multiple output streams to their target destinations without requiring a separate instance of the forwarder for each configured output stream.

Encoding

procOutFields

List of TenXObject field names to encode.

Type	Default	Category
List	[encodedOuterText](https://doc.log10x.com/api/js/#TenXBaseObject+encodedOuterText)]	Encoding

Specifies a list of TenXObject intrinsic/calculated/extracted field names to extract and write to the output.

procOutFilter

A JavaScript expression an TenXObject must evaluate as 'truthy' against to be written to this output.

Type	Default	Category
String	""	Encoding

Specifies a JavaScript expression that an TenXObject must evaluate as truthy to write its instance/template field values to this output.

For example, to only emit timestamped TenXObjects, specify:

procOutFilter: this.timestamped

procOutWriteObjects

If true, TenXObject field values specified by [procOutFields](https://doc.log10x.com/run/output/event/process/#procoutfields) are encoded to this output.

Type	Default	Category
String	true	Encoding

Determine whether to write procOutFields values of an TenXObject passed to this output. This configuration is typically defined using a JavaScript expression.

To enable/disable the output depending on whether a launch argument, environment variable, or JVM -D option evaluates as truthy, use:

  procOutWriteObjects: $=TenXEnv.get("myProcCmd")

To learn more see TenXEnv.get.

procOutWriteTemplates

If true, write new TenXTemplates to this output.

Type	Default	Category
String	false	Encoding

Controls whether to write template values of TenXObjects passed to this output. The output will only emit TenXTemplates that the pipeline did not load at startup via thetemplateFile argument, and will only emit an TenXTemplate once based on its templateHash value.

procOutEncodeType

The output format in which to encode TenXObject values. Possible values:[json, delimited].

Type	Default	Category
String	delimited	Encoding

Specifies how to encode procOutFields values to this output. Possible values:

• json: formats names and values for the current TenXObject as: {"field":"str","field2":1, "field3": true}
• delimited: formats values for the current TenXObject as: "str",1,true The procOutEncodeDelimiter argument determines the separator to use.

procOutEncodedLinePrefix

The line prefix prepended to the encoded event if 'procOutEncodeType' is 'delimited'.

Type	Default	Category
String		Encoding

Sets the char which is prepended to the emitted TenXObject field values when procOutEncodeType is delimited.

procOutEncodeDelimiter

The delimiter placed between key and values pairs if 'procOutEncodeType' is 'delimited'.

Type	Default	Category
String		Encoding

Sets the char delimiter to separate TenXObject field values when procOutEncodeType is delimited.

This module is defined in process/module.yaml.