Custom

Defines output destinations for TenXObjects, specifying location and compact format. Pipelines support multiple streams.

Extensions

Most users work with output modules rather than raw streams. Modules provide higher-level constructs for serialization via Java streams and log4j2 appenders.

See output extensions.

Output Modules

Output Modules package stream configuration, JavaScript and documentation files for writing TenXObjects to edge and cloud data destinations which include:

• Forwarders (Fluentd/Bit) - regulate and optimize before shipping
• Time-series (Prometheus, Datadog) - publish TenXSummary metrics
• Object Storage (S3, Azure Blobs) - index for in-place querying

Options

Specify the options below to configure multiple Custom output:

Name	Description	Category
outputEnabled	A JavaScript expression that must be evaluated as 'truthy' to enable this output	General
outputName	Logical name for the output stream	General
outputFilter	A JavaScript expression an TenXObject must evaluate as 'truthy' against to be encoded	General
outputType	Type of destination output. Possible values:[file, stdout, event, metric, stream]	General
outputPath	Logical output destination	General
outputArgs	Custom arguments passed to OutputStream/Writer instance	General
outputFields	A comma-delimited list of TenXObject fields to encode	Encode
outputEncodeType	Output format for encoding TenXObject values. Possible values:[json, delimited]	Encode
outputEncodedLinePrefix	Char line prefix to prepend when encoding delimited TenXObject field values	Encode
outputEncodeDelimiter	Char delimiter to use when separating encoded delimited TenXObject field values	Encode
outputEncodeGroupDelimiter	Delimiter to separate TenXObjects encoded into as a group	Encode
outputAsync	Determines whether encoding of this out is done in a dedicated thread	Advanced
outputForeach	Name of module options group for whose instances to create matching outputs	Advanced
outputGroup	Option names whose values for grouping/synchronizing output streams	Advanced

General

outputEnabled

A JavaScript expression that must be evaluated as 'truthy' to enable this output.

Type	Default	Category
String	""	General

Enables/disables this output stream. If set, the JavaScript expression must be truthy to open the output.

outputName

Logical name for the output stream.

Type	Required	Category
String	✔	General

Defines a unique logical name for the output (e.g., datadog). Each TenXObject will return this value via its outputName field.

outputFilter

A JavaScript expression an TenXObject must evaluate as 'truthy' against to be encoded.

Type	Default	Category
String	""	General

Defines a JavaScript expression an TenXObject must evaluate as truthy against to be written the target output. This filter operates in an AND relationship with the outputFilters regulator argument (i.e., both must be evaluated as truthy to pass).

outputType

Type of destination output. Possible values:[file, stdout, event, metric, stream].

Type	Required	Category
String	✔	General

Defines the mechanism for writing TenXObject, template and summary instance field values to this output.

Possible values:

• file: emit field values to a file on disk.

• stdout: emit field values to stdout/err device.

• event: utilizes the java log4j2 library to emit TenXObjects into a wide array of output destinations via log4j appenders (e.g., file, socket, http). To learn more see appender outputs.

• metric: utilizes the Micrometer library to emit [TenXSummary](TenXObjects instances as metric counter to time series output destinations (e.g., Prometheus). To learn more see metric outputs.

• stream: instantiates a sub-class of java.io.OutputStream/Writer to write TenXObject field values to a user-provided stream.

This method allows for programmatically defining output destinations. For an example, see the default outputStream configuration.

outputPath

Logical output destination.

Type	Required	Category
String	✔	General

Defines the concrete destination for this output stream.

This value is dependent on that of outputType. For:

• file: location on disk to append/overwrite

• event: a .yaml or .xml file containing a log4j2 configuration for a logger and appender(s) to log TenXObjects via the Message interface.

• metric: fully qualified Java class name implementing the MetricRegistryFactory interface to create a micrometer registry.

• stream: fully qualified class name derived from OutputStream or Writer to instantiate. The output will flush the stream/writer after serializing the values of outputFields. for each TenXObject. It is up to the class to implement any required batching and buffering logic. An EvaluatorBean reference provides an interface to the 10x JavaScript engine. The class to instantiate must declare a constructor with one of the following signatures:

public class MyOutputStream extends OutputStream {

  // This signature is used (if declared)
  MyOutputStream(Map<String, Object> config, com.log10x.api.bean.EvaluatorBean bean) {...}

  // otherwise use this signature 
  MyOutputStream(Map<String, Object> config) {...}

  // If neither are declared, utilize a parameterless constructor
  MyOutputStream() {...}
}

If the search for a matching constructor fails, the pipeline halts.

outputArgs

Custom arguments passed to OutputStream/Writer instance.

Type	Default	Category
List	[]	General

If outputType is stream, this argument specifies a list of values pairs (key1,value1...) passed to the constructor of the class specified by outputPath.

A Map combining these values and those of the output's module option group specified by the outputForeach argument(if present) provide the stream with module-specific context (e.g., auth tokens, host addresses).

Encode

outputFields

A comma-delimited list of TenXObject fields to encode.

Type	Required	Category
List	✔	Encode

Lists fields from each TenXObject to write to this output. Fields can be:

• Intrinsic: Built into all TenXObject and summary instances.
• Enriched: Added via initializer modules (e.g., symbol, GeoIP, file lookups).
• Extracted: Extracted from JSON/KV pairs in the log event.
• Calculated: Derived using custom JavaScript constructors.
• Expression: An inline JavaScript expression defining an output-only field used when writing instances to this output. The field name and value can be explicitly set (e.g., myFoo=foo()) or implicitly derived using the $= prefix (e.g., $=foo() sets the field name to foo).

Example:

outputFile:
  path: out.log
  fields: 
  - myJsonExtField
  - myEnrichedField
  - myCalcField
  - myExpField=TenXEnv.get("var") ? "myCalcField":"foo()" 
  - encode()

Below are examples of TenXObject fields which can be encoded to output:

Field	Description	Type
encode()	Losslessly compact the instance. The field name is automatically set to encode	Expression
fullText	Full text of the underlying raw/expanded event	Intrinsic
symbolSequence	Metric-complaint symbol values sequence	Enriched
symbolMessage	Logical event message symbol values sequence	Enriched
symbolOrigin	Symbol value with class context	Enriched
namespace_name	Kubernetes namespace context	Enriched
container_name	Kubernetes container context	Enriched
pod_name	Kubernetes pod context	Enriched
city	GeoIP reference lookup (city, country.. )	Enriched
lookupValue	Text file (e.g., .CSV, .TSV) lookup	Enriched
code	Calculated HTTP code field	Calculated
template	TenXTemplate representation (i.e., hidden class)	Intrinsic
timestamp	64bit Unix epoch timestamps	Intrinsic
vars	High-cardinality variables	Intrinsic
ipAddress	Embedded IPv4 addresses	Intrinsic

outputEncodeType

Output format for encoding TenXObject values. Possible values:[json, delimited].

Type	Default	Category
String	""	Encode

Specifies how to format TenXObject field values to a stream/event output Possible values:

• json: formats N specified outputFields names and values for the target TenXObject as: {"field":"str","field2":1, "field3": true, .. "fieldN": "value"}.
• delimited: formats N specified outputFields values for the target TenXObject as: "str",1,true,..N-value. The outputEncodeDelimiter option determines the value separator

NOTE: This argument does not apply to outputs whose outputType is set to metric.

outputEncodedLinePrefix

Char line prefix to prepend when encoding delimited TenXObject field values.

Type	Default	Category
String	""	Encode

Sets the char line prefix which will be prepended to TenXObject field values encoded to a stream/event output. This argument only applies when outputEncodeType is set to delimited.

outputEncodeDelimiter

Char delimiter to use when separating encoded delimited TenXObject field values.

Type	Default	Category
String	""	Encode

Sets the char delimiter for separate TenXObject field values encoded to a stream/event output. This argument only applies when outputEncodeType is set to delimited.

outputEncodeGroupDelimiter

Delimiter to separate TenXObjects encoded into as a group.

Type	Default	Category
String	""	Encode

Sets the string delimiter used to separate individual 10x events inside a group when encoded to a stream/event output.

Advanced

outputAsync

Determines whether encoding of this out is done in a dedicated thread.

Type	Default	Category
Boolean	true	Advanced

Controls whether the output uses a dedicated thread for encoding TenXObjects or does so from the calling thread.

outputForeach

Name of module options group for whose instances to create matching outputs.

Type	Default	Category
String	""	Advanced

Specifies the name of an options group for whose instances to replicate this output object. This value enables 10x modules to create multiple input instances, each receiving a unique set of arguments.

For an example, see httpOutput.

outputGroup

Option names whose values for grouping/synchronizing output streams.

Type	Default	Category
List	[]	Advanced

Enables synchronizing different outputs pointing to a shared logical destination.

For example, in cases where multiple outputs are pointing at the same output file or stdout, it is desirable to synchronize writing between the outputs to ensure that no 'interlacing' of TenXObject values takes place during write. For this, any variables from the options group instance defined by outputForeach can be used to define a unique 'key' for this output to synchronize it with other outputs sharing the same value.

The example below, taken from: file/stream.yaml groups all log4j appenders writing to the same output file via their associated 'fileOutput' options group instance's 'outputFilePath' value.

- type: event
  foreach: fileOutput

  path: run/modules/output/event/file
  group:
    - $outputFilePath

Specify * to group by all values of the option group instance.

This unit is defined in stream/unit.yaml.