File Output

Defines a file output to write TenXObject instance and template field values.

Configuration

To configure the File output module, Edit these settings.

Below is the default configuration from: file/config.yaml.

Edit Online Edit Locally

Edit config.yaml Locally

Download and save to:

• Linux / Docker / macOS
• Custom directory
  $TENX_CONFIG/run/output/event/file/config.yaml
• Within cloned repo
  ./pipelines/run/output/event/file/config.yaml

Download

# 🔟❎ 'run' file output configuration

# Configure file outputs to write TenXObject instance and template field values.
# To learn more see https://doc.log10x.com/run/output/event/file

# Set the 10x pipeline to 'run'
tenx: run

# =============================== Dependencies ================================

include:
  - run/modules/output/event/file

# ============================== File Options =================================

# Multiple file outputs can be defined below
outputFile:

  # --------------------------- 10x Encoding Output ---------------------------

    # Encode TenXObjects: https://doc.log10x.com/run/transform/#template-encoding

    # 'path' specifies a path on disk to write TenXObject field values
  - path: $=path("data/sample/output") + "/encoded.log"

    # 'filter' sets a JavaScript expression that TenXObjects must evaluate as truthy to write to the output.
    #  To learn more see https://doc.log10x.com/run/output/regulate/#filter-expressions
    filter: null

    # 'fields' defines the fields to write for each TenXObject sent to this output.
    #  To learn more see https://doc.log10x.com/run/output/stream/#outputfields  
    fields: 
      - encode() # https://doc.log10x.com/api/js/#TenXObject+encode

  # ----------------------------- Decode Output ------------------------------

    # Template-decode TenXObjects: https://doc.log10x.com/run/transform/#expand

  - path: $=path("data/sample/output") + "/decoded.log"

    filter: isEncoded  # https://doc.log10x.com/api/js/#TenXBaseObject+isEncoded   

    fields: 
      - text           # https://doc.log10x.com/api/js/#TenXBaseObject+text

  # ---------------------------- Template Output ----------------------------

  # Writes TenXTemplate values: https://doc.log10x.com/run/transform/#template

  - path: $=path("data/sample/output") + "/templates.json"

    # 'writeTemplates' controls whether to write new templates for TenXObjects written to this output.
    writeTemplates: true

    # 'append' controls whether to overwrite or append to the output file
    append: true

  # ----------------------------- Summary Output ------------------------------

    # Writes aggregated TenXSummaries: https://doc.log10x.com/run/aggregate

  - path: $=path("data/sample/output") + "/aggregated.csv"

    # 'header' is set to write the field names aggregated into the first line of the output
    #  We specify '$=yield' to wait until the 'enrichmentFields' env var have been evaluated 
    header: $=yield TenXString.join(",", TenXEnv.get("enrichmentFields"), "summaryVolume", "summaryBytes", "summaryTotals") + "\n"

    filter: isSummary                            # https://doc.log10x.com/api/js/#TenXBaseObject+isSummary                        

    fields: 
      - $=yield TenXEnv.get("enrichmentFields")  # https://doc.log10x.com/run/initialize/#enrichmentFields
      - summaryVolume                            # https://doc.log10x.com/api/js/#TenXSummary+summaryVolume
      - summaryBytes                             # https://doc.log10x.com/api/js/#TenXSummary+summaryBytes
      - summaryTotals                            # https://doc.log10x.com/api/js/#TenXSummary+summaryTotals

Options

Specify the options below to configure multiple File output:

Name	Description	Category
outputFilePath	Location of the output file	File
outputFileAppend	Overwrite or append to file	File
outputFileHeader	Header value to write to file	File
outputFileWriteObjects	If true, encode TenXObjects [outputFileFields](https://doc.log10x.com/run/output/event/file/#outputfilefields) to this output	Encoding
outputFileFilter	A JavaScript expression an TenXObject must evaluate as 'truthy' against to be written to file	Encoding
outputFileFields	List of TenXObject field names to encode	Encoding
outputFileEncodeType	The output format in which to encode TenXObject values. Possible values:[json, delimited]	Encoding
outputFileEncodedLinePrefix	The line prefix prepended to the encoded event if 'outputFileEncodeType' is 'delimited'	Encoding
outputFileEncodeDelimiter	The delimiter placed between key and values pairs if 'outputFileEncodeType' is 'delimited'	Encoding
outputFileEncodeGroupDelimiter	String delimiter to use when separating individual 10x events inside a group	Encoding
outputFileWriteTemplates	If true, write new TenXTemplates to this output	Encoding
outputFileLog4j2	If true, write using log4j2 appender	Advanced

File

outputFilePath

Location of the output file.

Type	Required	Category
File	✔	File

Specifies a path on disk to write TenXObject instance and template field values.

outputFileAppend

Overwrite or append to file.

Type	Default	Category
Boolean	false	File

Controls whether to overwrite the output file or append TenXObject instance/template field values to its end.

outputFileHeader

Header value to write to file.

Type	Default	Category
String	""	File

Specifies a header file to write to file in outputFileAppend is false.

Encoding

outputFileWriteObjects

If true, encode TenXObjects [outputFileFields](https://doc.log10x.com/run/output/event/file/#outputfilefields) to this output.

Type	Default	Category
String	true	Encoding

Determine whether to write outputFileFields values of an TenXObject passed to this output. This configuration is typically defined using a JavaScript expression. To enable/disable the output depending on whether a launch argument, environment variable, or JVM -D option evaluates as truthy, use:

  outputFileWriteObjects: $=TenXEnv.get("myFilePath")

To learn more see: TenXEnv.get.

outputFileFilter

A JavaScript expression an TenXObject must evaluate as 'truthy' against to be written to file.

Type	Default	Category
String	""	Encoding

Specifies a JavaScript expression that an TenXObject must evaluate as truthy to write its instance/template field values to this output. For example, 'outputFileFilter: this.timestamp' will only emit timestamped TenXObjects.

outputFileFields

List of TenXObject field names to encode.

Type	Default	Category
List	[]	Encoding

Specifies a list of TenXObject intrinsic/calculated/extracted field names to extract and write to the output.

outputFileEncodeType

The output format in which to encode TenXObject values. Possible values:[json, delimited].

Type	Default	Category
String	delimited	Encoding

Specifies how outputFileFields are encoded to a stream/event to this output. Possible values:

• json: formats names and values for the current TenXObject as: {"field":"str","field2":1, "field3": true}
• delimited: formats values for the current TenXObject as: "str",1,true The outputFileEncodeDelimiter argument determines the separator to use.

outputFileEncodedLinePrefix

The line prefix prepended to the encoded event if 'outputFileEncodeType' is 'delimited'.

Type	Default	Category
String	~	Encoding

Sets the char which is prepended to the emitted TenXObject field values when outputFileEncodeType is delimited.

outputFileEncodeDelimiter

The delimiter placed between key and values pairs if 'outputFileEncodeType' is 'delimited'.

Type	Default	Category
String	,	Encoding

Sets the delimiter to separate TenXObject field values when outputFileEncodeType is delimited.

outputFileEncodeGroupDelimiter

String delimiter to use when separating individual 10x events inside a group.

Type	Default	Category
String	""	Encoding

Sets the string delimiter used to separate individual 10x events inside a group when encoded to a stream/event output.

outputFileWriteTemplates

If true, write new TenXTemplates to this output.

Type	Default	Category
String	false	Encoding

Controls whether to write template values of TenXObjects passed to this output. The output will only emit TenXTemplates that the pipeline did not load at startup via the templateFile argument, and will only emit an TenXTemplate once based on its templateHash value.

Advanced

outputFileLog4j2

If true, write using log4j2 appender.

Type	Default	Category
Boolean	false	Advanced

Controls whether to write to file using an OutputStream or a log4j2 appender.

This module is defined in file/module.yaml.