File Input

Configures a input file path or glob pattern from which to read events to transform into typed TenXObjects.

This input does NOT tail its target file/glob. To tail files on disk, use a forwarder input.

Configuration

To configure the File input module, Edit these settings.

Below is the default configuration from: file/config.yaml.

# 🔟❎ 'run' File input configuration

# Configure file/glob event inputs. To learn more https://doc.log10x.com/run/input/file

# NOTE: input files are read once and are not tailed. To continuously tail files
# use 10x in conjunction with any of the supported file forwarders in:
# https://doc.log10x.com/run/input/forwarder (e.g., Fluentd/Bit, Logstash, Filebeat)

# Set the 10x pipeline to 'run'
tenx: run

# =============================== Dependencies ================================

include: run/modules/input/file

# =============================== File Options ================================

# Multiple file inputs can be defined below
inputFile:

  # ---------------------------- General Options ----------------------------

    # 'name' sets a unique logical name across all pipeline inputs
  - name: fileInput

  # ---------------------------- File Options ---------------------------------

    # 'path' specifies an absolute/canonical glob file path.
    path: $=path("data/sample/input/[!{.}]*.log")

    # ---------------------------- Parsing Options ----------------------------

    # 'extractors' defines a list of JSON/regex extractor names
    #  used to capture and redact event values from 'path' to transform 
    #  into TenXObjects. To learn more see: https://doc.log10x.com/run/input/extract         
    extractors: [  
    ]

    # 'sourcePattern' defines a regex pattern that captures a 'source' value
    #  for each event read from a file to transform into an TenXObject.
    #  This value will be used to sequence all events sharing this 'source' value correctly 
    #  so they are not mixed with events originating from different locations.
    sourcePattern: null  

    # 'sourceFields'  defines a list of JSON fields to extract from matching capture groups of the inputFileSourcePattern
    # or the entire event if not set as the 'source' value for each event read from this input. 
    sourceFields: [
    ]

    # --------------------------- Ancillary Options ---------------------------

    # 'printProgress' controls whether a progress gage is printed to the console    
    printProgress: true

Options

Specify the options below to configure multiple File input:

Name	Description
inputFileName	Logical name of this input
inputFilePath	A file path or glob pattern to read from
inputFileEnabled	Enable/disable this file input
inputFileExtractors	JSON / regex extractor names
inputFileSourcePattern	Event source capture regex pattern
inputFileSourceFields	Event source JSON fields
inputFilePrintProgress	Sets whether this input prints throughput stats to the console

`inputFileName`

Logical name of this input.

Type	Default
String	""

Sets a logical name (e.g., 'myInput') for this input. The inputName field returns this value at run time to allow for identifying and operating on instances originating from this input.

`inputFilePath`

A file path or glob pattern to read from.

Type	Default
File	path("\<user.dir>/data/sample/[!{.}]*.log")

specifies a path that is either an absolute or canonical file path. For example:

path: $=path("<user.dir>/data/sample/[!{.}]*.log")

`inputFileEnabled`

Enable/disable this file input.

Type	Default
Boolean	true

Specifies whether the open the input to read events from the target inputFilePath.

`inputFileExtractors`

JSON / regex extractor names.

Type	Default
List	[]

Defines a list of JSON/regex extractor names to capture events from 'inputFilePath' to transform into TenXObjects. Each item in the list should map to an extractor defined via a configuration file or startup arguments group. To learn more see: extractors.

`inputFileSourcePattern`

Event source capture regex pattern.

Type	Default
String	""

Defines a regex pattern to capture 'source' values for each event read from this input.

Source values provide a method for sequencing TenXObjects read from this input in separate logical queues.

For example, capturing a 'host' or 'fileName' value as a 'source' creates a separate logical queue for each unique file/host origin by which TenXObjects read from this input are aggregated and written to output.

The source member returns each TenXObject's assigned value at run time. .

`inputFileSourceFields`

Event source JSON fields.

Type	Default
List	[]

Defines a list of JSON fields to extract from matching capture groups of inputFileSourcePattern or the entire event if not set as the 'source' value for each event read from this input.

Source values provide a method for sequencing TenXObjects read from this input in separate logical queues.

For example, capturing a 'host' or 'fileName' value as a 'source' creates a separate logical queue for each unique file/host origin by which TenXObjects read from this input are aggregated and written to output.

The source member returns each TenXObject's assigned value at run time.

`inputFilePrintProgress`

Sets whether this input prints throughput stats to the console.

Type	Default
Boolean	false

Sets whether this input prints throughput stats to the console.

This module is defined in file/module.yaml.