Custom Input

Defines network, disk, and IPC locations for reading events into TenXObjects. Pipelines support multiple concurrent streams.

Input modules read from log analyzers, forwarders, and other sources.

Each stream provides:

Extractors: Select and redact input data before transformation
Source patterns: Associate events with their origin (log file, network resource)
Backpressure: Throttle data volume

Extensions

The input extensions API reads events from custom local and remote sources.

Options

Specify the options below to configure multiple Input stream:

Name	Description	Category
inputName	Logical name for the input stream	General
inputEnabled	A JavaScript expression that must be evaluated as 'truthy' to enable the input	General
inputPrintProgress	Sets whether this input reports progress to the console	General
inputType	Type of stream. Possible values:[stream, file, inline, stdin]	Stream
inputPath	Input stream location	Stream
inputArgs	Custom arguments passed to InputStream	Stream
inputExtractors	Names of extractors defining JSON fields/regex capture groups to select	Parse
inputEscapeScheme	Unescape scheme to apply to characters read from the underlying input	Parse
inputSourcePattern	A regex pattern used to extract the 'source' value	Parse
inputSourceFields	JSON fields from which to extract the 'source' value	Parse
inputSourceFilter	A regex pattern used to filter out input events from applying 'inputSourcePattern'	Parse
inputBackpressureIntervalBytesLimit	The max allowed number of bytes to read within 'inputBackpressureIntervalBytesDuration'	Backpressure
inputBackpressureIntervalBytesDuration	The time interval after which to reset the backpressure input quota	Backpressure
inputBackpressureTotalEventsLimit	The total number of bytes to read from input	Backpressure
inputBackpressureTotalBytesLimit	The total number of bytes to read from input	Backpressure
inputBackpressureTotalDuration	A JavaScript expression that evaluates the max duration in which an input can read events before closing	Backpressure
inputForeach	Name of option group for whose instances to create input streams	Advanced

General

`inputName`

Logical name for the input stream.

Type	Default	Category
String	an automatically generated input name	General

Defines a unique logical name for the input (e.g., myFluentd). Each TenXObject instantiated from events read from this input will return this value via its inputName field.

`inputEnabled`

A JavaScript expression that must be evaluated as 'truthy' to enable the input.

Type	Default	Category
String	true	General

enables/disables this input stream. If set, the JavaScript expression returns a truthy value to open the input. For example, to configure this value to use a startup argument/shell variable, use:

input:
   name: myInput
   enabled: TenXEnv.get("readFromStdin") 
  ...

`inputPrintProgress`

Sets whether this input reports progress to the console.

Type	Default	Category
String	""	General

Controls whether to print information on the volume of events read input to the console. The printProgress launch argument must also be set for this value to take effect.

Stream

`inputType`

Type of stream. Possible values:[stream, file, inline, stdin].

Type	Required	Category
String	✔	Stream

Determines the type of input. Possible values:

file: contents of a target file specified by inputPath. The file is read once and NOT tailed.
stdin: read from the process stdin device.
inline: read JSON-escaped event(s) from the value of inputPath separated by \n.
stream: read events from an InputStream or Reader sub-class specified by by 'inputPath'.

`inputPath`

Input stream location.

Type	Required	Category
String	✔	Stream

Specifies the location from which to read events. This value depends on inputType. For:

file: file path\glob on disk (relative or canonical), or `stdin`` to read from the process stdin device
inline: JSON-escaped content events separated by \n.
stream: fully qualified InputStream/ Reader sub-class to instantiate.
stdin: inputPath is ignored.

`inputArgs`

Custom arguments passed to InputStream.

Type	Default	Category
List	[]	Stream

If inputType is stream, this argument specifies a list of values pairs (key1,value1...) passed to the constructor of the InputStream/Reader sub-class specified by inputPath.

A Map combining these values and those of the input's module option group specified by the (inputForeach)[#inputforeach] argument (if present) provides the stream with module-specific context (e.g., auth tokens, host addresses) 'config' map.

An EvaluatorBean reference provides an interface to the 10x JavaScript engine.

For example:

  import com.log10x.api.bean.EvaluatorBean;
  import java.io.Reader;
  import java.util.Map;

  public class MyReader extends Reader {

    /**
    * If a constructor with this signature is found, it is used. 
    * 
    * @param config combines launch argument provided to the 10x Engine
    *                       for this input with those specified by 'inputArgs'  
    * @param bean       provides an interface to the 10x Engine
    */
    public MyReader(Map<String, Object> config, EvaluatorBean bean) {
    }

    /**
    * If the above constructor is not defined this signature is used. 
    * 
    * @param config combines launch argument provided to the 10x Engine
    *                       for this input with those specified by 'inputArgs'
    */

    public MyReader(Map<String, Object> config) {
    }

    /**
    * 
    * If the two signatures above are not defined a parameterless
    * constructor must be defined.
    */
    public MyReader(){
    } 

    @Override
    public int read(char[] cbuf, int off, int len) throws IOException {
      //Read data from the underlying input 
      return charsRead;
    }

    @Override
    public void close() throws IOException {
    // close input
    }
}

If the search for a matching constructor fails, the pipeline halts.

Parse

`inputExtractors`

Names of extractors defining JSON fields/regex capture groups to select.

Type	Default	Category
List	[]	Parse

Specifies a list of input extractors to apply on events read from this input stream.

If this value is empty, each line read from the stream of up to 65536 characters is used to instantiate an TenXObject. Lines that exceed that length force a virtual line break, and the following characters will form the next event until a line break is detected.

`inputEscapeScheme`

Unescape scheme to apply to characters read from the underlying input.

Type	Default	Category
String	""	Parse

Specifies an unescape algorithm to apply to characters read from the underlying input. Supported values: [json,java,xml,html,javaScript]. If not specified, characters are treated as not escaped.

To learn more see Unescape algorithms.

`inputSourcePattern`

A regex pattern used to extract the 'source' value.

Type	Default	Category
String	""	Parse

Defines a regex pattern that, if matched for the current event text is attached to each TenXObject extracted from it as its 'source' value.

The source value identifies the logical location (e.g., log file, host address) from which an event originated. For example, a forwarder input may read events from multiple log files (i.e., sources) concurrently.

Assigning source values ensures that single and multiline events (e.g., stack traces) read from a target input (e.g., Filebeat, Fluent Bit) originating from a source location (e.g., /var/log/a.log) are sequenced and grouped without interlacing with events from a different source (e.g., /var/log/b.log).

For example, when reading events originating from different log files, the sourcePattern can be set to: \"fileName\":"(.*?)\" to extract a 'fileName' JSON field within each event to ensure the resulting TenXObjects are sequenced and grouped (as in the case of stack traces) with other TenXObjects based on the actual log file from which they originated.

The TenXObject intrinsic source field provides access to this value at runtime to control how to sequence an instance with other TenXObjects read from the current input stream.

To learn more see event source.

`inputSourceFields`

JSON fields from which to extract the 'source' value.

Type	Default	Category
String	""	Parse

Specifies a list of JSON fields whose value are captured from every matching capture group within inputSourcePattern. If no source pattern is set the entire event is scanned for the target JSON field values. The TenXObject intrinsic source field provides access to joint value at runtime to control how to sequence an instance with other TenXObjects read from the current input stream. To learn more see event source.

`inputSourceFilter`

A regex pattern used to filter out input events from applying 'inputSourcePattern'.

Type	Default	Category
String	""	Parse

Specifies a regex pattern to match for inputSourcePattern to be applied.

This pattern provides an option to specify a fast-performing pattern to determine whether inputSourcePattern is applied to capture 'source' values for TenXObjects extracted for an event read from the input.

For example, inputSourceFilter can determine whether an input event begins with a target character sequence before a more complex inputSourcePattern is applied to select its 'source' value.

Backpressure

`inputBackpressureIntervalBytesLimit`

The max allowed number of bytes to read within 'inputBackpressureIntervalBytesDuration'.

Type	Default	Category
String	""	Backpressure

Sets the maximum number of bytes that can be read by the input within the target inputBackpressureIntervalBytesDuration. If exceeded, the input blocks further reading (if supported); otherwise, the input drops events from the pipeline for the remainder of the interval.

For example:

inputBackpressureIntervalBytesLimit: $=ParseBytes("10MB")

If set, inputBackpressureIntervalBytesDuration must also be set.

This option aims to prevent a situation where the inflow of data exceeds the memory allocated to the pipeline's host JVM via the XMX argument.

To learn more see JVM heap.

`inputBackpressureIntervalBytesDuration`

The time interval after which to reset the backpressure input quota.

Type	Default	Category
String	""	Backpressure

sets the interval in milliseconds for which inputBackpressureIntervalBytesLimit is enforced.

When the duration elapses, the input resets the counter against which inputBackpressureIntervalBytesLimit is measured.

If set, inputBackpressureIntervalBytesLimit must also be set.

For example:

inputBackpressureIntervalBytesDuration: $=parseDuration("1m")

`inputBackpressureTotalEventsLimit`

The total number of bytes to read from input.

Type	Default	Category
String	""	Backpressure

Set the max number of events a target pipeline input will read into the pipeline. This value limits the volume of events to read from a local/remote source (e.g., log analyzer).

`inputBackpressureTotalBytesLimit`

The total number of bytes to read from input.

Type	Default	Category
String	""	Backpressure

set the max number of bytes a target pipeline input will read into the pipeline. This value limits the volume of events to read from a local/remote source (e.g., log analyzer).

For example:

inputBackpressureTotalBytesLimit: $=ParseBytes("1GB")

`inputBackpressureTotalDuration`

A JavaScript expression that evaluates the max duration in which an input can read events before closing.

Type	Default	Category
String	""	Backpressure

set the max duration (e.g. 1m) an input can read events from a local/remote source before closing.

For example:

inputBackpressureTotalDuration: $=parseDuration("5m")

Advanced

`inputForeach`

Name of option group for whose instances to create input streams.

Type	Default	Category
String	""	Advanced

Specifies the name of an options group for whose instances to replicate this input stream object. This value enables 10x modules to create multiple input instances, each receiving a unique set of arguments.

For a full example, see Elastic input.

This unit is defined in stream/unit.yaml.