Input

Reads log/trace events from local and remote sources, transforming them into TenXObjects.

Inputs comprise two core components: streams and extractors to construct modules which read events from edge/cloud sources such as log analyzers, forwarders and object storage.

Multiple input streams can execute in conjunction, where each stream provides backpressure controls to ensure the pipeline can process the information received.

Modules

Stdin Device Input

Read events from stdin.

More info
File Inputs

Read events from an input file/glob path.

More info
Log Analyzer Inputs

Read events from log analyzers via REST.

More info
Log Forwarder Inputs

Report, regulate and optimize events collected by log forwarders.

More info
Object Storage Input

Stream events from object storage.

More info
Input Streams

Read events from a variety of sources to transform into TenXObjects.

More info
Input Extractors

Filter, redact, and capture events from an input stream to transform into TenXObjects.

More info

Configuration

To configure the Input unit, Edit these settings.

Below is the default configuration from: backpressure/config.yaml.

Edit Online

# 🔟❎ run input configuration

# This config file specifies general pipeline input settings
# To learn more see https://doc.log10x.com/run/input

tenx: run

# ============================== Concurrency Settings =========================

# 'inputThreadPoolSize' controls the maximum number of threads allocated to concurrently 
#  read events from inputs source (e.g., stream, stdin, file).
inputThreadPoolSize: 16 

# ============================= Backpressure Settings =========================

objectLimit:

# 'maxInstances' sets the max number of TenXObjects the pipeline can contain at any given moment
#  Default to 5K objects per 100MB ram of JVM -XMX (if specified), otherwise no limit
  maxInstances: $=TenXEnv.get("xmx") ? (parseInt(TenXEnv.get("xmx")) / 104857600 * 5000):0 

  # 'interval' sets the duration time between repeated backpressure tests once backpressure is detected. 
  interval: $=parseDuration("1s")

Options

Specify the options below to configure the Input:

Name	Description
objectLimitMaxInstances	Max number of TenXObjects that the pipeline can hold
objectLimitInterval	Defines interval between consecutive positive backpressure tests
inputThreadPoolSize	Sets the maximum number of threads to allocate for concurrent inputs.

`objectLimitMaxInstances`

Max number of TenXObjects that the pipeline can hold.

Type	Default
Number	10000

Sets the max number of TenXObjects that the pipeline can contain at any given moment, beyond which input(s) either block (if supported) or drop events from the pipeline. Once an object reaches the end of the pipeline, space becomes available for made for instantiating TenXObjects from events read input.

This value prevents scenarios where writing data to output is slower than reading events from input, which may cause backpressure inside the pipeline. Setting this value can avoid an out-of-mem error because of the object backlog.

Increasing the host JVM XMX argument can help address potential OOM issues.

`objectLimitInterval`

Defines interval between consecutive positive backpressure tests.

Type	Default
Number	1000

Sets the interval between backpressure tests after detection to allow time for clearing TenXObjects from the pipeline before attempting additional tests.

`inputThreadPoolSize`

Sets the maximum number of threads to allocate for concurrent inputs.

Type	Default
Number	16

Sets the maximum number of threads to allocate for concurrent inputs.

If the value is -

0: open a dedicated thread for each input
Less than 1: interpreted as a percentage of the number of available cores (e.g., 0.5 = use up to 50% of available cores)
1: allocate a single async thread
Greater than 1: interpreted as a fixed number of threads (e.g., 10 = 10 threads)

The number of threads should be limited when a large number of short-lived inputs are defined to avoid out-of-memory errors.

This unit is defined in input/unit.yaml.