Parallel Processor

Transform log/trace events read from inputs into typed TenXObjects using multiple cores.

Configuration

To configure the Parallel processor unit, Edit these settings.

Below is the default configuration from: parallelize/config.yaml.

# 🔟❎ 'run' event parallel processing configuration

# Transform log/trace events read from inputs into well-defined TenXObjects using multiple cores.
# To learn more see https://doc.log10x.com/run/transform/parallelize/

# Set the 10x pipeline to 'run'
tenx: run

# =============================== Parallel Options ============================

parallelEvent:

  # 'threadPoolSize' specifies the number of threads allocated to transform events into TenXObjects concurrently.
  #  If the value is -
  #  - = 0: transform events into TenXObjects synchronously using their input stream calling thread.
  #  - < 1: interpreted as a percentage of the number of available cores (e.g., 0.5 = use up to 50% of available cores)
  #  - = 1: allocate a single dedicated to transform events.
  #  - > 1: interpreted as a fixed number of threads (e.g., 10 = 10 threads)
  threadPoolSize: "0.5"

  # 'batchSize' specifies the maximum number of events to queue for concurrent processing before flushing.
  #  If 'threadPoolSize' is 1, this value is unused, and events are transformed into TenXObject synchronously 
  #  to their calling input's thread.  If 0, flush pending events after 'parallelEventFlushInterval' expires or the
  #  source input reaches end-of-file.
  batchSize: 1000

  # 'flushInterval'  specifies the maximum wait duration before flushing queued events 
  #  If 'parallelThreadPoolSize' is 1, this value is unused, and events are transformed into TenXObject synchronously 
  #  to their input thread. If 0, no wait flush interval is applied. 
  flushInterval: 2s

  # 'processingTimeout' specifies the maximum wait duration before dropping un-processed queued events 
  #  This value provides a backstop for overflowing the heap if the pipeline cannot dequeue 
  #  pending events to transform into TenXObjects. If 0, no timeout is applied.
  processingTimeout: 30s

Options

Specify the options below to configure the Parallel processor:

Name	Description
parallelEventThreadPoolSize	Number of threads to allocate to transform events into TenXObjects concurrently
parallelEventBatchSize	Maximum number of events to queue for concurrent processing before flushing
parallelEventFlushInterval	Maximum wait duration before flushing queued events
parallelEventProcessingTimeout	Maximum wait duration before dropping un-processed queued events

`parallelEventThreadPoolSize`

Number of threads to allocate to transform events into TenXObjects concurrently.

Type	Default
String	""

Controls the number of threads to allocate to transform events into TenXObjects concurrently.

If the value is -

Setting	Allocated Threads
0	Transform events into TenXObjects synchronously using their input stream calling thread
1	Interpreted as a percentage of the number of available cores (e.g., 0.5 = use up to 50% of available cores)
1	Allocate a single dedicated to transform events
1	Interpreted as a fixed number of threads (e.g., 10 = 10 threads)

`parallelEventBatchSize`

Maximum number of events to queue for concurrent processing before flushing.

Type	Default
Number	1000.0

Sets the number of events to queue before flushing them to transform into TenXObjects. If threadPoolSize is 1, this value is unused, and events are transformed into TenXObject synchronously to their calling input's thread. If 0, flush pending events after flushInterval expires or the source input reaches end-of-file.

`parallelEventFlushInterval`

Maximum wait duration before flushing queued events.

Type	Default
String	2s

Sets an interval period for (e.g. '2s) before flushing pending events to transform into TenXObjects. If threadPoolSize is 1, this value is unused, and events are transformed into TenXObject synchronously to their input thread. If 0, no wait flush interval is applied.

`parallelEventProcessingTimeout`

Maximum wait duration before dropping un-processed queued events.

Type	Default
String	30s

Sets a timeout period (e.g., '30s'), after which to drop queued events. This value provides a backstop for overflowing the heap if the pipeline cannot dequeue pending events to transform into TenXObject. If 0, no timeout is applied.

This unit is defined in parallelize/unit.yaml.