Skip to content

Run

Run the dev app on your log files locally to preview how edge and cloud apps will process them. No API key needed for previewing savings on local logs.

Setup Guide

Step 1: Install
Step 2: Set Environment Variables

Set these environment variables before running. See path configuration for details.

Variable Description
TENX_CONFIG Path to your configuration directory
TENX_API_KEY Your Log10x API key (get one). Not needed for previewing savings on local logs. 
export TENX_CONFIG=/path/to/your/config
export TENX_API_KEY=your-api-key  # skip if just previewing local logs

See best practices for managing secrets in production.

Step 3: Set Up Input Logs

Download sample logs (20 MB OpenTelemetry k8s):

mkdir -p $TENX_CONFIG/data/sample/input
curl -o $TENX_CONFIG/data/sample/input/otel-sample.log \
  https://log10x-public-assets.s3.amazonaws.com/samples/otel-k8s/medium/input/otel-sample.log

Copy your log files to the input directory:

mkdir -p $TENX_CONFIG/data/sample/input
cp /path/to/your/logs/* $TENX_CONFIG/data/sample/input/
Step 4: Symbol Paths (optional)

Symbol libraries tell the engine how to recognize log patterns from your frameworks and applications. A default library with 150+ frameworks is included. Set symbolPaths to add your own.

Step 5: Enrichments (optional)

Enrichments add context (severity, HTTP codes) for richer aggregation reports.

To enable:

  1. In the app config, uncomment the desired enrichment include entries
  2. In the initializers section below, configure each enrichment's settings
Step 6: Run

Best for: Quick local testing and development.

The CLI automatically loads configuration from $TENX_CONFIG:

tenx @apps/dev

Best for: Isolated testing with local configuration files.

Mount your config directory and pass environment variables:

docker run --rm \
  -v $TENX_CONFIG:/etc/tenx/config/ \
  -e TENX_CONFIG=/etc/tenx/config/ \
  -e TENX_API_KEY=${TENX_API_KEY} \
  ghcr.io/log-10x/pipeline-10x:latest \
  @apps/dev

Skip -e TENX_API_KEY if just previewing savings on local logs.

Best for: CI/CD pipelines with version-controlled configuration.

Use the @github macro to pull configuration from a repository:

docker run --rm \
  -e TENX_API_KEY=${TENX_API_KEY} \
  ghcr.io/log-10x/pipeline-10x:latest \
  '@github={"token": "<gh-token>", "repo": "my-user/my-repo"}' \
  @apps/dev

Skip -e TENX_API_KEY if just previewing savings on local logs.

Step 7: Preview Savings

Generate a shareable link to visualize your results in the Log10x Console:

python $TENX_MODULES/apps/dev/extract_for_web.py \
  $TENX_CONFIG/data/sample/output --open

The script generates a console URL and opens it in your browser — cost per event type, ROI projections, and top patterns by volume.

For privacy, use --local instead of --open — all processing stays on your machine, no data sent externally. Run with --help for all options.

Step 8: Analyze Output (Optional)

The File Output generates the following files in $TENX_CONFIG/data/sample/output:

Statistical analysis of your log patterns: top event types by volume and bytes, error rates, and level distributions.

See analysis tables for sample output.

Your original events losslessly compact (typically 50-70% volume reduction).

See reduction ratios for sample output.

Unique TenXTemplate patterns discovered in your logs, enabling lossless volume reduction.

See template analysis for sample output.

Perfect reconstruction of compact events, validating lossless expanding.

To validate lossless encoding:

# Clear input and copy encoded output
rm -f $TENX_CONFIG/data/sample/input/*.log
cp $TENX_CONFIG/data/sample/output/encoded.log $TENX_CONFIG/data/sample/input/

# Re-run to decode
tenx @apps/dev

# Compare with original—files match exactly
diff $TENX_CONFIG/data/sample/output/decoded.log /path/to/original.log

Configuration

To configure the Dev app, Edit these settings:

Main Config

Main Config

The main config file loads the Dev app's required modules.

Uncomment selected modules at: dev/config.yaml.

Edit Online

Edit Dev Config Locally 

#
# 🔟❎ dev app main config

# The dev app locally tests, and validates structured event processing.

# To learn more see https://doc.log10x.com/apps/dev

# ============================ Bootstrap Runtime ==============================

tenx: run

runtimeName: $=TenXEnv.get("TENX_RUNTIME_NAME", "myDev")

# ============================ Load App Modules ===============================

# Uncomment and edit selected config.yaml files (e.g., run/input/file/config.yaml)

include:

# ------------------------------ App settings ---------------------------------

  # Load general app settings:
  - dev

  # Load gitops settings for pulling additional assets from Github.
  # Edit the gitops/config.yaml file with specific config
  #
  # For additional info see - https://doc.log10x.com/config/github
  - gitops

# ------------------------------ Open Inputs ----------------------------------

  # read log/trace events from inputs to transform into well-defined TenXObjects:

  - run/input/file       # https://doc.log10x.com/run/input/file
  # - run/input/stdin    # https://doc.log10x.com/run/input/stdin

# --------------------------- Enrich TenXObjects ------------------------------

  # Uncomment and edit config files to enrich TenXObjects with calculated fields:

  - run/initialize/inputField   # https://doc.log10x.com/run/initialize/#inputfield
  - run/initialize/level        # https://doc.log10x.com/run/initialize/level
  - run/initialize/group        # https://doc.log10x.com/run/initialize/group
  - run/initialize/message      # https://doc.log10x.com/run/initialize/message
  - run/initialize/httpCode     # https://doc.log10x.com/run/initialize/httpCode
  - run/initialize/lookup       # https://doc.log10x.com/run/initialize/lookup
  - run/initialize/custom       # https://doc.log10x.com/run/transform/script/object
  - run/initialize/k8s          # https://doc.log10x.com/run/initialize/k8s
  # - run/initialize/geoIP        # https://doc.log10x.com/run/initialize/geoIP

# --------------------------- Output TenXObjects -------------------------------

  # Uncomment and edit to write TenXObjects summaries to output(s):

  - run/output/event/file          # https://doc.log10x.com/run/output/event/file
  # - run/output/event/stdout      # https://doc.log10x.com/run/output/event/stdout
  # - run/output/event/process     # https://doc.log10x.com/run/output/event/process
  # - run/output/event/http        # https://doc.log10x.com/run/output/event/http
  # - run/output/event/unix        # https://doc.log10x.com/run/output/event/unix
  # - run/output/event/forward     # https://doc.log10x.com/run/output/event/forward
Inputs

Inputs

Activate Event inputs to read events from local/remote sources to transform into TenXObjects.

file

Configure File inputs to read events from an input file/glob path.

Below is the default configuration from: file/config.yaml.

Edit Online

Edit File input Config Locally 

# 🔟❎ 'run' File input configuration

# Configure file/glob event inputs. To learn more https://doc.log10x.com/run/input/file

# NOTE: input files are read once and are not tailed. To continuously tail files
# use 10x in conjunction with any of the supported file forwarders in:
# https://doc.log10x.com/run/input/forwarder (e.g., Fluentd/Bit, Logstash, Filebeat)

# Set the 10x pipeline to 'run'
tenx: run

# =============================== Dependencies ================================

include: run/modules/input/file

# =============================== File Options ================================

# Multiple file inputs can be defined below
inputFile:

  # ---------------------------- General Options ----------------------------

    # 'name' sets a unique logical name across all pipeline inputs
  - name: fileInput

  # ---------------------------- File Options ---------------------------------

    # 'path' specifies an absolute/canonical glob file path.
    path: $=path("data/sample/input/[!{.}]*.log")

    # ---------------------------- Parsing Options ----------------------------

    # 'extractors' defines a list of JSON/regex extractor names
    #  used to capture and redact event values from 'path' to transform 
    #  into TenXObjects. To learn more see: https://doc.log10x.com/run/input/extract         
    extractors: [  
    ]

    # 'sourcePattern' defines a regex pattern that captures a 'source' value
    #  for each event read from a file to transform into an TenXObject.
    #  This value will be used to sequence all events sharing this 'source' value correctly 
    #  so they are not mixed with events originating from different locations.
    sourcePattern: null  

    # 'sourceFields'  defines a list of JSON fields to extract from matching capture groups of the inputFileSourcePattern
    # or the entire event if not set as the 'source' value for each event read from this input. 
    sourceFields: [
    ]

    # --------------------------- Ancillary Options ---------------------------

    # 'printProgress' controls whether a progress gage is printed to the console    
    printProgress: true

stdin

Configure the stdin device input to read events from stdin.

Below is the default configuration from: stdin/config.yaml.

Edit Online

Edit stdin device input Config Locally 

# 🔟❎ 'run' Stdin input configuration

# Configure a stdin event input. To learn more see https://doc.log10x.com/run/input/stdin/

# Set the 10x pipeline to 'run'
tenx: run

# =============================== Dependencies ================================

include: run/modules/input/stdin

# =============================== Stdin Options ===============================

# NOTE: a pipeline should configure no more than a single stdin input

stdin:

  # ---------------------------- Parsing Options ----------------------------

    # 'extractors' defines a list of JSON/regex extractor names
    #  used to capture and redact event values from 'path' to transform 
    #  into TenXObjects. To learn more see: https://doc.log10x.com/run/input/extract         
  extractors: [
  ]

  # 'sourcePattern' defines a regex pattern that captures a 'source' value
  #  for each event read from the file and transformed into an TenXObject.
  #  This value will be used to sequence all events sharing this 'source' value correctly 
  #  so they are not mixed with events originating from different locations.    
  sourcePattern: null
Initializers

Initializers

Activate an Object initializer to enrich TenXObjects with additional context.

message

Configure the Symbol Message Calculator to enrich TenXObjects with logical message symbol sequence and origin values.

Below is the default configuration from: message/config.yaml.

Edit Online

Edit Symbol Message Calculator Config Locally 

# 🔟❎ 'run' symbol lookup configuration

# Configure a symbol origin lookup to enrich TenXObjects
# To learn more see https://doc.log10x.com/run/initialize/message/

# Set the 10x pipeline to 'run'
tenx: run

# =============================== Dependencies ================================

include: run/modules/initialize/message

# ============================== Symbol Options ===============================

symbol:

  # 'types' specifies the types of symbols to search for; if the first type does not yield a result, the next one is tried etc.
  #  Supported values (case insensitive), PACKAGE, CLASS, METHOD, LOG, ENUM, CONST, TEXT, EXEC, ANY
  #  To learn more see https://doc.log10x.com/run/transform/symbol/#contexts
  contexts: log,exec

  # 'messageField' specifies the field name to assign the source/binary origin file name to target TenXObjects
  messageField: message_pattern

  # 'maxLen' specifies the max char len of the output messageField
  maxLen: 70

level

Configure the Level Classifier to classify TenXObjects with a severity level.

Below is the default configuration from: level/config.yaml.

Edit Online

Edit Level Classifier Config Locally 

# 🔟❎ 'run' level classifier configuration

# Configuration level classifier to enrich TenXObjects
# To learn more see https://doc.log10x.com/run/initialize/level/

# Set the 10x pipeline to 'run'
tenx: run

# =============================== Dependencies ================================

include: run/modules/initialize/level

# =============================== Level Options ===============================

level:

  # 'field' specify the field name to assign with the inferred severity level
  field: severity_level

  # 'terms' specify severity level classification, start-of-line matching
  #  Used to identify the severity level (TRACE, DEBUG, INFO, WARN, ERROR, CRITICAL) of a log line, typically marking
  #  the start of a log event (head). Sorted by likelihood and uniqueness. Used in a Trie for O(m) matching.
  terms:
    - 'trace=TRACE'          # Common in JavaScript (Winston) or custom logs; e.g., "trace Variable x = 5"
    - 'Trace=TRACE'          # Mixed case; e.g., "Trace Variable x = 5"
    - 'TRACE=TRACE'          # Uppercase, less common; e.g., "TRACE Message"
    - 'debug=DEBUG'          # Common in JavaScript (Winston) or custom logs; e.g., "debug Entering method foo"
    - 'Debug=DEBUG'          # Mixed case; e.g., "Debug Entering method foo"
    - 'DEBUG=DEBUG'          # Uppercase; e.g., "DEBUG Message"
    - 'DBG=DEBUG'            # Abbreviation, less common; e.g., "DBG Entering method foo"
    - 'info=INFO'            # Common in JavaScript (Winston) or custom logs; e.g., "info Application started"
    - 'Info=INFO'            # Mixed case; e.g., "Info Application started"
    - 'INFO=INFO'            # Uppercase; e.g., "INFO Message"
    - 'notice=INFO'          # Common in custom logs or lowercase syslog; e.g., "notice Configuration updated"
    - 'Notice=INFO'          # Mixed case; e.g., "Notice Configuration updated"
    - 'NOTICE=INFO'          # Uppercase, syslog; e.g., "NOTICE Message"
    - 'warn=WARN'            # Common in JavaScript (Winston) or custom logs; e.g., "warn Low disk space"
    - 'Warn=WARN'            # Mixed case; e.g., "Warn Low disk space"
    - 'WARN=WARN'            # Uppercase; e.g., "WARN Message"
    - 'warning=WARN'         # Common in custom logs or verbose frameworks; e.g., "warning Low disk space"
    - 'Warning=WARN'         # Mixed case; e.g., "Warning Low disk space"
    - 'WARNING=WARN'         # Uppercase; e.g., "WARNING Message"
    - 'error=ERROR'          # Common in JavaScript (Winston) or custom logs; e.g., "error Failed to connect"
    - 'Error=ERROR'          # Mixed case; e.g., "Error Failed to connect"
    - 'ERROR=ERROR'          # Uppercase; e.g., "ERROR Message"
    - 'err=ERROR'            # Lowercase abbreviation; e.g., "err Failed to connect"
    - 'ERR=ERROR'            # Uppercase abbreviation; e.g., "ERR Message"
    - 'TypeError=ERROR'      # JavaScript, stack trace; e.g., "TypeError: undefined is not a function"
    - 'ReferenceError=ERROR' # JavaScript, stack trace; e.g., "ReferenceError: x is not defined"
    - 'ValueError=ERROR'     # Python, stack trace; e.g., "ValueError: invalid literal"
    - 'PHP_Warning=ERROR'    # PHP, warning message; e.g., "PHP Warning: Undefined variable"
    - 'fatal=CRITICAL'       # Common in custom logs; e.g., "fatal System crash"
    - 'Fatal=CRITICAL'       # Mixed case; e.g., "Fatal System crash"
    - 'FATAL=CRITICAL'       # Uppercase; e.g., "FATAL Message"
    - 'critical=CRITICAL'    # Common in custom logs; e.g., "critical System crash"
    - 'Critical=CRITICAL'    # Mixed case; e.g., "Critical System crash"
    - 'CRITICAL=CRITICAL'    # Uppercase; e.g., "CRITICAL Message"
    - 'crit=CRITICAL'        # Lowercase abbreviation, syslog; e.g., "crit System crash"
    - 'CRIT=CRITICAL'        # Uppercase abbreviation, syslog; e.g., "CRIT Message"
    - 'alert=CRITICAL'       # Common in custom logs or syslog; e.g., "alert High CPU usage"
    - 'Alert=CRITICAL'       # Mixed case; e.g., "Alert High CPU usage"
    - 'ALERT=CRITICAL'       # Uppercase, syslog; e.g., "ALERT Message"
    - 'emerg=CRITICAL'       # Common in custom logs or lowercase syslog; e.g., "emerg System unusable"
    - 'Emerg=CRITICAL'       # Mixed case; e.g., "Emerg System unusable"
    - 'EMERG=CRITICAL'       # Uppercase, syslog; e.g., "EMERG Message"
    - 'panic=CRITICAL'       # Go panic, first line; e.g., "panic: runtime error: index out of range"
    - 'fatal_error=CRITICAL' # Go fatal error, first line; e.g., "fatal error: all goroutines are asleep"
    - 'thread_main_panicked_at=CRITICAL' # Rust panic, first line; e.g., "thread 'main' panicked at 'explicit panic'"
    - 'Segmentation_fault=CRITICAL' # C/C++ memory violation, first line; e.g., "Segmentation fault (core dumped)"
    - 'Aborted=CRITICAL'     # C/C++ termination, first line; e.g., "Aborted (core dumped)"
    - 'Assertion_failed=CRITICAL' # C/C++ assertion failure, first line; e.g., "Assertion failed: x > 0"
    - 'PHP_Fatal_error=CRITICAL' # PHP fatal error, first line; e.g., "PHP Fatal error: Out of memory"
    - 'Traceback_most_recent_call_last=CRITICAL' # Python stack trace, first line; e.g., "Traceback (most recent call last)"
    - 'Stack_trace=CRITICAL' # Generic stack trace, first line; e.g., "Stack trace"
    - 'Error=CRITICAL'       # JavaScript, stack trace; e.g., "Error: Something went wrong"
    - 'Fatal_error=CRITICAL' # PHP, stack trace; e.g., "Fatal error: Uncaught Exception"
    - 'Unhandled_exception=CRITICAL' # C#/Dart, stack trace; e.g., "Unhandled exception: System.NullReferenceException"
    - 'Exception_in_thread=CRITICAL' # Java, stack trace; e.g., "Exception in thread 'main' java.lang.RuntimeException"
    - 'goroutine=CRITICAL'   # Go, stack trace; e.g., "goroutine 1 [running]:"
    - 'failed=ERROR'         # ERROR-level entries in application, system, or web server logs

  # 'timestampPatterns' patterns for LogEvent head classification, start-of-line matching
  #  Used to identify timestamp formats in log lines and infer their implied severity level based on the first character.
  #  Formats are typical of Kubernetes kube-apiserver logs with varying microsecond precision. The first character
  #  ('I', 'W', 'E', 'F') determines the severity: I=INFO, W=WARN, E=ERROR, F=CRITICAL. Sorted by precision.
  timestampPatterns:
    - "'I'MMdd HH:mm:ss.S=INFO"          # Used in Kubernetes kube-apiserver logs with INFO prefix, 1-digit microsecond precision
    - "'I'MMdd HH:mm:ss.SS=INFO"         # Used in Kubernetes kube-apiserver logs with INFO prefix, 2-digit microsecond precision
    - "'I'MMdd HH:mm:ss.SSS=INFO"        # Used in Kubernetes kube-apiserver logs with INFO prefix, 3-digit microsecond precision
    - "'I'MMdd HH:mm:ss.SSSS=INFO"       # Used in Kubernetes kube-apiserver logs with INFO prefix, 4-digit microsecond precision
    - "'I'MMdd HH:mm:ss.SSSSS=INFO"      # Used in Kubernetes kube-apiserver logs with INFO prefix, 5-digit microsecond precision
    - "'I'MMdd HH:mm:ss.SSSSSS=INFO"     # Used in Kubernetes kube-apiserver logs with INFO prefix, 6-digit microsecond precision
    - "'W'MMdd HH:mm:ss.S=WARN"          # Used in Kubernetes kube-apiserver logs with WARNING prefix, 1-digit microsecond precision
    - "'W'MMdd HH:mm:ss.SS=WARN"         # Used in Kubernetes kube-apiserver logs with WARNING prefix, 2-digit microsecond precision
    - "'W'MMdd HH:mm:ss.SSS=WARN"        # Used in Kubernetes kube-apiserver logs with WARNING prefix, 3-digit microsecond precision
    - "'W'MMdd HH:mm:ss.SSSS=WARN"       # Used in Kubernetes kube-apiserver logs with WARNING prefix, 4-digit microsecond precision
    - "'W'MMdd HH:mm:ss.SSSSS=WARN"      # Used in Kubernetes kube-apiserver logs with WARNING prefix, 5-digit microsecond precision
    - "'W'MMdd HH:mm:ss.SSSSSS=WARN"     # Used in Kubernetes kube-apiserver logs with WARNING prefix, 6-digit microsecond precision
    - "'E'MMdd HH:mm:ss.S=ERROR"         # Used in Kubernetes kube-apiserver logs with ERROR prefix, 1-digit microsecond precision
    - "'E'MMdd HH:mm:ss.SS=ERROR"        # Used in Kubernetes kube-apiserver logs with ERROR prefix, 2-digit microsecond precision
    - "'E'MMdd HH:mm:ss.SSS=ERROR"       # Used in Kubernetes kube-apiserver logs with ERROR prefix, 3-digit microsecond precision
    - "'E'MMdd HH:mm:ss.SSSS=ERROR"      # Used in Kubernetes kube-apiserver logs with ERROR prefix, 4-digit microsecond precision
    - "'E'MMdd HH:mm:ss.SSSSS=ERROR"     # Used in Kubernetes kube-apiserver logs with ERROR prefix, 5-digit microsecond precision
    - "'E'MMdd HH:mm:ss.SSSSSS=ERROR"    # Used in Kubernetes kube-apiserver logs with ERROR prefix, 6-digit microsecond precision
    - "'F'MMdd HH:mm:ss.S=CRITICAL"      # Used in Kubernetes kube-apiserver logs with FATAL prefix, 1-digit microsecond precision
    - "'F'MMdd HH:mm:ss.SS=CRITICAL"     # Used in Kubernetes kube-apiserver logs with FATAL prefix, 2-digit microsecond precision
    - "'F'MMdd HH:mm:ss.SSS=CRITICAL"    # Used in Kubernetes kube-apiserver logs with FATAL prefix, 3-digit microsecond precision
    - "'F'MMdd HH:mm:ss.SSSS=CRITICAL"   # Used in Kubernetes kube-apiserver logs with FATAL prefix, 4-digit microsecond precision
    - "'F'MMdd HH:mm:ss.SSSSS=CRITICAL"  # Used in Kubernetes kube-apiserver logs with FATAL prefix, 5-digit microsecond precision
    - "'F'MMdd HH:mm:ss.SSSSSS=CRITICAL" # Used in Kubernetes kube-apiserver logs with FATAL prefix, 6-digit microsecond precision

group

Configure the Group initializer to combine multi-line events into TenXObject group instances.

Below is the default configuration from: group/config.yaml.

Edit Online

Edit Group initializer Config Locally 

# 🔟❎ 'run' event grouping configuration

# Group  sequences of TenXObjects to filter, aggregate and output as a single logical unit.
# To learn more see https://doc.log10x.com/run/transform/group/

# Set the 10x pipeline to 'run'
tenx: run

# =============================== Dependencies ================================

include: run/modules/initialize/group

# =============================== Group Options ===============================


# 'indicators'  specifies a list of `value:state` pairs that determine if a log line's start marks a group head (`true`) or child (`false`).
group:

  # 'indicators' specifies a list of strings that, when matched at the start of a log line's text, designate it as a group head.
  # Unmatched lines default to false, indicating they are group children.
  indicators:
    - '192.'              # Indicates a private IP address, common in web server logs (e.g., "192.168.1.1 - - [...]")
    - '10.'               # Indicates a private IP range, often in Kubernetes or internal network logs (e.g., "10.244.0.125 - - [...]")
    - '172.'              # Indicates a private IP range, typical in enterprise network logs (e.g., "172.16.0.1 - - [...]")
    - '127.'              # Indicates localhost, a common web server log initiator (e.g., "127.0.0.1 - - [...]")
    - 'GET '              # Indicates an HTTP GET request, marking the start of a web transaction log (e.g., "GET /index.html HTTP/1.1")
    - 'POST '             # Indicates an HTTP POST request, marking the start of a web transaction log (e.g., "POST /api HTTP/1.1")
    - 'PUT '              # Indicates an HTTP PUT request, marking the start of a web transaction log (e.g., "PUT /resource HTTP/1.1")
    - 'DELETE '           # Indicates an HTTP DELETE request, marking the start of a web transaction log (e.g., "DELETE /resource HTTP/1.1")
    - 'HEAD '             # Indicates an HTTP HEAD request, marking the start of a web transaction log (e.g., "HEAD /index.html HTTP/1.1")
    - 'OPTIONS '          # Indicates an HTTP OPTIONS request, marking the start of a web transaction log (e.g., "OPTIONS /api HTTP/1.1")
    - 'HTTP/'             # Indicates an HTTP protocol version, marking the start of a web transaction log (e.g., "GET /index.html HTTP/1.1")
    - 'kernel:'           # Indicates a Linux kernel log entry, marking the start of a system event (e.g., "kernel: [0.123456] Device initialized")
    - 'sshd['             # Indicates an SSH daemon log entry, marking the start of a security event (e.g., "sshd[1234]: Accepted password ...")
    - 'systemd['          # Indicates a systemd service log entry, marking the start of a system service event (e.g., "systemd[1]: Started service ...")
    - 'cron['             # Indicates a cron daemon log entry, marking the start of a scheduled task event (e.g., "cron[1234]: Running job ...")
    - 'syslog:'           # Indicates a syslog message, marking the start of a system log event (e.g., "syslog: Message ...")
    - 'rsyslogd:'         # Indicates an rsyslog daemon log entry, marking the start of a logging system event (e.g., "rsyslogd: Log started ...")
    - 'auditd['           # Indicates an audit daemon log entry, marking the start of a security audit event (e.g., "auditd[1234]: Audit event ...")
    - 'daemon:'           # Indicates a syslog daemon facility log, marking the start of a system service event (e.g., "daemon: Service started ...")
    - 'user:'             # Indicates a syslog user facility log, marking the start of a user-related event (e.g., "user: User logged in ...")
    - 'local0:'           # Indicates a syslog local facility log, marking the start of a custom system event (e.g., "local0: Custom message ...")
    - 'local1:'           # Indicates a syslog local facility log, marking the start of a custom system event (e.g., "local1: Custom message ...")
    - 'level='            # Indicates a structured log key, marking the start of a key-value log entry (e.g., "level=info msg=Started")
    - 'msg='              # Indicates a structured log key, marking the start of a message log entry (e.g., "msg=Application started")
    - 'message='          # Indicates a structured log key, marking the start of a message log entry (e.g., "message=Application started")
    - 'event='            # Indicates a structured log key, marking the start of an event log entry (e.g., "event=Service startup")
    - 'thread='           # Indicates a structured log key, marking the start of a thread-specific log entry (e.g., "thread=main Processing ...")
    - 'Starting '         # Indicates the start of a process initiation log (e.g., "Starting server on port 8080")
    - 'Stopping '         # Indicates the start of a process termination log (e.g., "Stopping service ...")
    - 'Running '          # Indicates the start of a process status log (e.g., "Running task ...")
    - 'Listening '        # Indicates the start of a network service log (e.g., "Listening on port 8080 ...")
    - 'Connecting '       # Indicates the start of a connection attempt log (e.g., "Connecting to database ...")
    - 'Connected '        # Indicates the start of a successful connection log (e.g., "Connected to database ...")
    - 'Disconnected '     # Indicates the start of a disconnection log (e.g., "Disconnected from server ...")
    - 'Processing '       # Indicates the start of a task processing log (e.g., "Processing request ...")
    - 'Received '         # Indicates the start of a data reception log (e.g., "Received message ...")
    - 'Sent '             # Indicates the start of a data transmission log (e.g., "Sent response ...")
    - 'User '             # Indicates the start of a user action log (e.g., "User logged in ...")
    - 'Authentication '   # Indicates the start of an authentication log (e.g., "Authentication successful ...")
    - 'Authorized '       # Indicates the start of an authorization log (e.g., "Authorized user access ...")
    - 'Failed '           # Indicates the start of a failure log (e.g., "Failed login attempt ...")
    - 'kubelet:=true'     # Indicates a Kubernetes kubelet log entry, marking the start of a node event (e.g., "kubelet: Starting kubelet")
    - 'pod:=true'         # Indicates a Kubernetes pod log entry, marking the start of a pod event (e.g., "pod: Starting container")
    - 'container:=true'   # Indicates a Kubernetes container log entry, marking the start of a container event (e.g., "container: Started")
    - 'namespace:=true'   # Indicates a Kubernetes namespace log entry, marking the start of a namespace event (e.g., "namespace: Created")
    - 'Traceback '        # Indicates the start of a Python stack trace (e.g., "Traceback (most recent call last):")
    - 'File "'            # Indicates a Python stack trace line (e.g., "File "/script.py", line 10")
    - 'Trace:'            # Indicates a Node.js console trace (e.g., "Trace: Show me")
    - 'Error:'            # Indicates a Node.js error log (e.g., "Error: Something went wrong")
    - 'Warning:'          # Indicates a Node.js warning log (e.g., "Warning: Deprecated method")
    - 'Exception in thread ' # Indicates a Java exception header (e.g., "Exception in thread 'main'")
    - 'goroutine '        # Indicates a Go goroutine stack trace (e.g., "goroutine 1 [running]:")
    - 'panic:'            # Indicates a Go panic (e.g., "panic: runtime error")
    - 'thread '           # Indicates a Rust thread panic (e.g., "thread 'main' panicked at")
    - 'stack backtrace:'  # Indicates a Rust backtrace header (e.g., "stack backtrace:")
    - 'PHP Warning:'      # Indicates a PHP warning (e.g., "PHP Warning: Undefined variable")
    - 'PHP Fatal error:'  # Indicates a PHP fatal error (e.g., "PHP Fatal error: Out of memory")
    - 'in /'              # Indicates a PHP/Ruby file path in stack trace (e.g., "in /path/to/file.php:10")
    - 'Exception:'        # Indicates a C# exception (e.g., "Exception: Invalid operation")
    - 'terminate called'  # Indicates a C++ termination (e.g., "terminate called after throwing an instance of 'std::exception'")
    - 'syslog:'           # Indicates a Linux syslog message (e.g., "syslog: Message ...")
    - 'IN='               # Indicates an iptables firewall input log (e.g., "IN=eth0 OUT=")
    - 'OUT='              # Indicates an iptables firewall output log (e.g., "OUT=eth0 SRC=")
    - 'SRC='              # Indicates an iptables firewall source IP log (e.g., "SRC=192.168.1.1 DST=")
    - 'DST='              # Indicates an iptables firewall destination IP log (e.g., "DST=10.0.0.1 LEN=")
    - 'KafkaServer:'      # Indicates a Kafka server log (e.g., "KafkaServer: Starting Kafka server")
    - 'Redis:'            # Indicates a Redis server log (e.g., "Redis: Server initialized")

  # 'negators' specifies patterns that mark a line as a continuation of a previous event (e.g., stack trace lines).
  # Lines matching these patterns are NOT standalone events and typically don't need their own message pattern.
  # Events that are NOT negators (isStandalone=true) will get a message even without timestamp/severity.
  negators:
    # Java/Kotlin/Scala stack traces
    - "\tat "             # Java stack frame (e.g., "\tat com.example.MyClass.method(MyClass.java:42)")
    - "    at "           # Java stack frame with spaces (e.g., "    at com.example.MyClass.method")
    - "\t... "            # Java truncated stack (e.g., "\t... 15 more")
    - "    ... "          # Java truncated stack with spaces
    - "Caused by: "       # Java chained exception (e.g., "Caused by: java.io.IOException")
    - "Suppressed: "      # Java suppressed exception (e.g., "Suppressed: java.lang.Exception")

    # .NET/C# stack traces
    - "   at "            # .NET stack frame (e.g., "   at MyNamespace.MyClass.Method()")
    - "--- End of "       # .NET inner exception marker (e.g., "--- End of inner exception stack trace ---")
    - " ---> "            # .NET inner exception chain

    # Python stack traces
    - "  File \""         # Python stack frame (e.g., '  File "/path/to/script.py", line 10')
    - "    raise "        # Python raise statement in traceback
    - "    return "       # Python return statement in traceback

    # Node.js/JavaScript stack traces
    - "    at "           # Node.js stack frame (e.g., "    at Object.<anonymous> (/path/file.js:10:15)")
    - "    at new "       # Node.js constructor stack frame
    - "    at Module."    # Node.js module stack frame
    - "    at Object."    # Node.js object stack frame

    # Go stack traces
    - "\t/"               # Go stack frame path (e.g., "\t/go/src/main.go:42 +0x123")
    - "created by "       # Go goroutine creator (e.g., "created by main.main")

    # Ruby stack traces
    - "\tfrom "           # Ruby stack frame (e.g., "\tfrom /path/to/file.rb:10:in `method'")
    - "        from "     # Ruby stack frame with spaces

    # Rust stack traces
    - "             at "  # Rust source location (13 spaces + at)
    - "  --> "            # Rust error pointer

    # PHP stack traces
    - "#0 "               # PHP stack frame (e.g., "#0 /path/to/file.php(10): function()")
    - "#1 "               # PHP stack frame
    - "#2 "               # PHP stack frame
    - "#3 "               # PHP stack frame
    - "#4 "               # PHP stack frame
    - "#5 "               # PHP stack frame
    - "  thrown in "      # PHP exception location

httpCode

Configure the HTTP Status Code Extractor to enrich TenXObjects with an extracted HTTP status code.

Below is the default configuration from: httpCode/config.yaml.

Edit Online

Edit HTTP Status Code Extractor Config Locally 

# 🔟❎ 'run' HTTP code classifier configuration

# The HTTP code classifier enriches TenXObjects with an extracted HTTP numeric response code (e.g., 404) field.

# Set the 10x pipeline to 'run'
tenx: run

# =============================== Dependencies ================================

include: run/modules/initialize/httpCode

# =============================== GeoIP Options ===============================

httpCode:

  # 'field' specifies the name of the field in which to assign an extracted HTTP status code, if found
  field: http_code

  # 'precludersBefore' specifies an array of characters or patterns that, when appearing immediately before a candidate HTTP status code in tokenized logs, indicate it is not a valid HTTP code.
  #  Used to preclude false positives like negative numbers ('-200'), floats ('.200'), or variables ('_200').
  #  Entries are single characters or short strings for O(1) lookup efficiency in the heuristic.
  precludersBefore: ['+', '$', '#', '@', '(', '[', '~', '-', '.', ':']

  # 'precludersAfter' specifies an array of characters or patterns that, when appearing immediately after a candidate HTTP status code in tokenized logs, indicate it is not a valid HTTP code.
  #  Used to preclude false positives like percentages ('200%'), floats ('200.'), or expressions ('200+').
  #  Entries are single characters or short strings for O(1) lookup efficiency in the heuristic.
  precludersAfter: ['%', ')', ']', '=', '!', '?', '+', '-', '.']

  # 'keywords' specifies an array of keywords indicating HTTP-related context in log events.
  #  Used to filter logs likely containing HTTP status codes, reducing false positives.
  #  Keywords cover HTTP methods, common log fields, and format-specific terms from sources like Apache, Nginx, HAProxy, Squid, IIS, Spring Boot.
  keywords:
    - HTTP/             # Indicates HTTP protocol version in request lines (e.g., "GET / HTTP/1.1").
    - GET               # HTTP method for retrieving resources.
    - POST              # HTTP method for submitting data.
    - PUT               # HTTP method for updating resources.
    - DELETE            # HTTP method for deleting resources.
    - HEAD              # HTTP method for retrieving headers only.
    - OPTIONS           # HTTP method for describing communication options.
    - TRACE             # HTTP method for diagnostic purposes.
    - PATCH             # HTTP method for partial updates.
    - request           # Common in log fields referring to HTTP requests (e.g., "requestMethod").
    - Request           # Title case variant for HTTP requests
    - response          # Common in log fields referring to HTTP responses (e.g., "responseSize").
    - Response          # Title case variant for HTTP responses
    - status            # Common key for HTTP status codes (e.g., "status":200).
    - Status            # Title case variant for HTTP status
    - code              # Shorthand for status code (e.g., "code":404).
    - Code              # Title case variant for status code
    - upstream_status   # Nginx-specific for backend status codes.
    - Upstream_status   # Title case variant for Nginx backend status
    - Upstream_Status   # Title case variant for Nginx backend status
    - http_status       # JSON key for HTTP status (e.g., "http_status":500).
    - Http_status       # Title case variant for JSON key for HTTP status
    - Http_Status       # Title case variant for JSON key for HTTP status
    - status_code       # Alternative key for status codes (e.g., "status_code":429).
    - statusCode        # Camel case variant for status codes (e.g., "statusCode":200)
    - Completed         # Spring Boot DispatcherServlet log indicator (e.g., "Completed 200 OK").
    - access            # Indicates access logs (e.g., Apache access log).
    - Access            # Title case variant for access logs
    - error             # Indicates error logs containing HTTP errors.
    - Error             # Title case variant for error logs
    - method            # Refers to HTTP request method (e.g., "method":"GET").
    - Method            # Title case variant for HTTP method
    - path              # Refers to request path or URI (e.g., "path":"/api").
    - Path              # Title case variant for request path
    - url               # Refers to request URL (e.g., "url":"http://example.com").
    - Url               # Title case variant for request URL
    - URL               # Title case variant for request URL
    - referrer          # HTTP referrer header field.
    - referrer          # Title case variant for referrer header field.
    - Referer           # Alternative spelling for HTTP referrer
    - Referer           # Title case variant for alternative spelling
    - user-agent        # HTTP user-agent header field.
    - User-agent        # Title case variant for user-agent
    - User-Agent        # Title case variant for user-agent
    - latency           # Common in logs for request/response time (e.g., HAProxy, ELB).
    - Latency           # Title case variant for latency
    - frontend          # HAProxy term for incoming connections.
    - Frontend          # Title case variant for HAProxy frontend
    - backend           # HAProxy term for upstream servers.
    - Backend           # Title case variant for HAProxy backend
    - TCP_MISS          # Squid hierarchy code prefix (e.g., "TCP_MISS/200").
    - W3SVC             # IIS log prefix (e.g., "W3SVC1").
    - DispatcherServlet # Spring Boot servlet class for HTTP handling.

  # 'validValues' specifies an array of all standard HTTP status codes as defined by the IANA HTTP Status Code Registry and relevant RFCs (e.g., RFC 9110, RFC 6585, RFC 7231-7235, RFC 7538, RFC 7725, RFC 8297, RFC 9111).
  #  Used to validate potential HTTP status codes in log events, ensuring only recognized codes are extracted.
  #  Each entry includes the numeric code followed by a comment describing its meaning and purpose.
  validValues:
    - '200' # OK: Standard response for successful HTTP requests.
    - '304' # Not Modified: Resource has not been modified since last requested.
    - '404' # Not Found: Server cannot find the requested resource.
    - '301' # Moved Permanently: Resource has been moved permanently to a new URI.
    - '302' # Found: Resource temporarily located at a different URI.
    - '403' # Forbidden: Client does not have access rights to the content.
    - '400' # Bad Request: Server cannot process due to client error.
    - '500' # Internal Server Error: Generic server error.
    - '206' # Partial Content: Server is delivering only part of the resource due to a range header.
    - '503' # Service Unavailable: Server temporarily unable to handle request.
    - '401' # Unauthorized: Authentication required and has failed or not provided.
    - '204' # No Content: Server processed request but no content is returned.
    - '502' # Bad Gateway: Server received an invalid response from upstream.
    - '201' # Created: Request has been fulfilled, resulting in new resource creation.
    - '429' # Too Many Requests: Client has sent too many requests in a given time.
    - '308' # Permanent Redirect: Resource permanently at another URI, method unchanged.
    - '307' # Temporary Redirect: Resource temporarily at another URI, method unchanged.
    - '303' # See Other: Response can be found under a different URI using GET.
    - '405' # Method Not Allowed: Request method is not supported for the resource.
    - '406' # Not Acceptable: Server cannot produce a response matching client’s Accept headers.
    - '408' # Request Timeout: Server timed out waiting for the request.
    - '409' # Conflict: Request conflicts with current state of the resource.
    - '410' # Gone: Resource is permanently unavailable.
    - '413' # Content Too Large: Request entity is larger than server limits.
    - '414' # URI Too Long: Request-URI is longer than the server can handle.
    - '415' # Unsupported Media Type: Media format of the requested data is not supported.
    - '416' # Range Not Satisfiable: Requested range cannot be fulfilled.
    - '422' # Unprocessable Content: Request is well-formed but semantically incorrect.
    - '504' # Gateway Timeout: Server, as gateway, did not get timely upstream response.
    - '505' # HTTP Version Not Supported: Server does not support the HTTP version.
    - '100' # Continue: Client should continue with request.
    - '101' # Switching Protocols: Server is switching protocols as requested.
    - '202' # Accepted: Request accepted for processing, but processing is not complete.
    - '203' # Non-Authoritative Information: Server is a transforming proxy with non-authoritative metadata.
    - '205' # Reset Content: Client should reset the document view.
    - '300' # Multiple Choices: Indicates multiple options for the resource.
    - '411' # Length Required: Content-Length header is required but not provided.
    - '412' # Precondition Failed: Server does not meet one of the preconditions.
    - '417' # Expectation Failed: Server cannot meet Expect header requirements.
    - '421' # Misdirected Request: Request sent to server unable to produce a response.
    - '423' # Locked: Resource is locked (WebDAV).
    - '424' # Failed Dependency: Request failed due to failure of a previous request.
    - '425' # Too Early: Server unwilling to risk processing a request that might be replayed.
    - '426' # Upgrade Required: Client should switch to a different protocol.
    - '428' # Precondition Required: Server requires conditional request headers.
    - '431' # Request Header Fields Too Large: Headers exceed server limits.
    - '451' # Unavailable For Legal Reasons: Resource access blocked for legal reasons.
    - '501' # Not Implemented: Server does not support the requested functionality.
    - '506' # Variant Also Negotiates: Server configuration error in content negotiation.
    - '507' # Insufficient Storage: Server cannot store the representation (WebDAV).
    - '508' # Loop Detected: Server detected an infinite loop in request processing.
    - '510' # Not Extended: Further extensions required for request fulfillment.
    - '511' # Network Authentication Required: Client needs to authenticate for network access.
    - '102' # Processing: Server has received and is processing the request.
    - '103' # Early Hints: Used to return some response headers before final HTTP message.
    - '207' # Multi-Status: Conveys information about multiple resources in WebDAV.
    - '208' # Already Reported: Used inside a DAV binding to avoid enumerating bindings repeatedly.
    - '226' # IM Used: Server has fulfilled a request for the resource using instance-manipulations.
    - '305' # Use Proxy: Resource is available only through a proxy.
    - '306' # (Unused): Previously used for Switch Proxy, now reserved.
    - '402' # Payment Required: Reserved for future use, e.g., digital payments.
    - '407' # Proxy Authentication Required: Client must authenticate with proxy.
    - '418' # (Unused): Previously “I’m a teapot” (RFC 2324, joke).

lookup

Configure the File Lookup to enrich TenXObjects with lookup table values.

Below is the default configuration from: lookup/config.yaml.

Edit Online

Edit File Lookup Config Locally 

# 🔟❎ 'run' lookup file configuration

# Configuration text lookup to enrich TenXObjects
# To learn more see https://doc.log10x.com/run/initialize/lookup/

# Set the 10x pipeline to 'run'
tenx: run

# =============================== Dependencies ================================

include: run/modules/initialize/lookup

# ============================== Lookup Options ===============================

lookup:

  # 'file' specifies the .csv/.tsv lookup table file
  file: $=path("pipelines/run/initialize/lookup/http.csv")

  # 'keyField' specifies the field whose value is query lookup key.
  #  The calculated code field below captures the penultimate HTTP code value from a web log event.
  #  For example, 200 for: Mar 10 12:34:56 webserver nginx: 192.168.1.10 - - [10/Mar/2025:12:34:56 +0000] "GET /index.html HTTP/1.1" 200 1024
  keyField: $=yield TenXEnv.get("httpCodeField")

  # 'valueField' specifies the name of the result field to assign to the lookup table value
  valueField: http_message

k8s

Configure the k8s Context Extractor to enrich TenXObjects with k8s pod and container information.

Below is the default configuration from: k8s/config.yaml.

Edit Online

Edit k8s Context Extractor Config Locally 

# 🔟❎ 'run' k8s lookup configuration

# Configure a k8s field extractors to enrich TenXObjects
# To learn more see https://doc.log10x.com/run/initialize/k8s/

# Set the 10x pipeline to 'run'
tenx: run

# =============================== Dependencies ================================

include: run/modules/initialize/k8s

# ============================== Symbol Options ===============================

k8s:

  # 'extractorName' specifies which extractor (e.g., fluentK8s/filebeatK8s) to use for k8s metadata extraction
  extractorName: fluentK8s # extract k8s context using Fluent schema

  # 'namespaceNameField' specifies the name of the field in which to assign an extracted k8 namespace name status code, if found
  namespaceNameField: k8s_namespace

  # 'containerNameField' specifies the name of the field in which to assign an extracted k8 container name status code, if found
  containerNameField: k8s_container

  # 'podNameField' specifies the name of the field in which to assign an extracted k8 pod name status code, if found
  podNameField: k8s_pod

geoIP

Configure the GeoIP lookup to geo-reference TenXObjects via their ipAddress value.

Below is the default configuration from: geoIP/config.yaml.

Edit Online

Edit GeoIP lookup Config Locally 

# 🔟❎ 'run' GeoIP lookup configuration

# The GeoIP lookup geo-references TenXObjects based on their ipAddress field value.
# To learn more see https://doc.log10x.com/api/js/#TenXObject+ipAddress

# Set the 10x pipeline to 'run'
tenx: run

# =============================== Dependencies ================================

include: run/modules/initialize/geoIP

# =============================== GeoIP Options ===============================

geoIP:

  # 'file' specifies the MaxMind GeoIP DB (.mmdb) file to load.
  # To learn more see https://doc.log10x.com/api/js/#TenXLookup.loadGeoIPDB
  file: data/run/lookup/geo.mmdb

  # 'countryField' specifies the field name for country geo-reference data.
  # Defaults to "country". Set to "" to disable country enrichment.
  countryField: country

  # Uncomment the following fields to enable additional geo-reference data as needed:
  # continentField: continent
  # subdivisionField: subdivision
  # cityField: city
  # postalField: postal
  # latitudeField: latitude
  # longitudeField: longitude
Event Outputs

Event Outputs

Activate Event stream outputs to write TenXObject and template values to event stream outputs (e.g., stdout, files).

file

Configure File outputs to write TenXObject field/template values to a file.

Below is the default configuration from: file/config.yaml.

Edit Online

Edit File output Config Locally 

# 🔟❎ 'run' file output configuration

# Configure file outputs to write TenXObject instance and template field values.
# To learn more see https://doc.log10x.com/run/output/event/file

# Set the 10x pipeline to 'run'
tenx: run

# =============================== Dependencies ================================

include:
  - run/modules/output/event/file

# ============================== File Options =================================

# Multiple file outputs can be defined below
outputFile:

  # --------------------------- 10x Encoding Output ---------------------------

    # Encode TenXObjects: https://doc.log10x.com/run/transform/#template-encoding

    # 'path' specifies a path on disk to write TenXObject field values
  - path: $=path("data/sample/output") + "/encoded.log"

    # 'filter' sets a JavaScript expression that TenXObjects must evaluate as truthy to write to the output.
    #  To learn more see https://doc.log10x.com/run/output/regulate/#filter-expressions
    filter: null

    # 'fields' defines the fields to write for each TenXObject sent to this output.
    #  To learn more see https://doc.log10x.com/run/output/stream/#outputfields  
    fields: 
      - encode() # https://doc.log10x.com/api/js/#TenXObject+encode

  # ----------------------------- Decode Output ------------------------------

    # Template-decode TenXObjects: https://doc.log10x.com/run/transform/#expand

  - path: $=path("data/sample/output") + "/decoded.log"

    filter: isEncoded  # https://doc.log10x.com/api/js/#TenXBaseObject+isEncoded   

    fields: 
      - text           # https://doc.log10x.com/api/js/#TenXBaseObject+text

  # ---------------------------- Template Output ----------------------------

  # Writes TenXTemplate values: https://doc.log10x.com/run/transform/#template

  - path: $=path("data/sample/output") + "/templates.json"

    # 'writeTemplates' controls whether to write new templates for TenXObjects written to this output.
    writeTemplates: true

    # 'append' controls whether to overwrite or append to the output file
    append: true

  # ----------------------------- Summary Output ------------------------------

    # Writes aggregated TenXSummaries: https://doc.log10x.com/run/aggregate

  - path: $=path("data/sample/output") + "/aggregated.csv"

    # 'header' is set to write the field names aggregated into the first line of the output
    #  We specify '$=yield' to wait until the 'enrichmentFields' env var have been evaluated 
    header: $=yield TenXString.join(",", TenXEnv.get("enrichmentFields"), "summaryVolume", "summaryBytes", "summaryTotals") + "\n"

    filter: isSummary                            # https://doc.log10x.com/api/js/#TenXBaseObject+isSummary                        

    fields: 
      - $=yield TenXEnv.get("enrichmentFields")  # https://doc.log10x.com/run/initialize/#enrichmentFields
      - summaryVolume                            # https://doc.log10x.com/api/js/#TenXSummary+summaryVolume
      - summaryBytes                             # https://doc.log10x.com/api/js/#TenXSummary+summaryBytes
      - summaryTotals                            # https://doc.log10x.com/api/js/#TenXSummary+summaryTotals

stdout

Configure Stdout/err outputs to write TenXObject and template values to a log4j2 stdout appender.

Below is the default configuration from: stdout/config.yaml.

Edit Online

Edit Stdout/err output Config Locally 

# 🔟❎ 'run' stdout output configuration

# Configure stdout/err streams for writing TenXObject instance and template values.
# To learn more see https://doc.log10x.com/run/output/event/stdout

# Set the 10x pipeline to 'run'
tenx: run

# =============================== Dependencies ================================

include: run/modules/output/event/stdout

# ============================= Stdout Options ================================

# Multiple stdout outputs can be defined below
stdout:

  # ---------------------------10x Encoding Output ----------------------------

    # Encode TenXObjects: https://doc.log10x.com/run/transform/#template-encoding

    # 'target' can be SYSTEM_OUT or SYSTEM_ERR
  - target: SYSTEM_OUT

    # 'filter' sets a JavaScript expression that TenXObjects must evaluate as truthy to write to the output.
    #  To learn more see https://doc.log10x.com/run/output/regulate/#filter-expressions
    filter: isObject                

    # 'fields' defines the fields to write for each TenXObject sent to this output.
    #  To learn more see https://doc.log10x.com/run/output/stream/#outputfields  
    fields: 
      - encode() # https://doc.log10x.com/api/js/#TenXObject+encode

  # ----------------------------- Decode Output -------------------------------

    # Template-decode TenXObjects: https://doc.log10x.com/run/transform/#expand

  - target: SYSTEM_OUT

    filter: isEncoded  # https://doc.log10x.com/api/js/#TenXBaseObject+isEncoded   

    fields: 
      - text           # https://doc.log10x.com/api/js/#TenXBaseObject+text

  # ---------------------------- Template Output ------------------------------

    # Writes TenXTemplate values: https://doc.log10x.com/run/transform/#template

  - target: SYSTEM_OUT

    # 'writeTemplates' controls whether to write new TenXObjects templates to stdout. 
    # To learn more see https://doc.log10x.com/run/output/event/stdout/#stdoutwritetemplates
    writeTemplates: true 

  # ----------------------------- Summary Output ------------------------------

    # Writes aggregated TenXSummaries: https://doc.log10x.com/run/aggregate/

  - target: SYSTEM_OUT

    filter: isSummary   # https://doc.log10x.com/api/js/#TenXBaseObject+isSummary                        

    fields: 
      - $=yield TenXEnv.get("enrichmentFields") # https://doc.log10x.com/run/initialize/#enrichmentFields
      - summaryVolume                           # https://doc.log10x.com/api/js/#TenXSummary+summaryVolume
      - summaryBytes                            # https://doc.log10x.com/api/js/#TenXSummary+summaryBytes
      - summaryTotals                           # https://doc.log10x.com/api/js/#TenXSummary+summaryTotals

Advanced Settings

To configure advanced options (optional) for the Dev app, Edit these settings:

Bootstrap

Bootstrap

Configure the Pipeline Bootstrapper to authenticate the log10x account and launch a target pipeline.

Below is the default configuration from: bootstrap/config.yaml.

Edit Online

Edit Pipeline Bootstrapper Config Locally 

# 🔟❎ run bootstrap configuration

# This config file specifies bootstrap options for the run pipeline.
# To learn more see https://doc.log10x.com/run/bootstrap

# To learn more see https://doc.log10x.com/config/

tenx: run

# =============================== Launch Settings ==============================

# 'apiKey' specifies an api key used authenticating against the 10x service
apiKey: $=TenXEnv.get("TENX_API_KEY", "NO-API-KEY")

# 'includePaths' specifies folders on disk for resolving relative config file/folder references in addition to the working folder
includePaths: []

# 'quiet' disables printing version information to the console.
# quiet: true

# 'jarFiles' specifies .jar files to dynamically load for use by compile, input and output API extensions.
jarFiles: []

# 'metricEndpoint' specifies the Prometheus endpoint to report usage/health metrics (enterprise version only).
# metricEndpoint: https://prometheus.log10x.com/api/v1/write

# 'disabledArgs' specifies a list of launch arguments that are disallowed from either command line or user config files.
disabledArgs: []

# 'debugEnvVars' list environment variables to debug
debugEnvVars: []
Github Sync

Github Sync

Below is the default configuration from: gitops/config.yaml (* Required Fields).

Edit Online

Edit config.yaml Locally 

# 🔟❎ GitOps configuration

# This config file pulls symbol, config and lookup files from a remote repo to
# enable centralized configuration management via GitHub.

# To learn more see https://doc.log10x.com/config/github

# The target 10x pipeline to run (e.g., run, compile) is specified in the config files pulled from the repo
tenx: any

include:

  - source: github

    options:
      # 'enabled' controls whether to enable this pull config
      enabled: false # (❗ REQUIRED)

      # 'token' specifies an access token for accessing the Github repo.
      #  To learn more see https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens
      token: $=TenXEnv.get("GH_TOKEN")

      # 'repo' specifies the GitHub user/repo from which to pull
      repo: null # (❗ REQUIRED)

      # 'branch' on 'repo' (if null, uses default repo default branch)
      branch: main

       # 'paths' specifies Glob patterns within 'repo' to pull and extract to a temp folder.
      paths:
        - test
        # - config/*.yaml      # https://doc.log10x.com/config/yaml
        # - symbols/*.10x.tar  # https://doc.log10x.com/run/symbol
        # - lookups/*.csv      # https://doc.log10x.com/api/js/#TenXLookup
        # - scripts/*.js       # https://doc.log10x.com/api/js

      # 'syncInterval' specifies an interval by which check to check the remote branch for updates (null to disable)
      #  This interval works in conjunction with: https://doc.log10x.com/run/reload
      syncInterval: 5min
Symbols

Symbols

Below is the default configuration from: symbol/config.yaml.

Edit Online

Edit config.yaml Locally 

# 🔟❎ 'run' symbol file configuration

# Loads symbol library files to transform events into well-defined TenXObjects.
# To learn more see https://doc.log10x.com/run/symbol

# Set the 10x pipeline to 'run'
tenx: run

# ============================ Symbol Options =================================

symbol:

  # 'paths' specifies the file/folder locations to scan for symbol library files.
  #  To learn more see https://doc.log10x.com/run/symbol/#symbolpaths
  paths:
    - $=path("data/shared/symbols", false)
    - $=path("<TENX_SYMBOLS_PATH>",  false)

  literals: []
Aggregators

Aggregators

Configure Aggregators to aggregate and summarize TenXObjects to publish as metrics.

Below is the default configuration from: dev/config.yaml.

Edit Online

Edit Aggregator Config Locally 

# 🔟❎ 'run' dev app aggregator config

# This file configures aggregators for the dev app.
# To learn more see https://doc.log10x.com/run/aggregate/

# Set the 10x pipeline to 'run'
tenx: run

# ============================== Aggregator Options ===========================

# Multiple aggregators can be defined below
aggregator:

  - name: devAggregator

    # 'filter' sets a JavaScript expression that TenXObjects must evaluate as truthy to be aggregated.
    #  To learn more see https://doc.log10x.com/run/output/regulate/#filter-expressions
    filter: null

    # 'fields' set the names of intrinsic/extracted/calculated fields whose combined values
    #  identify a unique row within the aggregator's in-mem table for
    #  each object passed to it. This aggregator defaults to enrichment fields.
    #  To learn more see https://doc.log10x.com/run/initialize/#enrichmentFields
    fields:
      - $=yield TenXEnv.get("enrichmentFields")

    # 'totalFields' specifies fields to extract and tally into generated TenXSummaries
    totalFields: []

    # 'flushThreshold' sets the max number of TenXObjects whose values to upsert into
    #  the aggregator's in-mem table. Once exceeded, the aggregator instantiates TenXSummaries 
    #  from the table rows to send to output and reset the table.
    flushThreshold: 0 

    # 'flushInterval' sets a periodic interval, after which the aggregator
    #  flushes the rows of its in-mem table into the pipeline as TenXSummary instances.
    flushInterval: null

    # 'maxCardinality' sets the maximum number of rows for the aggregator's
    #  in-mem table. If an TenXObjects whose 'values' are not contained in the table
    #  is passed to the aggregator, 'defaultValue' will be used for it instead
    #  until the aggregator flushes its in-mem table after 'flushInterval' or 'flushThreshold'
    #  are exceeded.
    maxCardinality: 65536

    # 'defaultValue' sets a constant value that is used instead of 'values' when
    #  'maxCardinality' is exceeded within 'flushInterval'
    defaultValue: unknown
Template

Template

Below is the default configuration from: template/config.yaml.

Edit Online

Edit config.yaml Locally 

# 🔟❎ 'run' template file configuration

# Load TenXTemplates .json files that define the structure/schema of TenXObjects.
# To learn more see https://doc.log10x.com/run/template/

# Set the 10x pipeline to 'run'
tenx: run

# =============================== Template ===================================

template:
  # 'files' specifies GLOB pattern for finding JSON-encoded TenXTemplates.
  files:
    - $=path("data/templates/*.json")
    - $=path("data/sample/output/*.json")

  # 'cacheSize' controls the maximum total byte size of templates held
  #  in the in-memory cache vs. on disk. Set to 0 to disable pruning.
  cacheSize: $=parseBytes("10MB")

# =============================== Variable ===================================

var:
  # 'placeholder' specifies a character to use when encoding a TenXTemplate
  #  to signify the location of a runtime variable value.
  placeholder: "$"

  # 'maxRecurIndexes' controls the maximum number of variable values to reuse.
  maxRecurIndexes: 10

# =============================== Timestamp ==================================

timestamp:
  # 'prefix' specifies a prefix for a TenXTemplate's timestamp tokens.
  prefix: (

  # 'postfix' specifies a postfix for a TenXTemplate's timestamp tokens.
  postfix: )
Transform

Transform

Configure the Transform to transform log and trace events into well-defined TenXObjects.

timestamp

Configure the Timestamp parser to extract alphanumeric and epoch timestamp values from input events.

Below is the default configuration from: timestamp/config.yaml.

Edit Online

Edit Timestamp parser Config Locally 

# 🔟❎ 'run' timestamp parser configuration

# Identify unix/alphanumeric timestamp structures within TenXTemplates.
# https://doc.log10x.com/run/transform/timestamp/

# Set the 10x pipeline to 'run'
tenx: run

# ============================= Timestamp Options =============================

timestamp:

  # 'maxPerObject' controls the max number of timestamps to add into an TenXObject's
  # 'timestamp' array. Set to 0 for unlimited.
  maxPerObject: 0

  # 'searchDirection' controls the direction(s) from with timestamps are searched for
  #  within the object's 'text' field. Possible values: 

  #  - fromStart: search for 'maxPerObject' from the start of 'text',  
  #    limiting search to 'searchLengthLimitFromStart' characters  

  #  - fromEnd: search backward for 'maxPerObject' from the end of 'text',
  #    limiting search to 'searchLengthLimitFromEnd' characters  

  #  - fromStartAndEnd: search for 'maxPerObject' from both the start and end of 'text'
  #    limiting search to 'searchLengthLimitFromStart' and 'searchLengthLimitFromEnd' respectively  

  #  - none: do not parse timestamps

  searchDirection: fromStartAndEnd

  # 'searchLengthLimitFromStart' limits the number of characters to search for
  #  timestamps from the beginning of the object's 'text' field. 0 for unlimited
  searchLengthLimitFromStart: 0

  # 'searchLengthLimitFromStart' limits the number of characters to search for
  #  timestamps from the end of the object's 'text' field. 0 for unlimited
  searchLengthLimitFromEnd: 0

  # 'zone' controls the timezone for formatting timestamp epoch
  #  values to string. The 'java.time.ZoneId.of(String zoneId)' 
  #  is used to obtain the timezone from the 'zone' value.
  #  If set to null, the host OS timezone is used. 
  zone: null

  # 'literals' contains an array of strings to treat as a part 
  #  of any timestamp candidate found when structuring an TenXObject.
  literals:
    - T
    - Z
    - I # Go INFO
    - E # Go ERROR
    - W # Go WARN

  # 'patterns' specifies an array of date-time formats to attempt when parsing timestamps from input events.
  #  Timestamp formats that appear frequently within an input stream can be 'bumped' higher within the list below.

  patterns:
  # Most common formats
  - "'I'MMdd HH:mm:ss.S"                       # Used in Kubernetes kube-apiserver logs with INFO prefix, 1-digit microsecond precision.
  - "'I'MMdd HH:mm:ss.SS"                      # Used in Kubernetes kube-apiserver logs with INFO prefix, 2-digit microsecond precision.
  - "'I'MMdd HH:mm:ss.SSS"                     # Used in Kubernetes kube-apiserver logs with INFO prefix, 3-digit microsecond precision.
  - "'I'MMdd HH:mm:ss.SSSS"                    # Used in Kubernetes kube-apiserver logs with INFO prefix, 4-digit microsecond precision.
  - "'I'MMdd HH:mm:ss.SSSSS"                   # Used in Kubernetes kube-apiserver logs with INFO prefix, 5-digit microsecond precision.
  - "'I'MMdd HH:mm:ss.SSSSSS"                  # Used in Kubernetes kube-apiserver logs with INFO prefix, 6-digit microsecond precision.
  - "'W'MMdd HH:mm:ss.S"                       # Used in Kubernetes kube-apiserver logs with WARNING prefix, 1-digit microsecond precision.
  - "'W'MMdd HH:mm:ss.SS"                      # Used in Kubernetes kube-apiserver logs with WARNING prefix, 2-digit microsecond precision.
  - "'W'MMdd HH:mm:ss.SSS"                     # Used in Kubernetes kube-apiserver logs with WARNING prefix, 3-digit microsecond precision.
  - "'W'MMdd HH:mm:ss.SSSS"                    # Used in Kubernetes kube-apiserver logs with WARNING prefix, 4-digit microsecond precision.
  - "'W'MMdd HH:mm:ss.SSSSS"                   # Used in Kubernetes kube-apiserver logs with WARNING prefix, 5-digit microsecond precision.
  - "'W'MMdd HH:mm:ss.SSSSSS"                  # Used in Kubernetes kube-apiserver logs with WARNING prefix, 6-digit microsecond precision.
  - "'E'MMdd HH:mm:ss.S"                       # Used in Kubernetes kube-apiserver logs with ERROR prefix, 1-digit microsecond precision.
  - "'E'MMdd HH:mm:ss.SS"                      # Used in Kubernetes kube-apiserver logs with ERROR prefix, 2-digit microsecond precision.
  - "'E'MMdd HH:mm:ss.SSS"                     # Used in Kubernetes kube-apiserver logs with ERROR prefix, 3-digit microsecond precision.
  - "'E'MMdd HH:mm:ss.SSSS"                    # Used in Kubernetes kube-apiserver logs with ERROR prefix, 4-digit microsecond precision.
  - "'E'MMdd HH:mm:ss.SSSSS"                   # Used in Kubernetes kube-apiserver logs with ERROR prefix, 5-digit microsecond precision.
  - "'E'MMdd HH:mm:ss.SSSSSS"                  # Used in Kubernetes kube-apiserver logs with ERROR prefix, 6-digit microsecond precision.
  - "'F'MMdd HH:mm:ss.S"                       # Used in Kubernetes kube-apiserver logs with FATAL prefix, 1-digit microsecond precision.
  - "'F'MMdd HH:mm:ss.SS"                      # Used in Kubernetes kube-apiserver logs with FATAL prefix, 2-digit microsecond precision.
  - "'F'MMdd HH:mm:ss.SSS"                     # Used in Kubernetes kube-apiserver logs with FATAL prefix, 3-digit microsecond precision.
  - "'F'MMdd HH:mm:ss.SSSS"                    # Used in Kubernetes kube-apiserver logs with FATAL prefix, 4-digit microsecond precision.
  - "'F'MMdd HH:mm:ss.SSSSS"                   # Used in Kubernetes kube-apiserver logs with FATAL prefix, 5-digit microsecond precision.
  - "'F'MMdd HH:mm:ss.SSSSSS"                  # Used in Kubernetes kube-apiserver logs with FATAL prefix, 6-digit microsecond precision.
  - "yyyy-MM-dd HH:mm:ss"                      # Widely used in Java applications, databases (e.g., MySQL, PostgreSQL), and application servers (e.g., Tomcat, JBoss).
  - "yyyy-MM-dd'T'HH:mm:ss.SSSZ"               # Common in web services, APIs, Java (with DateTimeFormatter.ISO_OFFSET_DATE_TIME), Python (with datetime.isoformat()), and systems requiring precise timestamps with timezone information.
  - "MMM dd HH:mm:ss"                          # Frequently seen in syslog, Unix-based systems, network devices (e.g., Cisco routers), and web servers (e.g., Apache, Nginx).
  - "dd/MMM/yyyy:HH:mm:ss Z"                   # Standard syslog format, used in firewalls (e.g., Cisco ASA, Palo Alto), network monitoring tools, and Unix-based systems.
  - "yyyy-MM-dd HH:mm:ss,SSS"                  # Common in Java logging frameworks like Log4j and Logback.
  - "yyyy-MM-dd'T'HH:mm:ss,SSS"                # Common in Java logging with ISO 8601 date-time, comma-separated milliseconds.
  - "MM/dd/yyyy HH:mm:ss"                      # Used in Windows event logs, .NET applications, and U.S.-based systems.
  - "EEE MMM dd HH:mm:ss yyyy"                 # Human-readable format used in various logs, including some web servers and application logs.
  - "yyyy-MM-dd'T'HH:mm:ss"                    # Simplified ISO 8601 format, used in many modern applications and frameworks.
  - "HH:mm:ss"                                 # Time-only format, used when the date is implied or provided separately, common in embedded systems and some programming languages.
  - "yyyyMMdd HH:mm:ss"                        # Compact format used in some legacy systems and batch processing logs.
  - "MMM dd, yyyy h:mm:ss a"                   # Human-readable format with 12-hour clock, used in application logs and some U.S.-based systems.
  - "MMM dd, yyyy hh:mm:ss a"                  # Human-readable format with 12-hour clock (padded hour), used in application logs and some U.S.-based systems.
  # Common formats with slight variations
  - "yyyy-MM-dd HH:mm:ss.SSS"                  # Extended precision format used in Java applications and databases requiring millisecond accuracy.
  - "dd-MMM-yyyy HH:mm:ss.SSS"                 # Common in Java logging frameworks (e.g., Log4j, SLF4J) and application logs requiring human-readable dates with millisecond precision.
  - "yyyy-MM-dd'T'HH:mm:ss.S'Z'"               # Microsecond precision ISO 8601 format, 1-digit precision.
  - "yyyy-MM-dd'T'HH:mm:ss.SS'Z'"              # Microsecond precision ISO 8601 format, 2-digit precision.
  - "yyyy-MM-dd'T'HH:mm:ss.SSS'Z'"             # Microsecond precision ISO 8601 format, 3-digit precision.
  - "yyyy-MM-dd'T'HH:mm:ss.SSSS'Z'"            # Microsecond precision ISO 8601 format, 4-digit precision.
  - "yyyy-MM-dd'T'HH:mm:ss.SSSSS'Z'"           # Microsecond precision ISO 8601 format, 5-digit precision.
  - "yyyy-MM-dd'T'HH:mm:ss.SSSSSS'Z'"          # Microsecond precision ISO 8601 format, 6-digit precision.
  - "yyyy-MM-dd'T'HH:mm:ss.S'Z'"               # Nanosecond precision ISO 8601 format, 1-digit precision.
  - "yyyy-MM-dd'T'HH:mm:ss.SS'Z'"              # Nanosecond precision ISO 8601 format, 2-digit precision.
  - "yyyy-MM-dd'T'HH:mm:ss.SSS'Z'"             # Nanosecond precision ISO 8601 format, 3-digit precision.
  - "yyyy-MM-dd'T'HH:mm:ss.SSSS'Z'"            # Nanosecond precision ISO 8601 format, 4-digit precision.
  - "yyyy-MM-dd'T'HH:mm:ss.SSSSS'Z'"           # Nanosecond precision ISO 8601 format, 5-digit precision.
  - "yyyy-MM-dd'T'HH:mm:ss.SSSSSS'Z'"          # Nanosecond precision ISO 8601 format, 6-digit precision.
  - "yyyy-MM-dd'T'HH:mm:ss.SSSSSSS'Z'"         # Nanosecond precision ISO 8601 format, 7-digit precision.
  - "yyyy-MM-dd'T'HH:mm:ss.SSSSSSSS'Z'"        # Nanosecond precision ISO 8601 format, 8-digit precision.
  - "yyyy-MM-dd'T'HH:mm:ss.SSSSSSSSS'Z'"       # Nanosecond precision ISO 8601 format, 9-digit precision.
  - "yyyy-MM-dd'T'HH:mm:ssZZZZZ"               # ISO 8601 with timezone offset, used in systems where timezone information is critical.
  - "yyyy-MM-dd HH:mm:ss.SSSZZZZZ"             # Similar to above but without the 'T' separator, used in database logs and application servers.
  - "yyyy-MM-dd HH:mm:ss.S"                    # Microsecond precision format, 1-digit precision.
  - "yyyy-MM-dd HH:mm:ss.SS"                   # Microsecond precision format, 2-digit precision.
  - "yyyy-MM-dd HH:mm:ss.SSS"                  # Microsecond precision format, 3-digit precision.
  - "yyyy-MM-dd HH:mm:ss.SSSS"                 # Microsecond precision format, 4-digit precision.
  - "yyyy-MM-dd HH:mm:ss.SSSSS"                # Microsecond precision format, 5-digit precision.
  - "yyyy-MM-dd HH:mm:ss.SSSSSS"               # Microsecond precision format, 6-digit precision.
  - "MM/dd/yyyy*HH:mm:ss*SSS"                  # Used in some U.S.-based systems, particularly in legacy applications or specific logging frameworks.
  - "M/d/yyyy h:mm:ss a:SSS"                   # Common in systems using 12-hour time format, such as some Windows applications or older logging systems.
  - "M/d/yyyy hh:mm:ss a:SSS"                  # Common in systems using 12-hour time format with milliseconds and padded hour, such as some Windows applications.
  - "M/dd/yyyy hh:mm:ss a"                     # Similar to above, used in systems where millisecond precision is not needed.
  - "yyyy-MM-dd'T'HH:mm:ss.SSSX"               # ISO 8601 with basic timezone offset format, used in systems that require standardized timestamps.
  - "yyyy-MM-dd'T'HH:mm:ss.SSSz"               # ISO 8601 with timezone name, used in applications that need to display timezone information.
  - "yyyy-MM-dd'T'HH:mm:ss'Z'"                 # ISO 8601 format assuming UTC, used in systems where all times are in UTC.
  - "yy/MM/dd HH:mm:ss"                        # Two-digit year format, used in compact logs or older systems.
  - "MMM dd HH:mm:ss ZZZZ"                     # Used in some Unix-based systems and web servers, includes timezone offset.
  - "HH:mm:ss,SSS"                             # Time-only format with milliseconds, used in performance logs or systems where date is provided separately.
  - "yyyy-MM-dd*HH:mm:ss"                      # Used in some application logs where the separator is a space or asterisk.
  - "yyyy MMM dd HH:mm:ss.SSS"                 # Human-readable format with milliseconds, used in some application logs.
  - "dd/MMM/yyyy HH:mm:ss"                     # Used in European systems and some web applications.
  - "yyyy-MM-dd'T'HH:mm:ss.SSS''Z''"           # ISO 8601 with milliseconds and literal 'Z', used in systems where 'Z' is explicitly included.
  - "MMM dd yyyy HH:mm:ss"                     # Human-readable format, used in various logs.
  - "yyyy-MM-dd HH:mm:ss ZZZZ"                 # Format with timezone offset, used in systems requiring timezone information.
  - "yyyy-MM-dd HH:mm:ssZZZZZ"                 # Similar to above, used in application logs.
  - "dd MMM yyyy HH:mm:ss"                     # European format, used in some web applications and databases.
  - "MMdd_HH:mm:ss"                            # Compact format without separators, used in file names or space-constrained logs.
  - "yyyy-MM-dd HH:mm:ss,SSSZZZZZ"             # Java logging format with timezone offset.
  - "yyyyMMdd HH:mm:ss.SSS"                    # Compact format with milliseconds, used in some legacy systems.
  - "yyyy/MM/dd HH:mm:ss"                      # Format used in some Asian systems, particularly in Japan.
  - "dd/MM/yyyy HH:mm:ss"                      # Common in European systems, including some web applications and databases.
  - "MM-dd-yyyy HH:mm:ss"                      # U.S. format variant, used in some older systems.
  - "yyyyMMddHHmmss"                           # Compact format for file names or database timestamps, used in systems where space is a concern.
  - "EEE, dd MMM yyyy HH:mm:ss zzz"            # RFC 1123 format, used in HTTP headers, web servers, and email systems.

fields

Configure the Field parser to scan TenXTemplates for JSON and KV fields.

Below is the default configuration from: fields/config.yaml.

Edit Online

Edit Field parser Config Locally 

# 🔟❎ 'run' TenXTemplate field extract configuration

# Configure how to extract JSON objects and KV structures from TenXTemplates.
# To learn more see https://doc.log10x.com/run/transform/fields/

# Set the 10x pipeline to 'run'
tenx: run

# =============================== Extract Options =============================

field:

  # 'extract' controls whether to scan TenXTemplate for  JSON
  #  objects or key-value lists (e.g., 'X=Y'). 
  extract: true

  # 'nameBreaks' controls which characters found to the left of a token
  #  whose is a candidate for being a 'key' in a KV field formation should serve 
  #  as a terminator for the search. For example, for an object whose 'text' field
  #  contains the following entry ',tx_result=OK' the desired key name should be 'tx_result'
  #  and as such, the name terminator character should be ',' vs. '_' (in which case
  #  the key would have been named 'result').
  nameBreaks: ', /\{}.()[]'

  # 'valueBreaks' controls which characters found to the right of a token
  #  whose is a candidate for being a 'value' in a KV field formation should serve 
  #  as a terminator for the search. For example, for an object whose 'text' field
  #  contains the following entry: 'status=RESULT_SUCCESS' the desired KV value 
  #  should be 'RESULT_SUCCESS', and as such, the value terminator character 
  #  should be ',' vs. '_' (in which case the value would be 'RESULT').
  valueBreaks: ', /\{}()[]'

symbol

Configure the Origin selector to select the source code/binary executable origin of TenXTemplate symbol values.

Below is the default configuration from: symbol/config.yaml.

Edit Online

Edit Origin selector Config Locally 

# 🔟❎ 'run' symbol origin configuration

# Identify the origin of symbol values within TenXTemplates.
# To learn more see https://doc.log10x.com/run/transform/symbol/

# Set the 10x pipeline to 'run'
tenx: run

# =============================== Origin Options ==============================

symbol:

  # 'maxOrigins' controls the number of source/binary origins to list 
  #  per symbol token sequence. As a series of tokens within 
  #  a target TenXTemplate (e.g., 'ERROR', 'could not connect to "{}') may appear 
  #  in multiple source/binary files within the pipeline's loaded library.
  #  The sorting algorithm configured below is used to select the 'maxSymbolOrigins' 
  #  topmost entries to list.
  maxOrigins: 64

  # 'sequenceReserved' defines a list of terms to ignore when searching
  #  for the symbol tokens that constitute an TenXTemplate's 'message' portion.
  #  For example, for an event whose text contains: 'connect success = true',
  #  the value 'true' will not be considered a part of the event's message,
  #  as the 'true' value is mostly likely the result of a variable boolean state.
  #  For more information, see: https://doc.log10x.com/api/js/#TenXObject+symbolSequence
  sequenceReserved:
    - "null"
    - "nil"
    - "true"
    - "false"
    - "to"
    - "the"
    - "a"
    - "at"
    - "for"
    - "log"
    - "info"
    - "http"  
    - TRACE
    - DEBUG
    - INFO
    - NOTICE
    - WARN
    - ERROR
    - CRITICAL
    - ALERT
    - EMERGENCY

  # ----------------------------- Debug Options -------------------------------

  debug:

    # 'symbol' debugging allows for verbose printing of the selection process
    #  for symbol tokens from the pipeline's 10x symbol files used to produce the
    #  results of 10x reflection functions.
    #  For more information, see: https://doc.log10x.com/api/js/#TenXObject+symbol


    # 'origins' outputs information about the origin selection process for an TenXTemplate symbol.
    #  For example, setting 'units' to 'foo.js' will output information about if/how it 
    #  was selected as the origin of TenXTemplate symbols.
    #  In other words, if a symbol (e.g., 'MyClass') has 'foo.java' as the source file
    #  from which it originated, adding 'foo.js' to 'units' will emit information
    #  about the selection process.
    #  Specifying '*' will emit information for all source /binary files that have
    #  been selected as the origin of any TenXTemplate objects within the pipeline.
    origins: [
    #  '*'
    ]

    # 'symbols' logs the origin selection process for symbol tokens within an TenXTemplate.
    #  For example, set 'symbols' to 'Could not connect to' to log how the 10x JavaScript 'symbolSequence' 
    #  determines the origin source code/binary file within the pipeline's symbol library. 

    #  Specifying '*' will emit information for all source code /binary files that have
    #  been selected as the origin of any TenXTemplate objects within the pipeline.
    symbols: [
     # '*'
    ]

group

Configure the Group sequencer to group and sequence TenXObjects.

Below is the default configuration from: group/config.yaml.

Edit Online

Edit Group sequencer Config Locally 

# 🔟❎ 'run' event grouping configuration

# Group  sequences of TenXObjects to filter, aggregate and output as a single logical unit.
# To learn more see https://doc.log10x.com/run/transform/group/

# Set the 10x pipeline to 'run'
tenx: run

# =============================== Group Options ===============================

group:

  # 'filters' specify JavaScript expressions an TenXObject instance/group must 
  #  evaluate as truthy against to be written to output
  filters: []

  # 'maxSize' defines the maximum number of TenXObjects to group
  #  before the group is sealed and forwarded into the pipeline. 
  #  Subsequent TenXObjects can form a new group.
  maxSize: 20000

  # 'flushTimeout' defines the max interval (e.g., 10s) to wait for 
  #  new events to be read from an input stream before it flushes any
  #  pending TenXObjects group into the pipeline.
  #  This mechanism is designed to avoid latencies in dispatching pending event
  #  groups to output destinations.
  flushTimeout: $=parseDuration("5s")

  # 'async' specifies whether to sequence and group TenXObjects in a dedicated thread
  async: true

parallelize

Configure the Parallel processor to distribute event parsing and transformation workloads across multiple cores..

Below is the default configuration from: parallelize/config.yaml.

Edit Online

Edit Parallel processor Config Locally 

# 🔟❎ 'run' event parallel processing configuration

# Transform log/trace events read from inputs into well-defined TenXObjects using multiple cores.
# To learn more see https://doc.log10x.com/run/transform/parallelize/

# Set the 10x pipeline to 'run'
tenx: run

# =============================== Parallel Options ============================

parallelEvent:

  # 'threadPoolSize' specifies the number of threads allocated to transform events into TenXObjects concurrently.
  #  If the value is -
  #  - = 0: transform events into TenXObjects synchronously using their input stream calling thread.
  #  - < 1: interpreted as a percentage of the number of available cores (e.g., 0.5 = use up to 50% of available cores)
  #  - = 1: allocate a single dedicated to transform events.
  #  - > 1: interpreted as a fixed number of threads (e.g., 10 = 10 threads)
  threadPoolSize: "0.5"

  # 'batchSize' specifies the maximum number of events to queue for concurrent processing before flushing.
  #  If 'threadPoolSize' is 1, this value is unused, and events are transformed into TenXObject synchronously 
  #  to their calling input's thread.  If 0, flush pending events after 'parallelEventFlushInterval' expires or the
  #  source input reaches end-of-file.
  batchSize: 1000

  # 'flushInterval'  specifies the maximum wait duration before flushing queued events 
  #  If 'parallelThreadPoolSize' is 1, this value is unused, and events are transformed into TenXObject synchronously 
  #  to their input thread. If 0, no wait flush interval is applied. 
  flushInterval: 2s

  # 'processingTimeout' specifies the maximum wait duration before dropping un-processed queued events 
  #  This value provides a backstop for overflowing the heap if the pipeline cannot dequeue 
  #  pending events to transform into TenXObjects. If 0, no timeout is applied.
  processingTimeout: 30s
Event Outputs

Event Outputs

Activate Event stream outputs to write TenXObject and template values to event stream outputs (e.g., stdout, files).

process

Configure Process outputs to write TenXObject and template values to a sub-process stdin.

Below is the default configuration from: process/config.yaml (* Required Fields).

Edit Online

Edit Process output Config Locally 

# 🔟❎ 'run' Fluent Bit output configuration

# Launch a Fluent Bit process into whose stdin pipe to write TenXObject instance/template field values.
# To learn more see https://doc.log10x.com/run/output/event/process/

# Set the 10x pipeline to 'run'
tenx: run

# =============================== Dependencies ================================

include: run/modules/output/event/process

# ============================ Fluent Bit Options =============================

# Multiple sub-process outputs can be defined below

procOut:

  # ----------------------------- Object Output -------------------------------

   # 'writeObjects' controls whether to write TenXObject field values to output.
  - writeObjects: false # (❗ REQUIRED)

    # -------------------------- Process Options ------------------------------

    # 'command' defines the OS-resolvable path of the sub-process to spawn
    command: /opt/fluent-bit/bin/fluent-bit

    args:
      - -i
      - stdin
      - -o
      - stdout

    # 'destroyMode' is set to 'cached' to allow a fluent-bit sub-process to
    #  to be reused across multiple 10x pipeline executions within a host JVM.
    destroyMode: cached

    # 'maxCacheSize' sets the size of the cache to used for caching fluent-bit instances
    maxCacheSize: 1

    # 'lazyLaunch' determines whether the fluent-bit instance will launch immediately
    # during the pipeline initialization, or only on-demand.
    lazyLaunch: true

    # 'startupWaitMs' and 'startupWaitPattern' controls the wait for the fluent-bit instance
    # to be ready for receiving events, before starting to emit to it.
    startupWaitMs: $=parseDuration("5s")
    startupWaitPattern: \[\s*sp\s*]\s+stream\s+processor\s+started\b

    # ------------------------------ Encoding ---------------------------------

    # 'filter' specifies an expression that any TenXObject passed to this output must
    #  evaluate as truthy against to write its instance/template values to stdout.
    #  For example, 'filter: this.timestamp' will only emit timestamped objects.
    #  To learn more see: https://doc.log10x.com/api/js/#TenXBaseObject+timestamped
    filter: null

    # 'fields' specifies a list of 10x intrinsic/calculated/extracted fields
    #  to extract from each target object passed to the output
    #  and written to stdout in the format specified by 'encodeType'
    #  If no fields are specified, encode/decode the target TenXObject.

    #  This example corresponds to the stdin format defined by fluent-bit in:
    #  https://docs.fluentbit.io/manual/pipeline/inputs/standard-input
    #  The 'text' argument below could be replaced by any list of intrinsic/calculated/extracted fields.
    #  The "" delimiter instructs 'joinFields' to produce JSON output.
    fields:  #
      - open="["
      - timestamp=dateFormat("+%s.%N")
      - separator=","
      - fields=joinFields("", "text")
      - close="]"

    # 'encodeType' specifies how 'fields' are encoded to stdout. Possible values:
    #    - 'json': formats names and values for the current TenXObject as: {"field":"str","field2":1, "field3": true}
    #    - 'delimited': formats values for the current TenXObject as: "str",1,true
    #                   The value separator (e.g., ',') is set by: 'encodeDelimiter'
    encodeType: delimited

    # 'encodeDelimiter' sets the char delimiter used to separate TenXObject
    #  field values when 'encodeType' is set to: 'delimited'
    encodeDelimiter: ' '

  # ----------------------------- Template Output -----------------------------

    # 'writeTemplates' controls whether to write new templates for TenXObjects written to
    #  this output. To learn more https://doc.log10x.com/run/template
  - writeTemplates: false  # (❗ REQUIRED)

    # -------------------------- Process Options ------------------------------

    # The output below writes the template values of TenXObjects
    # passed to this location. To learn more about TenXTemplates, see:
    # https://doc.log10x.com/run/template

    # 'command' defines the OS-resolvable path of the sub-process to spawn
    command: /opt/fluent-bit/bin/fluent-bit

    args:
      - -i
      - stdin
      - -o
      - stdout

    # 'destroyMode' is set to 'cached' to allow a fluent-bit sub-process to
    #  to be reused across multiple 10x pipeline executions within a host JVM.
    destroyMode: cached

    # 'maxCacheSize' sets the size of the cache to used for caching fluent-bit instances
    maxCacheSize: 1

    # 'lazyLaunch' determines whether the fluent-bit instance will launch immediately
    # during the pipeline initialization, or only on-demand.
    lazyLaunch: true

    # 'startupWaitMs' and 'startupWaitPattern' controls the wait for the fluent-bit instance
    # to be ready for receiving events, before starting to emit to it.
    startupWaitMs: $=parseDuration("5s")
    startupWaitPattern: \[\s*sp\s*]\s+stream\s+processor\s+started\b

http

Configure HTTP outputs to write TenXObject and template values to a log4j2 HTTP socket appender.

Below is the default configuration from: http/config.yaml (* Required Fields).

Edit Online

Edit HTTP output Config Locally 

# 🔟❎ 'run' HTTP log4j2 output configuration

# Configure HTTP outputs to write TenXObject instance and template values.
# To learn more see https://doc.log10x.com/run/output/event/http/

# Set the 10x pipeline to 'run' 
tenx: run

# =============================== Dependencies ================================

include: run/modules/output/event/http

# ============================== HTTP Options =================================

# Multiple HTTP outputs can be defined below
outputHttp:

  # ----------------------------- Object Output ------------------------------

    # 'writeObjects' controls whether to write TenXObject field values to output.
  - writeObjects: false # (❗ REQUIRED)

    # ---------------------------- HTTP Options -----------------------------

    # 'url' sets the target HTTP host to write TenXObject instance/template values.
    #  For example, the URL below posts to the Datadog log intake endpoint:
    #  https://http-intake.logs.datadoghq.com/v1/input?host=myHost&amp;ddsource=tenx&amp;service=loggingApp&ddtags=env:Dev
    url: http://127.0.0.1:8000

    # 'headers' are added to each HTTP request. For example,
    #  'DD-API-KEY=TenXEnv.get("apiKey"),DD-APPLICATION-KEY=TenXEnv.get("appKey")
    #  can be used to authenticate the Datadog intake example above.
    headers: [
       Content-Type=application/json    
    ]

    #'method' sets the HTTP method to use (e.g. GET, POST, DELETE)  
    method: POST

    # --------------------------- Encoding Options ----------------------------

    # 'filter' sets a JavaScript expression that TenXObjects must evaluate as truthy to write to the output.
    #  To learn more see https://doc.log10x.com/run/output/regulate/#filter-expressions
    filter: isObject                          

    # 'fields' defines the fields to write for each TenXObject sent to this output.
    #  To learn more see https://doc.log10x.com/run/output/stream/#outputfields 
    fields: 
      - encode() 

    # 'encodeType' specifies how 'fields' are encoded to output. Possible values: [json,delimited].
    #  To learn more see https://doc.log10x.com/run/output/event/outputStream/#outputstreamencodetype
    encodeType: json      

  # ---------------------------- Template Output ----------------------------

    # The HTTP output below writes new template of TenXObjects written to http url. 
    # To learn more about TenXTemplates, see https://doc.log10x.com/run/template

    # ---------------------------General Options -----------------------------

    # 'writeTemplates' controls whether to write new templates to this output. 
  - writeTemplates: false # (❗ REQUIRED)

    # ---------------------------- HTTP Options -------------------------------

    url: http://127.0.0.1:8000

    # 'headers' are added to each HTTP request the output makes. 
    headers: [
       Content-Type=application/json    
    ]

    #'method' sets the HTTP method to use (e.g. GET, POST, DELETE)  
    method: POST

unix

Configure Unix domain socket outputs to write TenXObject and template values to a log4j2 Unix domain socket appender.

Below is the default configuration from: unix/config.yaml (* Required Fields).

Edit Online

Edit Unix domain socket output Config Locally 

# 🔟❎ 'run' Unix socket output configuration

# Configure a UNIX socket output to write TenXObject instance and template field values.
# To learn more see https://doc.log10x.com/run/output/event/unix

# Set the 10x pipeline to 'run' 
tenx: run

# =============================== Dependencies ================================

include: run/modules/output/event/unix

# ============================== UNIX Socket Options ==========================

# Multiple UNIX socket outputs can be defined below
outputUnixSocket:

  # ----------------------------- Object Output ------------------------------

    # 'writeObjects' controls whether to write TenXObject field values to output.
  - writeObjects: false # (❗ REQUIRED)

    # ---------------------------- UNIX Options -----------------------------

    # 'url' sets the target socket address to write TenXObject instance values.
    address: /var/run/socket

    # --------------------------- Encoding Options ----------------------------

    # 'filter' sets a JavaScript expression that TenXObjects must evaluate as truthy to write to the output.
    #  To learn more see https://doc.log10x.com/run/output/regulate/#filter-expressions
    filter: isObject                          

    # 'fields' defines the fields to write for each TenXObject sent to this output.
    #  To learn more see https://doc.log10x.com/run/output/stream/#outputfields  
    fields: 
      - encode() 

    # 'encodeType' specifies how 'fields' are encoded to output. Possible values: [json,delimited].
    #  To learn more see https://doc.log10x.com/run/output/event/outputStream/#outputstreamencodetype
    encodeType: json      

  # ---------------------------- Template Output ----------------------------

    # 'writeTemplates' controls whether to write new templates for TenXObjects written to
    #  this output. To learn more see https://doc.log10x.com/run/template
  - writeTemplates: false

    # ---------------------------- UNIX Options -----------------------------

    # 'url' sets the target socket address to write new TenXTemplate values.
    address: /var/run/socket


This app is defined in dev/app.yaml.

Next: FAQ