Skip to content

Run

Run the dev app on your log files locally to preview how edge and cloud apps will process them. No API key needed for previewing savings on local logs.

Setup Guide

Step 1: Install
Step 2: Set Environment Variables

Set these environment variables before running. See path configuration for details.

Variable Description
TENX_CONFIG Path to your configuration directory
TENX_LICENSE_KEY Your Log10x license JWT (download from console). Not needed for previewing savings on local logs. 
export TENX_CONFIG=/path/to/your/config
export TENX_LICENSE_KEY="$(cat license.jwt)"   # skip if just previewing local logs

For production deployments, prefer mounting the license as a file and setting TENX_LICENSE_FILE, see deployment.

Step 3: Set Up Input Logs

Download sample logs (20 MB OpenTelemetry k8s):

mkdir -p $TENX_CONFIG/data/sample/input
curl -o $TENX_CONFIG/data/sample/input/otel-sample.log \
  https://log10x-public-assets.s3.amazonaws.com/samples/otel-k8s/medium/input/otel-sample.log

Copy your log files to the input directory:

mkdir -p $TENX_CONFIG/data/sample/input
cp /path/to/your/logs/* $TENX_CONFIG/data/sample/input/
Step 4: Symbol Paths (optional)

Symbol libraries tell the engine how to recognize log patterns from your frameworks and applications. A default library with 150+ frameworks is included. Set symbolPaths to add your own.

Step 5: Enrichments (optional)

Enrichments add context (severity, HTTP codes) for richer aggregation reports.

To enable:

  1. In the app config, uncomment the desired enrichment include entries
  2. In the initializers section below, configure each enrichment's settings
Step 6: Run

Best for: Quick local testing and development.

The CLI automatically loads configuration from $TENX_CONFIG:

tenx @apps/dev

Best for: Isolated testing with local configuration files.

Mount your config directory and pass environment variables:

docker run --rm \
  -v $TENX_CONFIG:/etc/tenx/config/ \
  -e TENX_CONFIG=/etc/tenx/config/ \
  -e TENX_LICENSE_KEY="$(cat license.jwt)" \
  log10x/pipeline-10x:latest \
  @apps/dev

Skip -e TENX_LICENSE_KEY to run with the image's built-in limited license (fine for previewing savings on local logs).

Best for: CI/CD pipelines with version-controlled configuration.

Use the @github macro to pull configuration from a repository:

docker run --rm \
  -e TENX_LICENSE_KEY="$(cat license.jwt)" \
  log10x/pipeline-10x:latest \
  '@github={"token": "<gh-token>", "repo": "my-user/my-repo"}' \
  @apps/dev

Skip -e TENX_LICENSE_KEY to run with the image's built-in limited license (fine for previewing savings on local logs).

Step 7: Preview Savings

When the pipeline completes, the Dev Output module automatically generates a console URL and opens it in your browser: cost per event type, ROI projections, and top patterns by volume.

To disable auto-open, set openBrowser: false in $TENX_CONFIG/run/output/event/dev/config.yaml. The URL is always printed to console.

For privacy, set localOnly: true. All processing stays on your machine, no data sent externally.

Step 8: Analyze Output (Optional)

The File Output generates the following files in $TENX_CONFIG/data/sample/output:

Statistical analysis of your log patterns: top event types by volume and bytes, error rates, and level distributions.

See analysis tables for sample output.

Your original events losslessly compact (typically 50-70% volume reduction).

See reduction ratios for sample output.

Unique TenXTemplate patterns discovered in your logs, enabling lossless volume reduction.

See template analysis for sample output.

Perfect reconstruction of compact events, validating lossless expanding.

To validate lossless encoding:

# Clear input and copy encoded output
rm -f $TENX_CONFIG/data/sample/input/*.log
cp $TENX_CONFIG/data/sample/output/encoded.log $TENX_CONFIG/data/sample/input/

# Re-run to decode
tenx @apps/dev

# Compare with original, files match exactly
diff $TENX_CONFIG/data/sample/output/decoded.log /path/to/original.log
Step 9: Teardown

Nothing runs in the background. Uninstall removes only what was installed.

brew uninstall log10x

sudo rm -rf /opt/tenx-edge /etc/tenx /etc/profile.d/tenx-edge.sh

If installed via DEB: sudo apt-get remove tenx-edge If installed via RPM: sudo yum remove tenx-edge

Uninstall from Settings > Apps > Installed apps.

docker rmi log10x/pipeline-10x:latest

Config Files

To configure the Dev app, Edit these files:

Main Config

Main Config

The main config file loads the Dev app's required modules.

Uncomment selected modules at: dev/config.yaml.

Edit Online

Edit Dev Config Locally 

#
# 🔟❎ dev app main config

# The dev app locally tests, and validates structured event processing.

# To learn more see https://doc.log10x.com/apps/dev

# ============================ Bootstrap Runtime ==============================

tenx: run

runtimeName: $=TenXEnv.get("TENX_RUNTIME_NAME", "myDev")

# ============================ Load App Modules ===============================

# Uncomment and edit selected config.yaml files (e.g., run/input/file/config.yaml)

include:

# ------------------------------ App settings ---------------------------------

  # Load general app settings:
  - dev

  # Load gitops settings for pulling additional assets from Github.
  # Edit the gitops/config.yaml file with specific config
  #
  # For additional info see - https://doc.log10x.com/config/github
  - gitops

# ------------------------------ Open Inputs ----------------------------------

  # read log/trace events from inputs to transform into well-defined TenXObjects:

  - run/input/file       # https://doc.log10x.com/run/input/file
  # - run/input/stdin    # https://doc.log10x.com/run/input/stdin
  # - run/input/forward  # https://doc.log10x.com/run/input/forward

# ----------------------------- Analyzer Inputs --------------------------------

  # Uncomment to read events from a remote log analyzer for reporting/analysis:

  # - run/input/analyzer/splunk           # https://doc.log10x.com/run/input/analyzer/splunk
  # - run/input/analyzer/elastic          # https://doc.log10x.com/run/input/analyzer/elastic
  # - run/input/analyzer/logzio           # https://doc.log10x.com/run/input/analyzer/logzio
  # - run/input/analyzer/coralogix        # https://doc.log10x.com/run/input/analyzer/coralogix
  # - run/input/analyzer/cloudwatchLogs   # https://doc.log10x.com/run/input/analyzer/cloudwatchLogs
  # - run/input/analyzer/datadogLogs      # https://doc.log10x.com/run/input/analyzer/datadogLogs
  # - run/input/analyzer/s3Logs           # https://doc.log10x.com/run/input/analyzer/s3Logs

# --------------------------- Enrich TenXObjects ------------------------------

  # Uncomment and edit config files to enrich TenXObjects with calculated fields:

  - run/initialize/inputField   # https://doc.log10x.com/run/initialize/#inputfield
  - run/initialize/level        # https://doc.log10x.com/run/initialize/level
  - run/initialize/group        # https://doc.log10x.com/run/initialize/group
  - run/initialize/message      # https://doc.log10x.com/run/initialize/message
  - run/initialize/httpCode     # https://doc.log10x.com/run/initialize/httpCode
  - run/initialize/lookup       # https://doc.log10x.com/run/initialize/lookup
  - run/initialize/custom       # https://doc.log10x.com/run/transform/script/object
  - run/initialize/k8s          # https://doc.log10x.com/run/initialize/k8s
  # - run/initialize/geoIP        # https://doc.log10x.com/run/initialize/geoIP

# --------------------------- Output TenXObjects -------------------------------

  # Uncomment and edit to write TenXObjects summaries to output(s):

  - run/output/event/file          # https://doc.log10x.com/run/output/event/file
  - run/output/event/dev           # https://doc.log10x.com/run/output/event/dev
  # - run/output/event/stdout      # https://doc.log10x.com/run/output/event/stdout
  # - run/output/event/process     # https://doc.log10x.com/run/output/event/process
  # - run/output/event/http        # https://doc.log10x.com/run/output/event/http
  # - run/output/event/unix        # https://doc.log10x.com/run/output/event/unix
  # - run/output/event/forward     # https://doc.log10x.com/run/output/event/forward

# -------------------------- Metric Outputs ----------------------------

  #  Uncomment and edit to publish aggregated TenXSummary instances to metric outputs:

  # - run/output/metric/log10x                     # https://doc.log10x.com/run/output/metric/log10x
  # - run/output/metric/prometheus/scrape          # https://doc.log10x.com/run/output/metric/prometheus/scrape
  # - run/output/metric/prometheus/push-gateway    # https://doc.log10x.com/run/output/metric/prometheus/push-gateway
  # - run/output/metric/prometheus/remote-write    # https://doc.log10x.com/run/output/metric/prometheus/remote-write
  # - run/output/metric/cloudwatch                 # https://doc.log10x.com/run/output/metric/cloudwatch
  # - run/output/metric/datadog                    # https://doc.log10x.com/run/output/metric/datadog
  # - run/output/metric/elastic                    # https://doc.log10x.com/run/output/metric/elastic
  # - run/output/metric/signalFx                   # https://doc.log10x.com/run/output/metric/signalFx
  # - run/output/metric/log                        # https://doc.log10x.com/run/output/metric/log
Inputs

Inputs

Activate Event inputs to read events from local/remote sources to transform into TenXObjects.

file

Configure File inputs to read events from an input file/glob path.

Below is the default configuration from: file/config.yaml.

Edit Online

Edit File input Config Locally 

# 🔟❎ 'run' File input configuration

# Configure file/glob event inputs. To learn more https://doc.log10x.com/run/input/file

# NOTE: input files are read once and are not tailed. To continuously tail files
# use 10x in conjunction with any of the supported file forwarders in:
# https://doc.log10x.com/run/input/forwarder (e.g., Fluentd/Bit, Logstash, Filebeat)

# Set the 10x pipeline to 'run'
tenx: run

# =============================== Dependencies ================================

include: run/modules/input/file

# =============================== File Options ================================

# Multiple file inputs can be defined below
inputFile:

  # ---------------------------- General Options ----------------------------

    # 'name' sets a unique logical name across all pipeline inputs
  - name: fileInput

  # ---------------------------- File Options ---------------------------------

    # 'path' specifies an absolute/canonical glob file path.
    path: $=path("data/sample/input/[!{.}]*.log")

    # ---------------------------- Parsing Options ----------------------------

    # 'extractors' defines a list of JSON/regex extractor names
    #  used to capture and redact event values from 'path' to transform 
    #  into TenXObjects. To learn more see: https://doc.log10x.com/run/input/extract         
    extractors: [  
    ]

    # 'sourcePattern' defines a regex pattern that captures a 'source' value
    #  for each event read from a file to transform into an TenXObject.
    #  This value will be used to sequence all events sharing this 'source' value correctly 
    #  so they are not mixed with events originating from different locations.
    sourcePattern: null  

    # 'sourceFields'  defines a list of JSON fields to extract from matching capture groups of the inputFileSourcePattern
    # or the entire event if not set as the 'source' value for each event read from this input. 
    sourceFields: [
    ]

    # --------------------------- Ancillary Options ---------------------------

    # 'printProgress' controls whether a progress gage is printed to the console    
    printProgress: true

stdin

Configure the stdin device input to read events from stdin.

Below is the default configuration from: stdin/config.yaml.

Edit Online

Edit stdin device input Config Locally 

# 🔟❎ 'run' Stdin input configuration

# Configure a stdin event input. To learn more see https://doc.log10x.com/run/input/stdin/

# Set the 10x pipeline to 'run'
tenx: run

# =============================== Dependencies ================================

include: run/modules/input/stdin

# =============================== Stdin Options ===============================

# NOTE: a pipeline should configure no more than a single stdin input

stdin:

  # ---------------------------- Parsing Options ----------------------------

    # 'extractors' defines a list of JSON/regex extractor names
    #  used to capture and redact event values from 'path' to transform 
    #  into TenXObjects. To learn more see: https://doc.log10x.com/run/input/extract         
  extractors: [
  ]

  # 'sourcePattern' defines a regex pattern that captures a 'source' value
  #  for each event read from the file and transformed into an TenXObject.
  #  This value will be used to sequence all events sharing this 'source' value correctly 
  #  so they are not mixed with events originating from different locations.    
  sourcePattern: null

forward

Configure the Forward protocol input to read events via the Fluent Forward protocol on a Unix domain socket or TCP port.

Below is the default configuration from: forward/config.yaml.

Edit Online

Edit Forward protocol input Config Locally 

# 🔟❎ 'run' Forward protocol input configuration
#
# Configure an input that receives events via the Fluent Forward protocol
# on a Unix domain socket. To learn more see https://doc.log10x.com/run/input/forward/

# Set the 10x pipeline to 'run'
tenx: run

# =============================== Dependencies ================================

include: run/modules/input/forward

# ======================== Forward Protocol Options ============================

forward:

  # ----------------------------- Input Options -----------------------------

  # 'inputPort' specifies the TCP port to listen on for Forward protocol messages
  inputPort: $=TenXEnv.get("TENX_FORWARD_INPUT_SOCKET", "24224")

  # 'inputPath' specifies a Unix domain socket path to listen on instead of TCP.
  #  When set, takes precedence over 'inputPort'.
  inputPath: $=TenXEnv.get("TENX_FORWARD_INPUT_ADDRESS", "")

  # ---------------------------- Parsing Options ----------------------------

  # 'extractors' defines a list of JSON/regex extractor names
  #  used to capture and redact event values from 'path' to transform
  #  into TenXObjects. To learn more see: https://doc.log10x.com/run/input/extract
  extractors: [
  ]

  # 'sourcePattern' defines a regex pattern that captures a 'source' value
  #  for each event received via the Forward protocol and transformed into
  #  a TenXObject. The Forward tag is injected as a "tag" field in each
  #  record, so the default pattern extracts the source from the tag.
  sourcePattern: null
Analyzer Inputs

Analyzer Inputs

Activate Log analyzer inputs to read events from log analyzers via REST.

splunk

Configure Splunk inputs to read events from a Splunk Cloud/on-premises deployment.

Below is the default configuration from: splunk/config.yaml (* Required Fields).

Edit Online

Edit Splunk input Config Locally 

# 🔟❎ 'run' Splunk input configuration

# Configure a Splunk event input
# To learn more see https://doc.log10x.com/run/input/analyzer/splunk/

# Set the 10x pipeline to 'run'
tenx: run

# =============================== Dependencies ================================

include: run/modules/input/analyzer/splunk

# =============================== Splunk Options ==============================

# Multiple Splunk inputs can be defined below

splunk:

    # ---------------------------- General Options ----------------------------

    # 'name' sets a unique logical name across all pipeline inputs
  - name: Splunk
    # Disabled by default - configure host/port to enable
    enabled: false

    # --------------------------- Connection Options --------------------------

    # 'host' and 'port' set the Splunk host address to connect to (e.g., '<deployment-name>.splunkcloud.com')
    host: null    # (❗ REQUIRED)
    port: null    # (Not mandatory if the host already encapsulates it)
    protocol: "https"

    # 'username' and 'password' used to authenticate against the Splunk deployment
    #  To learn more see https://docs.splunk.com/Documentation/Splunk/latest/RESTUM/RESTusing#Authentication_and_authorization
    username: $=TenXEnv.get("SPLUNK_USERNAME") # (❗ EnvVar REQUIRED)
    password: $=TenXEnv.get("SPLUNK_PASSWORD") # (❗ EnvVar REQUIRED)

    # ----------------------------- Query Options -----------------------------

    # 'pageSize' sets the number of events to retrieve with each result page
    #  Performance: Increase to 1000-2000 for high-volume environments
    pageSize: 500

    # 'query' sets the Splunk search query to execute for this job
    query: search *

    # --------------------------- Backpressure Options -----------------------

    # 'queryInterval' sets the interval between queries to the remote API
    #  Performance: Increase for rate-limited APIs; decrease for real-time needs
    queryInterval: $=parseDuration("2s")

    # 'totalDuration' sets the max duration to try reading from the the remote input
    #  Performance: Match to your job scheduling interval
    totalDuration: $=parseDuration("5min")

    # 'totalBytesLimit' sets the max total bytes to read from the remote input
    #  Performance: Increase for longer analysis windows (e.g., 200MB for 10min)
    totalBytesLimit: $=parseBytes("50MB")

    # 'totalEventsLimit' sets the max number of events to read the remote input
    #  Performance: Adjust based on memory capacity; each event consumes memory
    totalEventsLimit: 10000

    # --------------------------- Ancillary Options ---------------------------

    # 'printProgress' controls whether to print progress gage to the console
    #  This option helps debug and test the input
    printProgress: $=!TenXEnv.get("quiet")

elastic

Configure ElasticSearch inputs to read events from an ElasticSearch hosted/on-premises cluster.

Below is the default configuration from: elastic/config.yaml (* Required Fields).

Edit Online

Edit ElasticSearch input Config Locally 

# 🔟❎ 'run' Elastic input configuration

# Configure an Elastic event input
# To learn more see https://doc.log10x.com/run/input/analyzer/elasticsearch/

# Set the 10x pipeline to 'run'
tenx: run

# =============================== Dependencies ================================

include: run/modules/input/analyzer/elasticsearch

# =============================== Elastic Options ==============================

# Multiple Elastic inputs can be defined below:
elasticSearch:

    # 'name' sets a unique logical name across all pipeline inputs
  - name: ElasticSearch

    # --------------------------- Connection Options --------------------------

    # 'host' and 'port' set the ElasticSearch API endpoint to connect to
    #  For hosted Elastic (e.g. my-project-ec5189.es.us-central1.gcp.elastic.cloud).
    #  See https://cloud.elastic.co/deployments and https://i.stack.imgur.com/jXjLJ.png
    host: null # (❗ REQUIRED)
    port: "" # (not mandatory if the host already encapsulates it)

    # 'protocol' sets the connection protocol ('https' or 'http' for local testing)
    protocol: https

    # 'token' used to authenticate with. To learn more see:
    # https://www.elastic.co/guide/en/cloud-enterprise/current/ece-restful-api-authentication.html
    token: $=TenXEnv.get("ELASTIC_TOKEN") # (❗ REQUIRED)

    # ----------------------------- Query Options -----------------------------

    # 'targets' lists data streams, indices, and aliases to search.
    #  To search all data streams and indices, use '*' or '_all'.
    targets:
      - _all

    # 'startTime' and 'endTime' set the time frame of the query.
    startTime: now-1h
    endTime: now

    # --------------------------- Backpressure Options -----------------------

  # 'queryInterval' sets the interval between queries to the remote API
    queryInterval: $=parseDuration("2s")

    # 'totalDuration' sets the max duration to try reading from the the remote input
    totalDuration: $=parseDuration("5min")

    # 'totalBytesLimit' sets the max total bytes to read from the remote input
    totalBytesLimit: $=parseBytes("50MB")

    # 'totalEventsLimit' sets the max number of events to read the remote input
    totalEventsLimit: 10000

    # --------------------------- Ancillary Options ---------------------------

    # 'printProgress' controls whether to print a progress gage to the console
    #  This option helps debug and test the input
    printProgress: $=!TenXEnv.get("quiet")

    # --------------------------- Advanced Options ---------------------------

    # 'querySize' sets the number of documents retrieved with each results page.
    querySize: 500

    # 'query' sets the elastic query to request paginated documents from the  '/_search' REST endpoint.
    #  The default query (defined in 'input/analyzer/elasticsearch/query.txt')
    #  retrieves paginated values sorted in desc order by @timestamp
    query: null

logzio

Configure ElasticSearch inputs to read events from an ElasticSearch hosted/on-premises cluster.

Below is the default configuration from: logzio/config.yaml (* Required Fields).

Edit Online

Edit ElasticSearch input Config Locally 

# 🔟❎ 'run' Logz.io ElasticSearch input configuration

# Configures a Logz.io ElasticSearch event input
# To learn more see https://doc.log10x.com/run/input/analyzer/elastic/

# Set the 10x pipeline to 'run'
tenx: run

# =============================== Dependencies ================================

include: run/modules/input/analyzer/elasticsearch

# =============================== Logzio Options ==============================

# Multiple Logz.io inputs can be defined below:
elasticSearch:

    # 'name' sets a unique logical name across all pipeline inputs
  - name: logzioElasticSearch

    # --------------------------- Connection Options --------------------------

    # 'token' must be set to the Logz.io API token.
    #  See: https://docs.logz.io/user-guide/tokens/api-tokens.html
    token: null # (❗ REQUIRED)

    # 'host' and 'port' set the Logz.io service address to connect to
    host: api.logz.io
    port: 443

    # ----------------------------- Query Options -----------------------------

    # 'targets' lists data streams, indices, and aliases to search.
    #  To search all data streams and indices, use '*' or '_all'.
    targets:
      - v1

    # 'startTime' and 'endTime' set the time frame of the query.
    startTime: now-5m
    endTime: now

    # --------------------------- Backpressure Options -----------------------

    # 'queryInterval' sets the interval between queries to the remote API
    queryInterval: $=parseDuration("2s")

    # 'totalDuration' sets the max duration to try reading from the the remote input 
    totalDuration: $=parseDuration("5min")

    # 'totalBytesLimit' sets the max total bytes to read from the remote input
    totalBytesLimit: $=parseBytes("50MB")

    # 'totalEventsLimit' sets the max number of events to read the remote input
    totalEventsLimit: 10000

    # --------------------------- Ancillary Options ---------------------------

    # 'printProgress' controls whether to print a progress gage to the console
    #  This option helps debug and test the input
    printProgress: $=!TenXEnv.get("quiet")

    # --------------------------- Advanced Options ---------------------------

    # 'querySize' sets the number of documents retrieved with each results page.
    querySize: 500

    # 'query' sets the elastic query to request paginated documents from the  '/_search' REST endpoint.
    #  The default query (defined in 'input/analyzer/elasticsearch/query.txt')
    #  retrieves paginated values sorted in desc order by @timestamp
    query: null

    # 'endpoint' sets the REST API endpoint to query
    endpoint: search

    # 'messageField' defines the name of the field to extract from the retrieved
    # documents containing the log message.
    messageField: message

coralogix

Configure ElasticSearch inputs to read events from an ElasticSearch hosted/on-premises cluster.

Below is the default configuration from: coralogix/config.yaml (* Required Fields).

Edit Online

Edit ElasticSearch input Config Locally 

# 🔟❎ 'run' Coralogix ElasticSearch input configuration

# This configuration defines a Coralogix ElasticSearch input from
# which events are read and transformed into TenXObjects for aggregation and reporting.

# Set the 10x pipeline to 'run'
tenx: run

# =============================== Dependencies ================================

include: run/modules/input/analyzer/elasticsearch

# ============================== Coralogix Options ============================

# Multiple Coralogix inputs can be defined below
elasticSearch:

    # 'name' sets a unique logical name across all pipeline inputs
  - name: coralogixElasticSearch

    # ------------------------- Connection Options ----------------------------

    # 'token' must be set to the Coralogix send data API key
    #  To learn more see: https://coralogix.com/docs/user-guides/account-management/api-keys/send-your-data-api-key/
    token: null # (❗ REQUIRED)

    # 'authorizationKeyName' must be set to match Coralogix auth type
    #  To learn more see: https://coralogix.com/docs/user-guides/visualizations/hosted-opensearch-view/opensearch-api/
    authorizationKeyName: Bearer

    # 'host' and 'port' set the Coralogix service address to connect to
    host: api.coralogix.us/data/os-api

    # --------------------------- Query Options -------------------------------

    # 'targets' lists data streams, indices, and aliases to search.
    #  To search all data streams and indices, use '*' or '_all.'
    targets:
      - '*'

    # 'startTime' and 'endTime' set the time frame of the query.
    startTime: now-5m
    endTime: now

    # --------------------------- Backpressure Options -----------------------

    # 'queryInterval' sets the interval between queries to the remote API
    queryInterval: $=parseDuration("2s")

    # 'totalDuration' sets the max duration to try reading from the the remote input 
    totalDuration: $=parseDuration("5min")

    # 'totalBytesLimit' sets the max total bytes to read from the remote input
    totalBytesLimit: $=parseBytes("50MB")

    # 'totalEventsLimit' sets the max number of events to read the remote input
    totalEventsLimit: 10000

    # --------------------------- Ancillary Options ---------------------------

    # 'printProgress' controls whether to print a progress gage to the console
    #  This option helps debug and test the input
    printProgress: $=!TenXEnv.get("quiet")

    # --------------------------- Advanced Options ----------------------------

    # 'querySize' sets the number of documents retrieved with each results page.
    querySize: 500

    # 'query' sets the elastic query to request paginated documents from the  '/_search' REST endpoint.
    #  The default query (defined in 'input/analyzer/elasticsearch/query.txt')
    #  retrieves paginated values sorted in desc order by @timestamp
    query: null

    # 'timestampField' specifies the name of the field to get the event timestamp from.
    timestampField: coralogix.timestamp

    # 'messageField' defines the name of the field to extract from the retrieved
    #  documents containing the log message.
    messageField: log

cloudwatchLogs

Configure AWS Cloudwatch Logs inputs to read events from AWS CloudWatch Logs.

Below is the default configuration from: cloudwatchLogs/config.yaml (* Required Fields).

Edit Online

Edit AWS Cloudwatch Logs input Config Locally 

# 🔟❎ 'run' AWS CloudWatch Logs input configuration

# Configure an AWS CloudWatch Logs event input
# To learn more see https://doc.log10x.com/run/input/analyzer/cloudwatchLogs/

# Set the 10x pipeline to 'run'
tenx: run

# =============================== Dependencies ================================

include: run/modules/input/analyzer/cloudwatchLogs

# ============================ CloudWatch Options =============================

# Multiple CloudWatch inputs can be defined below
cloudwatchLogs:

    # 'name' sets a unique logical name across all pipeline inputs
  - name: CloudwatchLogs

    # --------------------------- Connection Options --------------------------

    # 'awsAccessKeyID' sets the AWS access key (e.g. 'AKIAIOSFODNN7EXAMPLE')
    #  To learn more see: https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html
    awsAccessKeyID: $=TenXEnv.get("AWS_ACCESS_KEY_ID") # (❗ EnvVar REQUIRED)

    # 'awsSecretKey' sets the AWS access secret key  (e.g. 'wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY')
    #  To learn more see: https://docs.aws.amazon.com/general/latest/gr/aws-sec-cred-types.html
    awsSecretKey: $=TenXEnv.get("AWS_SECRET_ACCESS_KEY") # (❗ EnvVar REQUIRED)

    # 'groupName' sets the log group name (e.g., 'my-log-group').
    #  To learn more see: https://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_GetLogEvents.html#CWL-GetLogEvents-request-logGroupName
    groupName: "" # (❗ REQUIRED)

    # 'streamName' sets the log stream name (e.g., 'my-log-stream').
    #  To learn more see: https://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_GetLogEvents.html#CWL-GetLogEvents-request-logStreamName
    streamName: "" # (❗ REQUIRED)

    # 'awsRegion' sets the region hosting the CloudWatch Logs stream
    #  To learn more see: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html
    awsRegion: $=TenXEnv.get("AWS_DEFAULT_REGION", "us-east-1")

    # ----------------------------- Query Options -----------------------------

    # 'limit' sets the number of documents retrieved with each results page.
    #  To learn more see: https://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_GetLogEvents.html#CWL-GetLogEvents-request-limit
    limit: 500

    # 'startFromHead' controls the direction of the search (old <-> new events).
    #  To learn more see: https://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_GetLogEvents.html#CWL-GetLogEvents-request-startFromHead
    startFromHead: true

    # 'startTime' sets the start of the time range, expressed as the number of milliseconds after Jan 1, 1970 00:00:00 UTC.
    #  The search includes events with a timestamp equal to or later than this value.
    #  The search excludes events with a timestamp earlier than this value. 
    #  To learn more see: https://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_GetLogEvents.html#CWL-GetLogEvents-request-startTime
    startTime: $=now("-240h")

    # 'endTime' sets the end of the time range, expressed as the number of milliseconds after Jan 1, 1970 00:00:00 UTC.
    #  The search excludes events with a timestamp equal to or later than this value.
    #  To learn more see: https://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_GetLogEvents.html#CWL-GetLogEvents-request-endTime
    endTime: $=now()

    # --------------------------- Backpressure Options -----------------------

    # 'queryInterval' sets the interval between queries to the remote API
    queryInterval: $=parseDuration("2s")

    # 'totalDuration' sets the max duration to try reading from the the remote input 
    totalDuration: $=parseDuration("5min")

    # 'totalBytesLimit' sets the max total bytes to read from the remote input
    totalBytesLimit: $=parseBytes("50MB")

    # 'totalEventsLimit' sets the max number of events to read the remote input
    totalEventsLimit: 10000

    # --------------------------- Ancillary Options ---------------------------

    # 'printProgress' controls whether to print a gage to the console
    #  This option helps debug and test the input
    printProgress: $=!TenXEnv.get("quiet")

    # 'messageField' sets the JSON field to extract from CloudWatch log messages
    #  For K8s/Docker logs with 10x encoding, use 'log' to extract the encoded log field
    messageField: log

datadogLogs

Configure Datadog Logs inputs to read events from Datadog Logs.

Below is the default configuration from: datadogLogs/config.yaml (* Required Fields).

Edit Online

Edit Datadog Logs input Config Locally 

# 🔟❎ 'run' Datadog Logs input configuration

# Configure a Datadog Logs event input
# To learn more see https://doc.log10x.com/run/input/analyzer/datadogLogs/

# Set the 10x pipeline to 'run'
tenx: run

# =============================== Dependencies ================================

include: run/modules/input/analyzer/datadogLogs

# ============================== Datadog Inputs ===============================

# Multiple Datadog inputs can be defined below
datadogLogs:

    # 'name' sets a unique logical name across all pipeline inputs
  - name: DatadogLogs

    # -------------------------- Connection Options ---------------------------

    # 'apiKey' and 'appKey' must be set to authenticate with the logs endpoint
    #  To learn more see: https://docs.datadoghq.com/account_management/api-app-keys/
    apiKey: $=TenXEnv.get("DD_API_KEY") # (❗ REQUIRED)

    appKey: $=TenXEnv.get("DD_APP_KEY") # (❗ REQUIRED)

    # 'host' points to the target Datadog site
    #  To learn more see: https://docs.datadoghq.com/getting_started/site/
    host: us5.datadoghq.com

    # ----------------------------- Query Options -----------------------------

    # 'query' sets the Datadog query used to request paginated documents.
    #  The default query (defined in 'run/modules/input/analyzer/datadog/query.txt') retrieves
    #  values sorted in desc order by @timestamp, starting from the value of 'searchFrom'
    query: null

    # 'searchFrom sets the initial value used to request the first page of results.
    searchFrom: now-1000h

    # 'querySize' sets the number of documents to retrieve with each paginated request
    querySize: 500

    # 'indexes' sets the lists of data indices to search in
    # If not provided, will use a default of '*'
    indexes: null

    # --------------------------- Backpressure Options -----------------------

    # 'queryInterval' sets the interval between queries to the remote API
    queryInterval: $=parseDuration("5s")

    # 'totalDuration' sets the max duration to try reading from the the remote input 
    totalDuration: $=parseDuration("5min")

    # 'totalBytesLimit' sets the max total bytes to read from the remote input
    totalBytesLimit: $=parseBytes("50MB")

    # 'totalEventsLimit' sets the max number of events to read the remote input
    totalEventsLimit: 10000

    # --------------------------- Ancillary Options ---------------------------

    # 'printProgress' controls whether to print a progress gage to the console
    #  This option helps test the input
    printProgress: $=!TenXEnv.get("quiet")
Initializers

Initializers

Activate an Object initializer to enrich TenXObjects with additional context.

message

Configure the Symbol Message Calculator to enrich TenXObjects with logical message symbol sequence and origin values.

Below is the default configuration from: message/config.yaml.

Edit Online

Edit Symbol Message Calculator Config Locally 

# 🔟❎ 'run' symbol lookup configuration

# Configure a symbol origin lookup to enrich TenXObjects
# To learn more see https://doc.log10x.com/run/initialize/message/

# Set the 10x pipeline to 'run'
tenx: run

# =============================== Dependencies ================================

include: run/modules/initialize/message

# ============================== Symbol Options ===============================

symbol:

  # 'types' specifies the types of symbols to search for; if the first type does not yield a result, the next one is tried etc.
  #  Supported values (case insensitive), PACKAGE, CLASS, METHOD, LOG, ENUM, CONST, TEXT, EXEC, ANY
  #  To learn more see https://doc.log10x.com/run/transform/symbol/#contexts
  contexts: log,exec

  # 'messageField' specifies the field name to assign the source/binary origin file name to target TenXObjects
  messageField: message_pattern

  # 'messageHashField' specifies the field name to assign a stable, URL-safe pattern hash (xxHash64 of the messageField value, base64url, 11 chars). Safe as a SIEM query term or forwarder filter key. Set empty to disable.
  messageHashField: tenx_hash

  # 'maxLen' specifies the max char len of the output messageField
  maxLen: 120

level

Configure the Level Classifier to classify TenXObjects with a severity level.

Below is the default configuration from: level/config.yaml.

Edit Online

Edit Level Classifier Config Locally 

# 🔟❎ 'run' level classifier configuration

# Configuration level classifier to enrich TenXObjects
# To learn more see https://doc.log10x.com/run/initialize/level/

# Set the 10x pipeline to 'run'
tenx: run

# =============================== Dependencies ================================

include: run/modules/initialize/level

# =============================== Level Options ===============================

level:

  # 'field' specify the field name to assign with the inferred severity level
  field: severity_level

  # 'terms' specify severity level classification, start-of-line matching
  #  Used to identify the severity level (TRACE, DEBUG, INFO, WARN, ERROR, CRITICAL) of a log line, typically marking
  #  the start of a log event (head). Sorted by likelihood and uniqueness. Used in a Trie for O(m) matching.
  terms:
    - 'trace=TRACE'          # Common in JavaScript (Winston) or custom logs; e.g., "trace Variable x = 5"
    - 'Trace=TRACE'          # Mixed case; e.g., "Trace Variable x = 5"
    - 'TRACE=TRACE'          # Uppercase, less common; e.g., "TRACE Message"
    - 'debug=DEBUG'          # Common in JavaScript (Winston) or custom logs; e.g., "debug Entering method foo"
    - 'Debug=DEBUG'          # Mixed case; e.g., "Debug Entering method foo"
    - 'DEBUG=DEBUG'          # Uppercase; e.g., "DEBUG Message"
    - 'DBG=DEBUG'            # Abbreviation, less common; e.g., "DBG Entering method foo"
    - 'info=INFO'            # Common in JavaScript (Winston) or custom logs; e.g., "info Application started"
    - 'Info=INFO'            # Mixed case; e.g., "Info Application started"
    - 'INFO=INFO'            # Uppercase; e.g., "INFO Message"
    - 'notice=INFO'          # Common in custom logs or lowercase syslog; e.g., "notice Configuration updated"
    - 'Notice=INFO'          # Mixed case; e.g., "Notice Configuration updated"
    - 'NOTICE=INFO'          # Uppercase, syslog; e.g., "NOTICE Message"
    - 'warn=WARN'            # Common in JavaScript (Winston) or custom logs; e.g., "warn Low disk space"
    - 'Warn=WARN'            # Mixed case; e.g., "Warn Low disk space"
    - 'WARN=WARN'            # Uppercase; e.g., "WARN Message"
    - 'warning=WARN'         # Common in custom logs or verbose frameworks; e.g., "warning Low disk space"
    - 'Warning=WARN'         # Mixed case; e.g., "Warning Low disk space"
    - 'WARNING=WARN'         # Uppercase; e.g., "WARNING Message"
    - 'error=ERROR'          # Common in JavaScript (Winston) or custom logs; e.g., "error Failed to connect"
    - 'Error=ERROR'          # Mixed case; e.g., "Error Failed to connect"
    - 'ERROR=ERROR'          # Uppercase; e.g., "ERROR Message"
    - 'err=ERROR'            # Lowercase abbreviation; e.g., "err Failed to connect"
    - 'ERR=ERROR'            # Uppercase abbreviation; e.g., "ERR Message"
    - 'TypeError=ERROR'      # JavaScript, stack trace; e.g., "TypeError: undefined is not a function"
    - 'ReferenceError=ERROR' # JavaScript, stack trace; e.g., "ReferenceError: x is not defined"
    - 'ValueError=ERROR'     # Python, stack trace; e.g., "ValueError: invalid literal"
    - 'PHP_Warning=ERROR'    # PHP, warning message; e.g., "PHP Warning: Undefined variable"
    - 'fatal=CRITICAL'       # Common in custom logs; e.g., "fatal System crash"
    - 'Fatal=CRITICAL'       # Mixed case; e.g., "Fatal System crash"
    - 'FATAL=CRITICAL'       # Uppercase; e.g., "FATAL Message"
    - 'critical=CRITICAL'    # Common in custom logs; e.g., "critical System crash"
    - 'Critical=CRITICAL'    # Mixed case; e.g., "Critical System crash"
    - 'CRITICAL=CRITICAL'    # Uppercase; e.g., "CRITICAL Message"
    - 'crit=CRITICAL'        # Lowercase abbreviation, syslog; e.g., "crit System crash"
    - 'CRIT=CRITICAL'        # Uppercase abbreviation, syslog; e.g., "CRIT Message"
    - 'alert=CRITICAL'       # Common in custom logs or syslog; e.g., "alert High CPU usage"
    - 'Alert=CRITICAL'       # Mixed case; e.g., "Alert High CPU usage"
    - 'ALERT=CRITICAL'       # Uppercase, syslog; e.g., "ALERT Message"
    - 'emerg=CRITICAL'       # Common in custom logs or lowercase syslog; e.g., "emerg System unusable"
    - 'Emerg=CRITICAL'       # Mixed case; e.g., "Emerg System unusable"
    - 'EMERG=CRITICAL'       # Uppercase, syslog; e.g., "EMERG Message"
    - 'panic=CRITICAL'       # Go panic, first line; e.g., "panic: runtime error: index out of range"
    - 'fatal_error=CRITICAL' # Go fatal error, first line; e.g., "fatal error: all goroutines are asleep"
    - 'thread_main_panicked_at=CRITICAL' # Rust panic, first line; e.g., "thread 'main' panicked at 'explicit panic'"
    - 'Segmentation_fault=CRITICAL' # C/C++ memory violation, first line; e.g., "Segmentation fault (core dumped)"
    - 'Aborted=CRITICAL'     # C/C++ termination, first line; e.g., "Aborted (core dumped)"
    - 'Assertion_failed=CRITICAL' # C/C++ assertion failure, first line; e.g., "Assertion failed: x > 0"
    - 'PHP_Fatal_error=CRITICAL' # PHP fatal error, first line; e.g., "PHP Fatal error: Out of memory"
    - 'Traceback_most_recent_call_last=CRITICAL' # Python stack trace, first line; e.g., "Traceback (most recent call last)"
    - 'Stack_trace=CRITICAL' # Generic stack trace, first line; e.g., "Stack trace"
    - 'Error=CRITICAL'       # JavaScript, stack trace; e.g., "Error: Something went wrong"
    - 'Fatal_error=CRITICAL' # PHP, stack trace; e.g., "Fatal error: Uncaught Exception"
    - 'Unhandled_exception=CRITICAL' # C#/Dart, stack trace; e.g., "Unhandled exception: System.NullReferenceException"
    - 'Exception_in_thread=CRITICAL' # Java, stack trace; e.g., "Exception in thread 'main' java.lang.RuntimeException"
    - 'goroutine=CRITICAL'   # Go, stack trace; e.g., "goroutine 1 [running]:"
    - 'failed=ERROR'         # ERROR-level entries in application, system, or web server logs

  # 'timestampPatterns' patterns for LogEvent head classification, start-of-line matching
  #  Used to identify timestamp formats in log lines and infer their implied severity level based on the first character.
  #  Formats are typical of Kubernetes kube-apiserver logs with varying microsecond precision. The first character
  #  ('I', 'W', 'E', 'F') determines the severity: I=INFO, W=WARN, E=ERROR, F=CRITICAL. Sorted by precision.
  timestampPatterns:
    - "'I'MMdd HH:mm:ss.S=INFO"          # Used in Kubernetes kube-apiserver logs with INFO prefix, 1-digit microsecond precision
    - "'I'MMdd HH:mm:ss.SS=INFO"         # Used in Kubernetes kube-apiserver logs with INFO prefix, 2-digit microsecond precision
    - "'I'MMdd HH:mm:ss.SSS=INFO"        # Used in Kubernetes kube-apiserver logs with INFO prefix, 3-digit microsecond precision
    - "'I'MMdd HH:mm:ss.SSSS=INFO"       # Used in Kubernetes kube-apiserver logs with INFO prefix, 4-digit microsecond precision
    - "'I'MMdd HH:mm:ss.SSSSS=INFO"      # Used in Kubernetes kube-apiserver logs with INFO prefix, 5-digit microsecond precision
    - "'I'MMdd HH:mm:ss.SSSSSS=INFO"     # Used in Kubernetes kube-apiserver logs with INFO prefix, 6-digit microsecond precision
    - "'W'MMdd HH:mm:ss.S=WARN"          # Used in Kubernetes kube-apiserver logs with WARNING prefix, 1-digit microsecond precision
    - "'W'MMdd HH:mm:ss.SS=WARN"         # Used in Kubernetes kube-apiserver logs with WARNING prefix, 2-digit microsecond precision
    - "'W'MMdd HH:mm:ss.SSS=WARN"        # Used in Kubernetes kube-apiserver logs with WARNING prefix, 3-digit microsecond precision
    - "'W'MMdd HH:mm:ss.SSSS=WARN"       # Used in Kubernetes kube-apiserver logs with WARNING prefix, 4-digit microsecond precision
    - "'W'MMdd HH:mm:ss.SSSSS=WARN"      # Used in Kubernetes kube-apiserver logs with WARNING prefix, 5-digit microsecond precision
    - "'W'MMdd HH:mm:ss.SSSSSS=WARN"     # Used in Kubernetes kube-apiserver logs with WARNING prefix, 6-digit microsecond precision
    - "'E'MMdd HH:mm:ss.S=ERROR"         # Used in Kubernetes kube-apiserver logs with ERROR prefix, 1-digit microsecond precision
    - "'E'MMdd HH:mm:ss.SS=ERROR"        # Used in Kubernetes kube-apiserver logs with ERROR prefix, 2-digit microsecond precision
    - "'E'MMdd HH:mm:ss.SSS=ERROR"       # Used in Kubernetes kube-apiserver logs with ERROR prefix, 3-digit microsecond precision
    - "'E'MMdd HH:mm:ss.SSSS=ERROR"      # Used in Kubernetes kube-apiserver logs with ERROR prefix, 4-digit microsecond precision
    - "'E'MMdd HH:mm:ss.SSSSS=ERROR"     # Used in Kubernetes kube-apiserver logs with ERROR prefix, 5-digit microsecond precision
    - "'E'MMdd HH:mm:ss.SSSSSS=ERROR"    # Used in Kubernetes kube-apiserver logs with ERROR prefix, 6-digit microsecond precision
    - "'F'MMdd HH:mm:ss.S=CRITICAL"      # Used in Kubernetes kube-apiserver logs with FATAL prefix, 1-digit microsecond precision
    - "'F'MMdd HH:mm:ss.SS=CRITICAL"     # Used in Kubernetes kube-apiserver logs with FATAL prefix, 2-digit microsecond precision
    - "'F'MMdd HH:mm:ss.SSS=CRITICAL"    # Used in Kubernetes kube-apiserver logs with FATAL prefix, 3-digit microsecond precision
    - "'F'MMdd HH:mm:ss.SSSS=CRITICAL"   # Used in Kubernetes kube-apiserver logs with FATAL prefix, 4-digit microsecond precision
    - "'F'MMdd HH:mm:ss.SSSSS=CRITICAL"  # Used in Kubernetes kube-apiserver logs with FATAL prefix, 5-digit microsecond precision
    - "'F'MMdd HH:mm:ss.SSSSSS=CRITICAL" # Used in Kubernetes kube-apiserver logs with FATAL prefix, 6-digit microsecond precision

group

Configure the Group initializer to combine multi-line events into TenXObject group instances.

Below is the default configuration from: group/config.yaml.

Edit Online

Edit Group initializer Config Locally 

# 🔟❎ 'run' event grouping configuration

# Group  sequences of TenXObjects to filter, aggregate and output as a single logical unit.
# To learn more see https://doc.log10x.com/run/transform/group/

# Set the 10x pipeline to 'run'
tenx: run

# =============================== Dependencies ================================

include: run/modules/initialize/group

# =============================== Group Options ===============================


# 'indicators'  specifies a list of `value:state` pairs that determine if a log line's start marks a group head (`true`) or child (`false`).
group:

  # 'indicators' specifies a list of strings that, when matched at the start of a log line's text, designate it as a group head.
  # Unmatched lines default to false, indicating they are group children.
  indicators:
    - '{'                 # Indicates a pretty-printed JSON/object log body (e.g., payment '{ resource: {...} }'); the opening brace heads the multi-line object so it groups as one event. Matches the log body only, grouping reads inputField, not the wrapped envelope.
    - '192.'              # Indicates a private IP address, common in web server logs (e.g., "192.168.1.1 - - [...]")
    - '10.'               # Indicates a private IP range, often in Kubernetes or internal network logs (e.g., "10.244.0.125 - - [...]")
    - '172.'              # Indicates a private IP range, typical in enterprise network logs (e.g., "172.16.0.1 - - [...]")
    - '127.'              # Indicates localhost, a common web server log initiator (e.g., "127.0.0.1 - - [...]")
    - 'GET '              # Indicates an HTTP GET request, marking the start of a web transaction log (e.g., "GET /index.html HTTP/1.1")
    - 'POST '             # Indicates an HTTP POST request, marking the start of a web transaction log (e.g., "POST /api HTTP/1.1")
    - 'PUT '              # Indicates an HTTP PUT request, marking the start of a web transaction log (e.g., "PUT /resource HTTP/1.1")
    - 'DELETE '           # Indicates an HTTP DELETE request, marking the start of a web transaction log (e.g., "DELETE /resource HTTP/1.1")
    - 'HEAD '             # Indicates an HTTP HEAD request, marking the start of a web transaction log (e.g., "HEAD /index.html HTTP/1.1")
    - 'OPTIONS '          # Indicates an HTTP OPTIONS request, marking the start of a web transaction log (e.g., "OPTIONS /api HTTP/1.1")
    - 'HTTP/'             # Indicates an HTTP protocol version, marking the start of a web transaction log (e.g., "GET /index.html HTTP/1.1")
    - 'kernel:'           # Indicates a Linux kernel log entry, marking the start of a system event (e.g., "kernel: [0.123456] Device initialized")
    - 'sshd['             # Indicates an SSH daemon log entry, marking the start of a security event (e.g., "sshd[1234]: Accepted password ...")
    - 'systemd['          # Indicates a systemd service log entry, marking the start of a system service event (e.g., "systemd[1]: Started service ...")
    - 'cron['             # Indicates a cron daemon log entry, marking the start of a scheduled task event (e.g., "cron[1234]: Running job ...")
    - 'syslog:'           # Indicates a syslog message, marking the start of a system log event (e.g., "syslog: Message ...")
    - 'rsyslogd:'         # Indicates an rsyslog daemon log entry, marking the start of a logging system event (e.g., "rsyslogd: Log started ...")
    - 'auditd['           # Indicates an audit daemon log entry, marking the start of a security audit event (e.g., "auditd[1234]: Audit event ...")
    - 'daemon:'           # Indicates a syslog daemon facility log, marking the start of a system service event (e.g., "daemon: Service started ...")
    - 'user:'             # Indicates a syslog user facility log, marking the start of a user-related event (e.g., "user: User logged in ...")
    - 'local0:'           # Indicates a syslog local facility log, marking the start of a custom system event (e.g., "local0: Custom message ...")
    - 'local1:'           # Indicates a syslog local facility log, marking the start of a custom system event (e.g., "local1: Custom message ...")
    - 'level='            # Indicates a structured log key, marking the start of a key-value log entry (e.g., "level=info msg=Started")
    - 'msg='              # Indicates a structured log key, marking the start of a message log entry (e.g., "msg=Application started")
    - 'message='          # Indicates a structured log key, marking the start of a message log entry (e.g., "message=Application started")
    - 'event='            # Indicates a structured log key, marking the start of an event log entry (e.g., "event=Service startup")
    - 'thread='           # Indicates a structured log key, marking the start of a thread-specific log entry (e.g., "thread=main Processing ...")
    - 'Starting '         # Indicates the start of a process initiation log (e.g., "Starting server on port 8080")
    - 'Stopping '         # Indicates the start of a process termination log (e.g., "Stopping service ...")
    - 'Running '          # Indicates the start of a process status log (e.g., "Running task ...")
    - 'Listening '        # Indicates the start of a network service log (e.g., "Listening on port 8080 ...")
    - 'Connecting '       # Indicates the start of a connection attempt log (e.g., "Connecting to database ...")
    - 'Connected '        # Indicates the start of a successful connection log (e.g., "Connected to database ...")
    - 'Disconnected '     # Indicates the start of a disconnection log (e.g., "Disconnected from server ...")
    - 'Processing '       # Indicates the start of a task processing log (e.g., "Processing request ...")
    - 'Received '         # Indicates the start of a data reception log (e.g., "Received message ...")
    - 'Sent '             # Indicates the start of a data transmission log (e.g., "Sent response ...")
    - 'User '             # Indicates the start of a user action log (e.g., "User logged in ...")
    - 'Authentication '   # Indicates the start of an authentication log (e.g., "Authentication successful ...")
    - 'Authorized '       # Indicates the start of an authorization log (e.g., "Authorized user access ...")
    - 'Failed '           # Indicates the start of a failure log (e.g., "Failed login attempt ...")
    - 'kubelet:=true'     # Indicates a Kubernetes kubelet log entry, marking the start of a node event (e.g., "kubelet: Starting kubelet")
    - 'pod:=true'         # Indicates a Kubernetes pod log entry, marking the start of a pod event (e.g., "pod: Starting container")
    - 'container:=true'   # Indicates a Kubernetes container log entry, marking the start of a container event (e.g., "container: Started")
    - 'namespace:=true'   # Indicates a Kubernetes namespace log entry, marking the start of a namespace event (e.g., "namespace: Created")
    - 'Traceback '        # Indicates the start of a Python stack trace (e.g., "Traceback (most recent call last):")
    - 'File "'            # Indicates a Python stack trace line (e.g., "File "/script.py", line 10")
    - 'Trace:'            # Indicates a Node.js console trace (e.g., "Trace: Show me")
    - 'Error:'            # Indicates a Node.js error log (e.g., "Error: Something went wrong")
    - 'Warning:'          # Indicates a Node.js warning log (e.g., "Warning: Deprecated method")
    - 'Exception in thread ' # Indicates a Java exception header (e.g., "Exception in thread 'main'")
    - 'goroutine '        # Indicates a Go goroutine stack trace (e.g., "goroutine 1 [running]:")
    - 'panic:'            # Indicates a Go panic (e.g., "panic: runtime error")
    - 'thread '           # Indicates a Rust thread panic (e.g., "thread 'main' panicked at")
    - 'stack backtrace:'  # Indicates a Rust backtrace header (e.g., "stack backtrace:")
    - 'PHP Warning:'      # Indicates a PHP warning (e.g., "PHP Warning: Undefined variable")
    - 'PHP Fatal error:'  # Indicates a PHP fatal error (e.g., "PHP Fatal error: Out of memory")
    - 'in /'              # Indicates a PHP/Ruby file path in stack trace (e.g., "in /path/to/file.php:10")
    - 'Exception:'        # Indicates a C# exception (e.g., "Exception: Invalid operation")
    - 'terminate called'  # Indicates a C++ termination (e.g., "terminate called after throwing an instance of 'std::exception'")
    - 'syslog:'           # Indicates a Linux syslog message (e.g., "syslog: Message ...")
    - 'IN='               # Indicates an iptables firewall input log (e.g., "IN=eth0 OUT=")
    - 'OUT='              # Indicates an iptables firewall output log (e.g., "OUT=eth0 SRC=")
    - 'SRC='              # Indicates an iptables firewall source IP log (e.g., "SRC=192.168.1.1 DST=")
    - 'DST='              # Indicates an iptables firewall destination IP log (e.g., "DST=10.0.0.1 LEN=")
    - 'KafkaServer:'      # Indicates a Kafka server log (e.g., "KafkaServer: Starting Kafka server")
    - 'Redis:'            # Indicates a Redis server log (e.g., "Redis: Server initialized")

httpCode

Configure the HTTP Status Code Extractor to enrich TenXObjects with an extracted HTTP status code.

Below is the default configuration from: httpCode/config.yaml.

Edit Online

Edit HTTP Status Code Extractor Config Locally 

# 🔟❎ 'run' HTTP code classifier configuration

# The HTTP code classifier enriches TenXObjects with an extracted HTTP numeric response code (e.g., 404) field.

# Set the 10x pipeline to 'run'
tenx: run

# =============================== Dependencies ================================

include: run/modules/initialize/httpCode

# =============================== GeoIP Options ===============================

httpCode:

  # 'field' specifies the name of the field in which to assign an extracted HTTP status code, if found
  field: http_code

  # 'precludersBefore' specifies an array of characters or patterns that, when appearing immediately before a candidate HTTP status code in tokenized logs, indicate it is not a valid HTTP code.
  #  Used to preclude false positives like negative numbers ('-200'), floats ('.200'), or variables ('_200').
  #  Entries are single characters or short strings for O(1) lookup efficiency in the heuristic.
  precludersBefore: ['+', '$', '#', '@', '(', '[', '~', '-', '.', ':']

  # 'precludersAfter' specifies an array of characters or patterns that, when appearing immediately after a candidate HTTP status code in tokenized logs, indicate it is not a valid HTTP code.
  #  Used to preclude false positives like percentages ('200%'), floats ('200.'), or expressions ('200+').
  #  Entries are single characters or short strings for O(1) lookup efficiency in the heuristic.
  precludersAfter: ['%', ')', ']', '=', '!', '?', '+', '-', '.']

  # 'keywords' specifies an array of keywords indicating HTTP-related context in log events.
  #  Used to filter logs likely containing HTTP status codes, reducing false positives.
  #  Keywords cover HTTP methods, common log fields, and format-specific terms from sources like Apache, Nginx, HAProxy, Squid, IIS, Spring Boot.
  keywords:
    - HTTP/             # Indicates HTTP protocol version in request lines (e.g., "GET / HTTP/1.1").
    - GET               # HTTP method for retrieving resources.
    - POST              # HTTP method for submitting data.
    - PUT               # HTTP method for updating resources.
    - DELETE            # HTTP method for deleting resources.
    - HEAD              # HTTP method for retrieving headers only.
    - OPTIONS           # HTTP method for describing communication options.
    - TRACE             # HTTP method for diagnostic purposes.
    - PATCH             # HTTP method for partial updates.
    - request           # Common in log fields referring to HTTP requests (e.g., "requestMethod").
    - Request           # Title case variant for HTTP requests
    - response          # Common in log fields referring to HTTP responses (e.g., "responseSize").
    - Response          # Title case variant for HTTP responses
    - status            # Common key for HTTP status codes (e.g., "status":200).
    - Status            # Title case variant for HTTP status
    - code              # Shorthand for status code (e.g., "code":404).
    - Code              # Title case variant for status code
    - upstream_status   # Nginx-specific for backend status codes.
    - Upstream_status   # Title case variant for Nginx backend status
    - Upstream_Status   # Title case variant for Nginx backend status
    - http_status       # JSON key for HTTP status (e.g., "http_status":500).
    - Http_status       # Title case variant for JSON key for HTTP status
    - Http_Status       # Title case variant for JSON key for HTTP status
    - status_code       # Alternative key for status codes (e.g., "status_code":429).
    - statusCode        # Camel case variant for status codes (e.g., "statusCode":200)
    - Completed         # Spring Boot DispatcherServlet log indicator (e.g., "Completed 200 OK").
    - access            # Indicates access logs (e.g., Apache access log).
    - Access            # Title case variant for access logs
    - error             # Indicates error logs containing HTTP errors.
    - Error             # Title case variant for error logs
    - method            # Refers to HTTP request method (e.g., "method":"GET").
    - Method            # Title case variant for HTTP method
    - path              # Refers to request path or URI (e.g., "path":"/api").
    - Path              # Title case variant for request path
    - url               # Refers to request URL (e.g., "url":"http://example.com").
    - Url               # Title case variant for request URL
    - URL               # Title case variant for request URL
    - referrer          # HTTP referrer header field.
    - referrer          # Title case variant for referrer header field.
    - Referer           # Alternative spelling for HTTP referrer
    - Referer           # Title case variant for alternative spelling
    - user-agent        # HTTP user-agent header field.
    - User-agent        # Title case variant for user-agent
    - User-Agent        # Title case variant for user-agent
    - latency           # Common in logs for request/response time (e.g., HAProxy, ELB).
    - Latency           # Title case variant for latency
    - frontend          # HAProxy term for incoming connections.
    - Frontend          # Title case variant for HAProxy frontend
    - backend           # HAProxy term for upstream servers.
    - Backend           # Title case variant for HAProxy backend
    - TCP_MISS          # Squid hierarchy code prefix (e.g., "TCP_MISS/200").
    - W3SVC             # IIS log prefix (e.g., "W3SVC1").
    - DispatcherServlet # Spring Boot servlet class for HTTP handling.

  # 'strictKeywords' is a short, high-signal subset of keywords used as a
  #  structural guard during template initialization. After a candidate HTTP
  #  status variable token is found in the template via 'validValues', the
  #  initializer requires at least one of these strict markers to appear
  #  within a small token window (±5) around the candidate before the
  #  httpToken position is bound.
  #
  #  Without this check, any numeric variable that happens to hold a value
  #  in 100..599 on the first observed event gets wrongly bound as the HTTP
  #  code position, e.g. a kafka config limit `max.request.size = 200`, a
  #  JVM `-Dclk.tck=100`, or any framework metric that emits counts in the
  #  low hundreds. Subsequent events at the same variable position then
  #  carry arbitrary non-HTTP values (1, 2, 2000, ...) that get emitted as
  #  http_code garbage.
  #
  #  Entries here should be ONLY tokens that unambiguously mark an HTTP
  #  exchange. The loose list in 'keywords' above (which includes terms
  #  like `error`, `path`, `url`) is too broad for adjacency gating.
  strictKeywords:
    - HTTP/             # HTTP protocol version marker in request lines (e.g., "HTTP/1.1 200 OK").
    - status            # JSON/KV key for HTTP status (e.g., "status":200, status=500).
    - Status            # Title case variant.
    - statusCode        # CamelCase variant of status code key.
    - status_code       # Snake case variant.
    - http_status       # HTTP-prefixed status key.
    - Http_Status       # Title case variant.
    - Http_status       # Mixed case variant.
    - upstream_status   # Nginx backend status key.
    - Upstream_Status   # Title case variant.
    - code              # JSON/KV key for response code (e.g., "code":404).
    - Code              # Title case variant (also catches `statusCode`-style splits).
    - Completed         # Spring Boot DispatcherServlet log phrase (e.g., "Completed 200 OK").
    - DispatcherServlet # Spring Boot servlet class that always emits a status next.
    - TCP_MISS          # Squid hierarchy code prefix (always followed by "/<code>").
    - response          # HTTP response marker.
    - Response          # Title case variant.
    - GET               # HTTP method verbs, always appear in request-log entries with codes.
    - POST
    - PUT
    - DELETE
    - PATCH
    - HEAD
    - OPTIONS
    - TRACE

  # 'validValues' specifies an array of all standard HTTP status codes as defined by the IANA HTTP Status Code Registry and relevant RFCs (e.g., RFC 9110, RFC 6585, RFC 7231-7235, RFC 7538, RFC 7725, RFC 8297, RFC 9111).
  #  Used to validate potential HTTP status codes in log events, ensuring only recognized codes are extracted.
  #  Each entry includes the numeric code followed by a comment describing its meaning and purpose.
  validValues:
    - '200' # OK: Standard response for successful HTTP requests.
    - '304' # Not Modified: Resource has not been modified since last requested.
    - '404' # Not Found: Server cannot find the requested resource.
    - '301' # Moved Permanently: Resource has been moved permanently to a new URI.
    - '302' # Found: Resource temporarily located at a different URI.
    - '403' # Forbidden: Client does not have access rights to the content.
    - '400' # Bad Request: Server cannot process due to client error.
    - '500' # Internal Server Error: Generic server error.
    - '206' # Partial Content: Server is delivering only part of the resource due to a range header.
    - '503' # Service Unavailable: Server temporarily unable to handle request.
    - '401' # Unauthorized: Authentication required and has failed or not provided.
    - '204' # No Content: Server processed request but no content is returned.
    - '502' # Bad Gateway: Server received an invalid response from upstream.
    - '201' # Created: Request has been fulfilled, resulting in new resource creation.
    - '429' # Too Many Requests: Client has sent too many requests in a given time.
    - '308' # Permanent Redirect: Resource permanently at another URI, method unchanged.
    - '307' # Temporary Redirect: Resource temporarily at another URI, method unchanged.
    - '303' # See Other: Response can be found under a different URI using GET.
    - '405' # Method Not Allowed: Request method is not supported for the resource.
    - '406' # Not Acceptable: Server cannot produce a response matching client’s Accept headers.
    - '408' # Request Timeout: Server timed out waiting for the request.
    - '409' # Conflict: Request conflicts with current state of the resource.
    - '410' # Gone: Resource is permanently unavailable.
    - '413' # Content Too Large: Request entity is larger than server limits.
    - '414' # URI Too Long: Request-URI is longer than the server can handle.
    - '415' # Unsupported Media Type: Media format of the requested data is not supported.
    - '416' # Range Not Satisfiable: Requested range cannot be fulfilled.
    - '422' # Unprocessable Content: Request is well-formed but semantically incorrect.
    - '504' # Gateway Timeout: Server, as gateway, did not get timely upstream response.
    - '505' # HTTP Version Not Supported: Server does not support the HTTP version.
    - '100' # Continue: Client should continue with request.
    - '101' # Switching Protocols: Server is switching protocols as requested.
    - '202' # Accepted: Request accepted for processing, but processing is not complete.
    - '203' # Non-Authoritative Information: Server is a transforming proxy with non-authoritative metadata.
    - '205' # Reset Content: Client should reset the document view.
    - '300' # Multiple Choices: Indicates multiple options for the resource.
    - '411' # Length Required: Content-Length header is required but not provided.
    - '412' # Precondition Failed: Server does not meet one of the preconditions.
    - '417' # Expectation Failed: Server cannot meet Expect header requirements.
    - '421' # Misdirected Request: Request sent to server unable to produce a response.
    - '423' # Locked: Resource is locked (WebDAV).
    - '424' # Failed Dependency: Request failed due to failure of a previous request.
    - '425' # Too Early: Server unwilling to risk processing a request that might be replayed.
    - '426' # Upgrade Required: Client should switch to a different protocol.
    - '428' # Precondition Required: Server requires conditional request headers.
    - '431' # Request Header Fields Too Large: Headers exceed server limits.
    - '451' # Unavailable For Legal Reasons: Resource access blocked for legal reasons.
    - '501' # Not Implemented: Server does not support the requested functionality.
    - '506' # Variant Also Negotiates: Server configuration error in content negotiation.
    - '507' # Insufficient Storage: Server cannot store the representation (WebDAV).
    - '508' # Loop Detected: Server detected an infinite loop in request processing.
    - '510' # Not Extended: Further extensions required for request fulfillment.
    - '511' # Network Authentication Required: Client needs to authenticate for network access.
    - '102' # Processing: Server has received and is processing the request.
    - '103' # Early Hints: Used to return some response headers before final HTTP message.
    - '207' # Multi-Status: Conveys information about multiple resources in WebDAV.
    - '208' # Already Reported: Used inside a DAV binding to avoid enumerating bindings repeatedly.
    - '226' # IM Used: Server has fulfilled a request for the resource using instance-manipulations.
    - '305' # Use Proxy: Resource is available only through a proxy.
    - '306' # (Unused): Previously used for Switch Proxy, now reserved.
    - '402' # Payment Required: Reserved for future use, e.g., digital payments.
    - '407' # Proxy Authentication Required: Client must authenticate with proxy.
    - '418' # (Unused): Previously “I’m a teapot” (RFC 2324, joke).

lookup

Configure the File Lookup to enrich TenXObjects with lookup table values.

Below is the default configuration from: lookup/config.yaml.

Edit Online

Edit File Lookup Config Locally 

# 🔟❎ 'run' lookup file configuration

# Configuration text lookup to enrich TenXObjects
# To learn more see https://doc.log10x.com/run/initialize/lookup/

# Set the 10x pipeline to 'run'
tenx: run

# =============================== Dependencies ================================

include: run/modules/initialize/lookup

# ============================== Lookup Options ===============================

lookup:

  # 'file' specifies the .csv/.tsv lookup table file
  file: $=path("pipelines/run/initialize/lookup/http.csv")

  # 'keyField' specifies the field whose value is query lookup key.
  #  The calculated code field below captures the penultimate HTTP code value from a web log event.
  #  For example, 200 for: Mar 10 12:34:56 webserver nginx: 192.168.1.10 - - [10/Mar/2025:12:34:56 +0000] "GET /index.html HTTP/1.1" 200 1024
  keyField: $=yield TenXEnv.get("httpCodeField")

  # 'valueField' specifies the name of the result field to assign to the lookup table value
  valueField: http_message

k8s

Configure the k8s Context Extractor to enrich TenXObjects with k8s pod and container information.

Below is the default configuration from: k8s/config.yaml.

Edit Online

Edit k8s Context Extractor Config Locally 

# 🔟❎ 'run' k8s lookup configuration

# Configure a k8s field extractors to enrich TenXObjects
# To learn more see https://doc.log10x.com/run/initialize/k8s/

# Set the 10x pipeline to 'run'
tenx: run

# =============================== Dependencies ================================

include: run/modules/initialize/k8s

# ============================== Symbol Options ===============================

k8s:

  # 'extractorName' specifies which extractor (e.g., fluentK8s/filebeatK8s) to use for k8s metadata extraction
  extractorName: fluentK8s # extract k8s context using Fluent schema

  # 'namespaceNameField' specifies the name of the field in which to assign an extracted k8 namespace name status code, if found
  namespaceNameField: k8s_namespace

  # 'containerNameField' specifies the name of the field in which to assign an extracted k8 container name status code, if found
  containerNameField: k8s_container

  # 'podNameField' specifies the name of the field in which to assign an extracted k8 pod name status code, if found
  podNameField: k8s_pod

geoIP

Configure the GeoIP lookup to geo-reference TenXObjects via their ipAddress value.

Below is the default configuration from: geoIP/config.yaml.

Edit Online

Edit GeoIP lookup Config Locally 

# 🔟❎ 'run' GeoIP lookup configuration

# The GeoIP lookup geo-references TenXObjects based on their ipAddress field value.
# To learn more see https://doc.log10x.com/api/js/#TenXObject+ipAddress

# Set the 10x pipeline to 'run'
tenx: run

# =============================== Dependencies ================================

include: run/modules/initialize/geoIP

# =============================== GeoIP Options ===============================

geoIP:

  # 'file' specifies the MaxMind GeoIP DB (.mmdb) file to load.
  # To learn more see https://doc.log10x.com/api/js/#TenXLookup.loadGeoIPDB
  file: data/run/lookup/geo.mmdb

  # 'countryField' specifies the field name for country geo-reference data.
  # Defaults to "country". Set to "" to disable country enrichment.
  countryField: country

  # Uncomment the following fields to enable additional geo-reference data as needed:
  # continentField: continent
  # subdivisionField: subdivision
  # cityField: city
  # postalField: postal
  # latitudeField: latitude
  # longitudeField: longitude
Event Outputs

Event Outputs

Activate Event stream outputs to write TenXObject and template values to event stream outputs (e.g., stdout, files).

file

Configure File outputs to write TenXObject field/template values to a file.

Below is the default configuration from: file/config.yaml.

Edit Online

Edit File output Config Locally 

# 🔟❎ 'run' file output configuration

# Configure file outputs to write TenXObject instance and template field values.
# To learn more see https://doc.log10x.com/run/output/event/file

# Set the 10x pipeline to 'run'
tenx: run

# =============================== Dependencies ================================

include:
  - run/modules/output/event/file

# ============================== File Options =================================

# Multiple file outputs can be defined below
outputFile:

  # --------------------------- 10x Encoding Output ---------------------------

    # Encode TenXObjects: https://doc.log10x.com/run/transform/#template-encoding

    # 'path' specifies a path on disk to write TenXObject field values
  - path: $=path("data/sample/output") + "/encoded.log"

    # 'filter' sets a JavaScript expression that TenXObjects must evaluate as truthy to write to the output.
    #  To learn more see https://doc.log10x.com/run/output/receive/#filter-expressions
    filter: null

    # 'fields' defines the fields to write for each TenXObject sent to this output.
    #  To learn more see https://doc.log10x.com/run/output/stream/#outputfields  
    fields: 
      - encode() # https://doc.log10x.com/api/js/#TenXObject+encode

  # ----------------------------- Decode Output ------------------------------

    # Template-decode TenXObjects: https://doc.log10x.com/run/transform/#expand

  - path: $=path("data/sample/output") + "/decoded.log"

    filter: isEncoded  # https://doc.log10x.com/api/js/#TenXBaseObject+isEncoded   

    fields: 
      - text           # https://doc.log10x.com/api/js/#TenXBaseObject+text

  # ---------------------------- Template Output ----------------------------

  # Writes TenXTemplate values: https://doc.log10x.com/run/transform/#template

  - path: $=path("data/sample/output") + "/templates.json"

    # 'writeTemplates' controls whether to write new templates for TenXObjects written to this output.
    writeTemplates: true

    # 'append' controls whether to overwrite or append to the output file
    append: true

  # ----------------------------- Summary Output ------------------------------

    # Writes aggregated TenXSummaries: https://doc.log10x.com/run/aggregate

  - path: $=path("data/sample/output") + "/aggregated.csv"

    # 'header' is set to write the field names aggregated into the first line of the output
    #  We specify '$=yield' to wait until the 'enrichmentFields' env var have been evaluated 
    header: $=yield TenXString.join(",", TenXEnv.get("enrichmentFields"), "summaryVolume", "summaryBytes", "summaryTotals") + "\n"

    filter: isSummary                            # https://doc.log10x.com/api/js/#TenXBaseObject+isSummary                        

    fields: 
      - $=yield TenXEnv.get("enrichmentFields")  # https://doc.log10x.com/run/initialize/#enrichmentFields
      - summaryVolume                            # https://doc.log10x.com/api/js/#TenXSummary+summaryVolume
      - summaryBytes                             # https://doc.log10x.com/api/js/#TenXSummary+summaryBytes
      - summaryTotals                            # https://doc.log10x.com/api/js/#TenXSummary+summaryTotals

stdout

Configure Stdout/err outputs to write TenXObject and template values to a log4j2 stdout appender.

Below is the default configuration from: stdout/config.yaml.

Edit Online

Edit Stdout/err output Config Locally 

# 🔟❎ 'run' stdout output configuration

# Configure stdout/err streams for writing TenXObject instance and template values.
# To learn more see https://doc.log10x.com/run/output/event/stdout

# Set the 10x pipeline to 'run'
tenx: run

# =============================== Dependencies ================================

include: run/modules/output/event/stdout

# ============================= Stdout Options ================================

# Multiple stdout outputs can be defined below
stdout:

  # ---------------------------10x Encoding Output ----------------------------

    # Encode TenXObjects: https://doc.log10x.com/run/transform/#template-encoding

    # 'target' can be SYSTEM_OUT or SYSTEM_ERR
  - target: SYSTEM_OUT

    # 'filter' sets a JavaScript expression that TenXObjects must evaluate as truthy to write to the output.
    #  To learn more see https://doc.log10x.com/run/output/receive/#filter-expressions
    filter: isObject                

    # 'fields' defines the fields to write for each TenXObject sent to this output.
    #  To learn more see https://doc.log10x.com/run/output/stream/#outputfields  
    fields: 
      - encode() # https://doc.log10x.com/api/js/#TenXObject+encode

  # ----------------------------- Decode Output -------------------------------

    # Template-decode TenXObjects: https://doc.log10x.com/run/transform/#expand

  - target: SYSTEM_OUT

    filter: isEncoded  # https://doc.log10x.com/api/js/#TenXBaseObject+isEncoded   

    fields: 
      - text           # https://doc.log10x.com/api/js/#TenXBaseObject+text

  # ---------------------------- Template Output ------------------------------

    # Writes TenXTemplate values: https://doc.log10x.com/run/transform/#template

  - target: SYSTEM_OUT

    # 'writeTemplates' controls whether to write new TenXObjects templates to stdout. 
    # To learn more see https://doc.log10x.com/run/output/event/stdout/#stdoutwritetemplates
    writeTemplates: true 

  # ----------------------------- Summary Output ------------------------------

    # Writes aggregated TenXSummaries: https://doc.log10x.com/run/aggregate/

  - target: SYSTEM_OUT

    filter: isSummary   # https://doc.log10x.com/api/js/#TenXBaseObject+isSummary                        

    fields: 
      - $=yield TenXEnv.get("enrichmentFields") # https://doc.log10x.com/run/initialize/#enrichmentFields
      - summaryVolume                           # https://doc.log10x.com/api/js/#TenXSummary+summaryVolume
      - summaryBytes                            # https://doc.log10x.com/api/js/#TenXSummary+summaryBytes
      - summaryTotals                           # https://doc.log10x.com/api/js/#TenXSummary+summaryTotals

dev

Configure Dev outputs to generate a shareable console URL from dev app analysis results.

Below is the default configuration from: dev/config.yaml.

Edit Online

Edit Dev output Config Locally 

# 🔟❎ 'run' dev analysis output configuration

# Configure the dev output to generate a shareable console URL from analysis results.
# To learn more see https://doc.log10x.com/run/output/event/dev

# Set the 10x pipeline to 'run'
tenx: run

# =============================== Dependencies ================================

include:
  - run/modules/output/event/dev

# ============================== Dev Output Options ============================

devOutput:

    # 'enabled' controls whether to generate a console URL from analysis results
  - enabled: true

    # 'openBrowser' auto-opens the console URL in the default browser
    openBrowser: true

    # 'localOnly' outputs JSON to console without generating a URL.
    #  Use for privacy when you prefer not to generate an external URL.
    localOnly: false

    # 'baseUrl' sets the base URL of the console viewer
    baseUrl: https://console.log10x.com

    # 'sourceName' sets the display name of the input log source in the console viewer
    sourceName: your logs

    # 'dailyGb' sets the estimated daily log volume in GB for ROI projections.
    #  Adjust to match your environment's actual daily log volume.
    dailyGb: 100

    # 'filePath' specifies a path on disk to write the analysis JSON.
    #  If not specified, the JSON is only encoded into the URL.
    # filePath:

    # 'topPatterns' sets the number of top log patterns (by bytes) to include
    topPatterns: 10

    # 'vrTemplates' sets the number of template samples for volume reduction proof
    vrTemplates: 10

    # 'vrEventsPerTemplate' sets the number of encoded event samples per template
    vrEventsPerTemplate: 1

Advanced Config Files

To configure advanced options (optional) for the Dev app, Edit these files:

Bootstrap

Bootstrap

Configure the Pipeline Bootstrapper to authenticate the log10x account and launch a target pipeline.

Below is the default configuration from: bootstrap/config.yaml.

Edit Online

Edit Pipeline Bootstrapper Config Locally 

# 🔟❎ run bootstrap configuration

# This config file specifies bootstrap options for the run pipeline.
# To learn more see https://doc.log10x.com/run/bootstrap

# To learn more see https://doc.log10x.com/config/

tenx: run

# =============================== Launch Settings ==============================

# 'licenseKey' specifies a signed JWT license token used for authenticating against the 10x service
licenseKey: $=TenXEnv.get("TENX_LICENSE_KEY", "NO-LICENSE")

# 'licenseFile' specifies a path to a file containing the signed JWT license token.
# Preferred over 'licenseKey' on production hosts: the token does not appear in 'ps' output.
licenseFile: $=TenXEnv.get("TENX_LICENSE_FILE")

# 'airgapped' suppresses every call to the 10x SaaS gateway: the license is verified locally against the embedded
# public key, and ALL outbound calls to the 10x SaaS gateway are suppressed (no DNS, no TCP).
# Use for deployments where security review forbids vendor phone-home.
airgapped: $=TenXEnv.get("TENX_AIRGAPPED", "false")

# 'includePaths' specifies folders on disk for resolving relative config file/folder references in addition to the working folder
includePaths: []

# 'quiet' disables printing version information to the console.
# quiet: true

# 'jarFiles' specifies .jar files to dynamically load for use by compile, input and output API extensions.
jarFiles: []

# 'metricEndpoint' specifies the Prometheus endpoint to report usage/health metrics (enterprise version only).
# metricEndpoint: https://prometheus.log10x.com/api/v1/write

# 'disabledArgs' specifies a list of launch arguments that are disallowed from either command line or user config files.
disabledArgs: []

# 'debugEnvVars' list environment variables to debug
debugEnvVars: []
Github Sync

Github Sync

Below is the default configuration from: gitops/config.yaml.

Edit Online

Edit config.yaml Locally 

# 🔟❎ GitOps configuration

# This config file pulls symbol, config and lookup files from a remote repo to
# enable centralized configuration management via GitHub.

# To learn more see https://doc.log10x.com/config/github

# The target 10x pipeline to run (e.g., run, compile) is specified in the config files pulled from the repo
tenx: any

include:

  - source: github

    options:
      # Env-var overrides: GH_ENABLED, GH_REPO, GH_BRANCH, GH_PATH, GH_SYNC_INTERVAL.
      # Defaults below are for the compact-receiver GitOps reload test , 
      # they pull test/compact-lookup.csv from talwgx/test every 30s.
      # For a real deployment, set GH_ENABLED=false OR override the repo.
      enabled: $=TenXEnv.get("GH_ENABLED", false)

      # 'token' specifies an access token for accessing the Github repo.
      token: $=TenXEnv.get("GH_TOKEN")

      # 'repo' specifies the GitHub user/repo from which to pull
      repo: $=TenXEnv.get("GH_REPO", "talwgx/test")

      # 'branch' on 'repo' (if null, uses default repo default branch)
      branch: $=TenXEnv.get("GH_BRANCH", "main")

      # 'paths' specifies Glob patterns within 'repo' to pull and extract to a temp folder.
      # (List items don't honor $= expressions; hardcoded glob. Override repo+branch
      # to target a different repo layout. Compact lookup must live as test/*.csv.)
      paths:
        - test/*.csv

      # 'syncInterval' specifies an interval by which check to check the remote branch for updates (null to disable).
      # This interval works in conjunction with: https://doc.log10x.com/run/reload
      syncInterval: $=TenXEnv.get("GH_SYNC_INTERVAL", "30s")
Symbols

Symbols

Below is the default configuration from: symbol/config.yaml.

Edit Online

Edit config.yaml Locally 

# 🔟❎ 'run' symbol file configuration

# Loads symbol library files to transform events into well-defined TenXObjects.
# To learn more see https://doc.log10x.com/run/symbol

# Set the 10x pipeline to 'run'
tenx: run

# ============================ Symbol Options =================================

symbol:

  # 'paths' specifies the file/folder locations to scan for symbol library files.
  #  To learn more see https://doc.log10x.com/run/symbol/#symbolpaths
  paths:
    - $=path("data/shared/symbols", false)
    - $=path("<TENX_SYMBOLS_PATH>",  false)

  literals: []
Aggregators

Aggregators

Configure Aggregators to aggregate and summarize TenXObjects to publish as metrics.

Below is the default configuration from: dev/config.yaml.

Edit Online

Edit Aggregator Config Locally 

# 🔟❎ 'run' dev app aggregator config

# This file configures aggregators for the dev app.
# To learn more see https://doc.log10x.com/run/aggregate/

# Set the 10x pipeline to 'run'
tenx: run

# ============================== Aggregator Options ===========================

# Multiple aggregators can be defined below
aggregator:

  - name: devAggregator

    # 'filter' sets a JavaScript expression that TenXObjects must evaluate as truthy to be aggregated.
    #  To learn more see https://doc.log10x.com/run/output/receive/#filter-expressions
    filter: null

    # 'fields' set the names of intrinsic/extracted/calculated fields whose combined values
    #  identify a unique row within the aggregator's in-mem table for
    #  each object passed to it. This aggregator defaults to enrichment fields.
    #  To learn more see https://doc.log10x.com/run/initialize/#enrichmentFields
    fields:
      - $=yield TenXEnv.get("enrichmentFields")

    # 'totalFields' specifies fields to extract and tally into generated TenXSummaries
    totalFields: []

    # 'flushThreshold' sets the max number of TenXObjects whose values to upsert into
    #  the aggregator's in-mem table. Once exceeded, the aggregator instantiates TenXSummaries 
    #  from the table rows to send to output and reset the table.
    flushThreshold: 0 

    # 'flushInterval' sets a periodic interval, after which the aggregator
    #  flushes the rows of its in-mem table into the pipeline as TenXSummary instances.
    flushInterval: null

    # 'maxCardinality' sets the maximum number of rows for the aggregator's
    #  in-mem table. If an TenXObjects whose 'values' are not contained in the table
    #  is passed to the aggregator, 'defaultValue' will be used for it instead
    #  until the aggregator flushes its in-mem table after 'flushInterval' or 'flushThreshold'
    #  are exceeded.
    maxCardinality: 65536

    # 'defaultValue' sets a constant value that is used instead of 'values' when
    #  'maxCardinality' is exceeded within 'flushInterval'
    defaultValue: unknown
Template

Template

Below is the default configuration from: template/config.yaml.

Edit Online

Edit config.yaml Locally 

# 🔟❎ 'run' template file configuration

# Load TenXTemplates .json files that define the structure/schema of TenXObjects.
# To learn more see https://doc.log10x.com/run/template/

# Set the 10x pipeline to 'run'
tenx: run

# =============================== Template ===================================

template:
  # 'files' specifies GLOB pattern for finding JSON-encoded TenXTemplates.
  files:
    - $=path("data/templates/*.json")
    - $=path("data/sample/output/*.json")

  # 'cacheSize' controls the maximum total byte size of templates held
  #  in the in-memory cache vs. on disk. Set to 0 to disable pruning.
  cacheSize: $=parseBytes("10MB")

# =============================== Variable ===================================

var:
  # 'placeholder' specifies a character to use when encoding a TenXTemplate
  #  to signify the location of a runtime variable value.
  placeholder: "$"

  # 'maxRecurIndexes' controls the maximum number of variable values to reuse.
  maxRecurIndexes: 10

# =============================== Timestamp ==================================

timestamp:
  # 'prefix' specifies a prefix for a TenXTemplate's timestamp tokens.
  prefix: (

  # 'postfix' specifies a postfix for a TenXTemplate's timestamp tokens.
  postfix: )
Transform

Transform

Configure the Transform to transform log and trace events into well-defined TenXObjects.

timestamp

Configure the Timestamp parser to extract alphanumeric and epoch timestamp values from input events.

Below is the default configuration from: timestamp/config.yaml.

Edit Online

Edit Timestamp parser Config Locally 

# 🔟❎ 'run' timestamp parser configuration

# Identify unix/alphanumeric timestamp structures within TenXTemplates.
# https://doc.log10x.com/run/transform/timestamp/

# Set the 10x pipeline to 'run'
tenx: run

# ============================= Timestamp Options =============================

timestamp:

  # 'maxPerObject' controls the max number of timestamps to add into an TenXObject's
  # 'timestamp' array. Set to 0 for unlimited.
  maxPerObject: 0

  # 'searchDirection' controls the direction(s) from with timestamps are searched for
  #  within the object's 'text' field. Possible values: 

  #  - fromStart: search for 'maxPerObject' from the start of 'text',  
  #    limiting search to 'searchLengthLimitFromStart' characters  

  #  - fromEnd: search backward for 'maxPerObject' from the end of 'text',
  #    limiting search to 'searchLengthLimitFromEnd' characters  

  #  - fromStartAndEnd: search for 'maxPerObject' from both the start and end of 'text'
  #    limiting search to 'searchLengthLimitFromStart' and 'searchLengthLimitFromEnd' respectively  

  #  - none: do not parse timestamps

  searchDirection: fromStartAndEnd

  # 'searchLengthLimitFromStart' limits the number of characters to search for
  #  timestamps from the beginning of the object's 'text' field. 0 for unlimited
  searchLengthLimitFromStart: 0

  # 'searchLengthLimitFromStart' limits the number of characters to search for
  #  timestamps from the end of the object's 'text' field. 0 for unlimited
  searchLengthLimitFromEnd: 0

  # 'zone' controls the timezone for formatting timestamp epoch
  #  values to string. The 'java.time.ZoneId.of(String zoneId)' 
  #  is used to obtain the timezone from the 'zone' value.
  #  If set to null, the host OS timezone is used. 
  zone: null

  # 'literals' contains an array of strings to treat as a part 
  #  of any timestamp candidate found when structuring an TenXObject.
  literals:
    - T
    - Z
    - I # Go INFO
    - E # Go ERROR
    - W # Go WARN

  # 'patterns' specifies an array of date-time formats to attempt when parsing timestamps from input events.
  #  Timestamp formats that appear frequently within an input stream can be 'bumped' higher within the list below.

  patterns:
  # Most common formats
  - "'I'MMdd HH:mm:ss.S"                       # Used in Kubernetes kube-apiserver logs with INFO prefix, 1-digit microsecond precision.
  - "'I'MMdd HH:mm:ss.SS"                      # Used in Kubernetes kube-apiserver logs with INFO prefix, 2-digit microsecond precision.
  - "'I'MMdd HH:mm:ss.SSS"                     # Used in Kubernetes kube-apiserver logs with INFO prefix, 3-digit microsecond precision.
  - "'I'MMdd HH:mm:ss.SSSS"                    # Used in Kubernetes kube-apiserver logs with INFO prefix, 4-digit microsecond precision.
  - "'I'MMdd HH:mm:ss.SSSSS"                   # Used in Kubernetes kube-apiserver logs with INFO prefix, 5-digit microsecond precision.
  - "'I'MMdd HH:mm:ss.SSSSSS"                  # Used in Kubernetes kube-apiserver logs with INFO prefix, 6-digit microsecond precision.
  - "'W'MMdd HH:mm:ss.S"                       # Used in Kubernetes kube-apiserver logs with WARNING prefix, 1-digit microsecond precision.
  - "'W'MMdd HH:mm:ss.SS"                      # Used in Kubernetes kube-apiserver logs with WARNING prefix, 2-digit microsecond precision.
  - "'W'MMdd HH:mm:ss.SSS"                     # Used in Kubernetes kube-apiserver logs with WARNING prefix, 3-digit microsecond precision.
  - "'W'MMdd HH:mm:ss.SSSS"                    # Used in Kubernetes kube-apiserver logs with WARNING prefix, 4-digit microsecond precision.
  - "'W'MMdd HH:mm:ss.SSSSS"                   # Used in Kubernetes kube-apiserver logs with WARNING prefix, 5-digit microsecond precision.
  - "'W'MMdd HH:mm:ss.SSSSSS"                  # Used in Kubernetes kube-apiserver logs with WARNING prefix, 6-digit microsecond precision.
  - "'E'MMdd HH:mm:ss.S"                       # Used in Kubernetes kube-apiserver logs with ERROR prefix, 1-digit microsecond precision.
  - "'E'MMdd HH:mm:ss.SS"                      # Used in Kubernetes kube-apiserver logs with ERROR prefix, 2-digit microsecond precision.
  - "'E'MMdd HH:mm:ss.SSS"                     # Used in Kubernetes kube-apiserver logs with ERROR prefix, 3-digit microsecond precision.
  - "'E'MMdd HH:mm:ss.SSSS"                    # Used in Kubernetes kube-apiserver logs with ERROR prefix, 4-digit microsecond precision.
  - "'E'MMdd HH:mm:ss.SSSSS"                   # Used in Kubernetes kube-apiserver logs with ERROR prefix, 5-digit microsecond precision.
  - "'E'MMdd HH:mm:ss.SSSSSS"                  # Used in Kubernetes kube-apiserver logs with ERROR prefix, 6-digit microsecond precision.
  - "'F'MMdd HH:mm:ss.S"                       # Used in Kubernetes kube-apiserver logs with FATAL prefix, 1-digit microsecond precision.
  - "'F'MMdd HH:mm:ss.SS"                      # Used in Kubernetes kube-apiserver logs with FATAL prefix, 2-digit microsecond precision.
  - "'F'MMdd HH:mm:ss.SSS"                     # Used in Kubernetes kube-apiserver logs with FATAL prefix, 3-digit microsecond precision.
  - "'F'MMdd HH:mm:ss.SSSS"                    # Used in Kubernetes kube-apiserver logs with FATAL prefix, 4-digit microsecond precision.
  - "'F'MMdd HH:mm:ss.SSSSS"                   # Used in Kubernetes kube-apiserver logs with FATAL prefix, 5-digit microsecond precision.
  - "'F'MMdd HH:mm:ss.SSSSSS"                  # Used in Kubernetes kube-apiserver logs with FATAL prefix, 6-digit microsecond precision.
  - "yyyy-MM-dd HH:mm:ss.SSSSSSZ"              # Used in macOS unified logs (log show), with microsecond precision and timezone offset (e.g., -0400).
  - "yyyy-MM-dd HH:mm:ss"                      # Widely used in Java applications, databases (e.g., MySQL, PostgreSQL), and application servers (e.g., Tomcat, JBoss).
  - "yyyy-MM-dd'T'HH:mm:ss.SSSZ"               # Common in web services, APIs, Java (with DateTimeFormatter.ISO_OFFSET_DATE_TIME), Python (with datetime.isoformat()), and systems requiring precise timestamps with timezone information.
  - "MMM dd HH:mm:ss"                          # Frequently seen in syslog, Unix-based systems, network devices (e.g., Cisco routers), and web servers (e.g., Apache, Nginx).
  - "dd/MMM/yyyy:HH:mm:ss Z"                   # Standard syslog format, used in firewalls (e.g., Cisco ASA, Palo Alto), network monitoring tools, and Unix-based systems.
  - "yyyy-MM-dd HH:mm:ss,SSS"                  # Common in Java logging frameworks like Log4j and Logback.
  - "yyyy-MM-dd'T'HH:mm:ss,SSS"                # Common in Java logging with ISO 8601 date-time, comma-separated milliseconds.
  - "MM/dd/yyyy HH:mm:ss"                      # Used in Windows event logs, .NET applications, and U.S.-based systems.
  - "EEE MMM dd HH:mm:ss yyyy"                 # Human-readable format used in various logs, including some web servers and application logs.
  - "yyyy-MM-dd'T'HH:mm:ss"                    # Simplified ISO 8601 format, used in many modern applications and frameworks.
  - "HH:mm:ss"                                 # Time-only format, used when the date is implied or provided separately, common in embedded systems and some programming languages.
  - "yyyyMMdd HH:mm:ss"                        # Compact format used in some legacy systems and batch processing logs.
  - "MMM dd, yyyy h:mm:ss a"                   # Human-readable format with 12-hour clock, used in application logs and some U.S.-based systems.
  - "MMM dd, yyyy hh:mm:ss a"                  # Human-readable format with 12-hour clock (padded hour), used in application logs and some U.S.-based systems.
  # Common formats with slight variations
  - "yyyy-MM-dd HH:mm:ss.SSS"                  # Extended precision format used in Java applications and databases requiring millisecond accuracy.
  - "dd-MMM-yyyy HH:mm:ss.SSS"                 # Common in Java logging frameworks (e.g., Log4j, SLF4J) and application logs requiring human-readable dates with millisecond precision.
  - "yyyy-MM-dd'T'HH:mm:ss.S'Z'"               # Microsecond precision ISO 8601 format, 1-digit precision.
  - "yyyy-MM-dd'T'HH:mm:ss.SS'Z'"              # Microsecond precision ISO 8601 format, 2-digit precision.
  - "yyyy-MM-dd'T'HH:mm:ss.SSS'Z'"             # Microsecond precision ISO 8601 format, 3-digit precision.
  - "yyyy-MM-dd'T'HH:mm:ss.SSSS'Z'"            # Microsecond precision ISO 8601 format, 4-digit precision.
  - "yyyy-MM-dd'T'HH:mm:ss.SSSSS'Z'"           # Microsecond precision ISO 8601 format, 5-digit precision.
  - "yyyy-MM-dd'T'HH:mm:ss.SSSSSS'Z'"          # Microsecond precision ISO 8601 format, 6-digit precision.
  - "yyyy-MM-dd'T'HH:mm:ss.S'Z'"               # Nanosecond precision ISO 8601 format, 1-digit precision.
  - "yyyy-MM-dd'T'HH:mm:ss.SS'Z'"              # Nanosecond precision ISO 8601 format, 2-digit precision.
  - "yyyy-MM-dd'T'HH:mm:ss.SSS'Z'"             # Nanosecond precision ISO 8601 format, 3-digit precision.
  - "yyyy-MM-dd'T'HH:mm:ss.SSSS'Z'"            # Nanosecond precision ISO 8601 format, 4-digit precision.
  - "yyyy-MM-dd'T'HH:mm:ss.SSSSS'Z'"           # Nanosecond precision ISO 8601 format, 5-digit precision.
  - "yyyy-MM-dd'T'HH:mm:ss.SSSSSS'Z'"          # Nanosecond precision ISO 8601 format, 6-digit precision.
  - "yyyy-MM-dd'T'HH:mm:ss.SSSSSSS'Z'"         # Nanosecond precision ISO 8601 format, 7-digit precision.
  - "yyyy-MM-dd'T'HH:mm:ss.SSSSSSSS'Z'"        # Nanosecond precision ISO 8601 format, 8-digit precision.
  - "yyyy-MM-dd'T'HH:mm:ss.SSSSSSSSS'Z'"       # Nanosecond precision ISO 8601 format, 9-digit precision.
  - "yyyy-MM-dd'T'HH:mm:ssZZZZZ"               # ISO 8601 with timezone offset, used in systems where timezone information is critical.
  - "yyyy-MM-dd HH:mm:ss.SSSZZZZZ"             # Similar to above but without the 'T' separator, used in database logs and application servers.
  - "yyyy-MM-dd HH:mm:ss.S"                    # Microsecond precision format, 1-digit precision.
  - "yyyy-MM-dd HH:mm:ss.SS"                   # Microsecond precision format, 2-digit precision.
  - "yyyy-MM-dd HH:mm:ss.SSS"                  # Microsecond precision format, 3-digit precision.
  - "yyyy-MM-dd HH:mm:ss.SSSS"                 # Microsecond precision format, 4-digit precision.
  - "yyyy-MM-dd HH:mm:ss.SSSSS"                # Microsecond precision format, 5-digit precision.
  - "yyyy-MM-dd HH:mm:ss.SSSSSS"               # Microsecond precision format, 6-digit precision.
  - "MM/dd/yyyy*HH:mm:ss*SSS"                  # Used in some U.S.-based systems, particularly in legacy applications or specific logging frameworks.
  - "M/d/yyyy h:mm:ss a:SSS"                   # Common in systems using 12-hour time format, such as some Windows applications or older logging systems.
  - "M/d/yyyy hh:mm:ss a:SSS"                  # Common in systems using 12-hour time format with milliseconds and padded hour, such as some Windows applications.
  - "M/dd/yyyy hh:mm:ss a"                     # Similar to above, used in systems where millisecond precision is not needed.
  - "yyyy-MM-dd'T'HH:mm:ss.SSSX"               # ISO 8601 with basic timezone offset format, used in systems that require standardized timestamps.
  - "yyyy-MM-dd'T'HH:mm:ss.SSSz"               # ISO 8601 with timezone name, used in applications that need to display timezone information.
  - "yyyy-MM-dd'T'HH:mm:ss'Z'"                 # ISO 8601 format assuming UTC, used in systems where all times are in UTC.
  - "yy/MM/dd HH:mm:ss"                        # Two-digit year format, used in compact logs or older systems.
  - "MMM dd HH:mm:ss ZZZZ"                     # Used in some Unix-based systems and web servers, includes timezone offset.
  - "HH:mm:ss,SSS"                             # Time-only format with milliseconds, used in performance logs or systems where date is provided separately.
  - "yyyy-MM-dd*HH:mm:ss"                      # Used in some application logs where the separator is a space or asterisk.
  - "yyyy MMM dd HH:mm:ss.SSS"                 # Human-readable format with milliseconds, used in some application logs.
  - "dd/MMM/yyyy HH:mm:ss"                     # Used in European systems and some web applications.
  - "yyyy-MM-dd'T'HH:mm:ss.SSS''Z''"           # ISO 8601 with milliseconds and literal 'Z', used in systems where 'Z' is explicitly included.
  - "MMM dd yyyy HH:mm:ss"                     # Human-readable format, used in various logs.
  - "yyyy-MM-dd HH:mm:ss ZZZZ"                 # Format with timezone offset, used in systems requiring timezone information.
  - "yyyy-MM-dd HH:mm:ssZZZZZ"                 # Similar to above, used in application logs.
  - "dd MMM yyyy HH:mm:ss"                     # European format, used in some web applications and databases.
  - "MMdd_HH:mm:ss"                            # Compact format without separators, used in file names or space-constrained logs.
  - "yyyy-MM-dd HH:mm:ss,SSSZZZZZ"             # Java logging format with timezone offset.
  - "yyyyMMdd HH:mm:ss.SSS"                    # Compact format with milliseconds, used in some legacy systems.
  - "yyyy/MM/dd HH:mm:ss"                      # Format used in some Asian systems, particularly in Japan.
  - "dd/MM/yyyy HH:mm:ss"                      # Common in European systems, including some web applications and databases.
  - "MM-dd-yyyy HH:mm:ss"                      # U.S. format variant, used in some older systems.
  - "yyyyMMddHHmmss"                           # Compact format for file names or database timestamps, used in systems where space is a concern.
  - "EEE, dd MMM yyyy HH:mm:ss zzz"            # RFC 1123 format, used in HTTP headers, web servers, and email systems.

fields

Configure the Field parser to scan TenXTemplates for JSON and KV fields.

Below is the default configuration from: fields/config.yaml.

Edit Online

Edit Field parser Config Locally 

# 🔟❎ 'run' TenXTemplate field extract configuration

# Configure how to extract JSON objects and KV structures from TenXTemplates.
# To learn more see https://doc.log10x.com/run/transform/fields/

# Set the 10x pipeline to 'run'
tenx: run

# =============================== Extract Options =============================

field:

  # 'extract' controls whether to scan TenXTemplate for  JSON
  #  objects or key-value lists (e.g., 'X=Y'). 
  extract: true

  # 'nameBreaks' controls which characters found to the left of a token
  #  whose is a candidate for being a 'key' in a KV field formation should serve 
  #  as a terminator for the search. For example, for an object whose 'text' field
  #  contains the following entry ',tx_result=OK' the desired key name should be 'tx_result'
  #  and as such, the name terminator character should be ',' vs. '_' (in which case
  #  the key would have been named 'result').
  nameBreaks: ', /\{}.()[]'

  # 'valueBreaks' controls which characters found to the right of a token
  #  whose is a candidate for being a 'value' in a KV field formation should serve 
  #  as a terminator for the search. For example, for an object whose 'text' field
  #  contains the following entry: 'status=RESULT_SUCCESS' the desired KV value 
  #  should be 'RESULT_SUCCESS', and as such, the value terminator character 
  #  should be ',' vs. '_' (in which case the value would be 'RESULT').
  valueBreaks: ', /\{}()[]'

symbol

Configure the Origin selector to select the source code/binary executable origin of TenXTemplate symbol values.

Below is the default configuration from: symbol/config.yaml.

Edit Online

Edit Origin selector Config Locally 

# 🔟❎ 'run' symbol origin configuration

# Identify the origin of symbol values within TenXTemplates.
# To learn more see https://doc.log10x.com/run/transform/symbol/

# Set the 10x pipeline to 'run'
tenx: run

# =============================== Origin Options ==============================

symbol:

  # 'maxOrigins' controls the number of source/binary origins to list 
  #  per symbol token sequence. As a series of tokens within 
  #  a target TenXTemplate (e.g., 'ERROR', 'could not connect to "{}') may appear 
  #  in multiple source/binary files within the pipeline's loaded library.
  #  The sorting algorithm configured below is used to select the 'maxSymbolOrigins' 
  #  topmost entries to list.
  maxOrigins: 64

  # 'sequenceReserved' defines a list of terms to ignore when searching
  #  for the symbol tokens that constitute an TenXTemplate's 'message' portion.
  #  For example, for an event whose text contains: 'connect success = true',
  #  the value 'true' will not be considered a part of the event's message,
  #  as the 'true' value is mostly likely the result of a variable boolean state.
  #  For more information, see: https://doc.log10x.com/api/js/#TenXObject+symbolSequence
  sequenceReserved:
    # literals
    - "null"
    - "nil"
    - "true"
    - "false"
    # stopwords
    - "to"
    - "the"
    - "a"
    - "at"
    - "for"
    # generic log words
    - "log"
    - "info"
    - "http"
    # severity levels
    - TRACE
    - DEBUG
    - INFO
    - NOTICE
    - WARN
    - ERROR
    - CRITICAL
    - ALERT
    - EMERGENCY
    # stopwords
    - "with"
    - "in"
    - "on"
    - "of"
    - "from"
    - "by"
    # structured / envelope keys
    - "service"
    - "instance"
    - "component"
    - "signal"
    - "exporter"
    - "resource"
    - "scope"
    - "attributes"
    - "trace"
    - "span"
    - "flags"
    - "context"
    - "baggage"
    - "severity"
    - "timestamp"
    # tokenizer artifact
    - "t"
    # k8s / structured keys
    - "pod"
    - "container"
    - "namespace"
    - "logger"
    - "level"

  # ----------------------------- Debug Options -------------------------------

  debug:

    # 'symbol' debugging allows for verbose printing of the selection process
    #  for symbol tokens from the pipeline's 10x symbol files used to produce the
    #  results of 10x reflection functions.
    #  For more information, see: https://doc.log10x.com/api/js/#TenXObject+symbol


   # 'loggerName' controls the name of the log4j logger used by origins and symbols below
    loggerName: "symbolDebugger"

    # 'origins' outputs information about the origin selection process for an TenXTemplate symbol.
    #  For example, setting 'units' to 'foo.js' will output information about if/how it 
    #  was selected as the origin of TenXTemplate symbols.
    #  In other words, if a symbol (e.g., 'MyClass') has 'foo.java' as the source file
    #  from which it originated, adding 'foo.js' to 'units' will emit information
    #  about the selection process.
    #  Specifying '*' will emit information for all source /binary files that have
    #  been selected as the origin of any TenXTemplate objects within the pipeline.
    origins: [
    #  '*'
    ]

    # 'symbols' logs the origin selection process for symbol tokens within an TenXTemplate.
    #  For example, set 'symbols' to 'Could not connect to' to log how the 10x JavaScript 'symbolSequence' 
    #  determines the origin source code/binary file within the pipeline's symbol library. 

    #  Specifying '*' will emit information for all source code /binary files that have
    #  been selected as the origin of any TenXTemplate objects within the pipeline.
    symbols: [''
    #  '*'
    ]

group

Configure the Group sequencer to group and sequence TenXObjects.

Below is the default configuration from: group/config.yaml.

Edit Online

Edit Group sequencer Config Locally 

# 🔟❎ 'run' event grouping configuration

# Group  sequences of TenXObjects to filter, aggregate and output as a single logical unit.
# To learn more see https://doc.log10x.com/run/transform/group/

# Set the 10x pipeline to 'run'
tenx: run

# =============================== Group Options ===============================

group:

  # 'filters' specify JavaScript expressions an TenXObject instance/group must 
  #  evaluate as truthy against to be written to output
  filters: []

  # 'maxSize' defines the maximum number of TenXObjects to group
  #  before the group is sealed and forwarded into the pipeline. 
  #  Subsequent TenXObjects can form a new group.
  maxSize: 20000

  # 'flushTimeout' defines the max interval (e.g., 10s) to wait for 
  #  new events to be read from an input stream before it flushes any
  #  pending TenXObjects group into the pipeline.
  #  This mechanism is designed to avoid latencies in dispatching pending event
  #  groups to output destinations.
  flushTimeout: $=parseDuration("5s")

  # 'async' specifies whether to sequence and group TenXObjects in a dedicated thread
  async: true

parallelize

Configure the Parallel processor to distribute event parsing and transformation workloads across multiple cores..

Below is the default configuration from: parallelize/config.yaml.

Edit Online

Edit Parallel processor Config Locally 

# 🔟❎ 'run' event parallel processing configuration

# Transform log/trace events read from inputs into well-defined TenXObjects using multiple cores.
# To learn more see https://doc.log10x.com/run/transform/parallelize/

# Set the 10x pipeline to 'run'
tenx: run

# =============================== Parallel Options ============================

parallelEvent:

  # 'threadPoolSize' specifies the number of threads allocated to transform events into TenXObjects concurrently.
  #  If the value is -
  #  - = 0: transform events into TenXObjects synchronously using their input stream calling thread.
  #  - < 1: interpreted as a percentage of the number of available cores (e.g., 0.5 = use up to 50% of available cores)
  #  - = 1: allocate a single dedicated to transform events.
  #  - > 1: interpreted as a fixed number of threads (e.g., 10 = 10 threads)
  threadPoolSize: "0.5"

  # 'batchSize' specifies the maximum number of events to queue for concurrent processing before flushing.
  #  If 'threadPoolSize' is 1, this value is unused, and events are transformed into TenXObject synchronously 
  #  to their calling input's thread.  If 0, flush pending events after 'flushInterval' expires or the
  #  source input reaches end-of-file.
  batchSize: 1000

  # 'flushInterval'  specifies the maximum wait duration before flushing queued events 
  #  If 'threadPoolSize' is 1, this value is unused, and events are transformed into TenXObject synchronously 
  #  to their input thread. If 0, no wait flush interval is applied. 
  flushInterval: 2s

  # 'processingTimeout' specifies the maximum wait duration before dropping un-processed queued events 
  #  This value provides a backstop for overflowing the heap if the pipeline cannot dequeue 
  #  pending events to transform into TenXObjects. If 0, no timeout is applied.
  processingTimeout: 30s
Event Outputs

Event Outputs

Activate Event stream outputs to write TenXObject and template values to event stream outputs (e.g., stdout, files).

process

Configure Process outputs to write TenXObject and template values to a sub-process stdin.

Below is the default configuration from: process/config.yaml (* Required Fields).

Edit Online

Edit Process output Config Locally 

# 🔟❎ 'run' Fluent Bit output configuration

# Launch a Fluent Bit process into whose stdin pipe to write TenXObject instance/template field values.
# To learn more see https://doc.log10x.com/run/output/event/process/

# Set the 10x pipeline to 'run'
tenx: run

# =============================== Dependencies ================================

include: run/modules/output/event/process

# ============================ Fluent Bit Options =============================

# Multiple sub-process outputs can be defined below

procOut:

  # ----------------------------- Object Output -------------------------------

   # 'writeObjects' controls whether to write TenXObject field values to output.
  - writeObjects: false # (❗ REQUIRED)

    # -------------------------- Process Options ------------------------------

    # 'command' defines the OS-resolvable path of the sub-process to spawn
    command: /opt/fluent-bit/bin/fluent-bit

    args:
      - -i
      - stdin
      - -o
      - stdout

    # 'destroyMode' is set to 'cached' to allow a fluent-bit sub-process to
    #  to be reused across multiple 10x pipeline executions within a host JVM.
    destroyMode: cached

    # 'maxCacheSize' sets the size of the cache to used for caching fluent-bit instances
    maxCacheSize: 1

    # 'lazyLaunch' determines whether the fluent-bit instance will launch immediately
    # during the pipeline initialization, or only on-demand.
    lazyLaunch: true

    # 'startupWaitMs' and 'startupWaitPattern' controls the wait for the fluent-bit instance
    # to be ready for receiving events, before starting to emit to it.
    startupWaitMs: $=parseDuration("5s")
    startupWaitPattern: \[\s*sp\s*]\s+stream\s+processor\s+started\b

    # ------------------------------ Encoding ---------------------------------

    # 'filter' specifies an expression that any TenXObject passed to this output must
    #  evaluate as truthy against to write its instance/template values to stdout.
    #  For example, 'filter: this.timestamp' will only emit timestamped objects.
    #  To learn more see: https://doc.log10x.com/api/js/#TenXBaseObject+timestamped
    filter: null

    # 'fields' specifies a list of 10x intrinsic/calculated/extracted fields
    #  to extract from each target object passed to the output
    #  and written to stdout in the format specified by 'encodeType'
    #  If no fields are specified, encode/decode the target TenXObject.

    #  This example corresponds to the stdin format defined by fluent-bit in:
    #  https://docs.fluentbit.io/manual/pipeline/inputs/standard-input
    #  The 'text' argument below could be replaced by any list of intrinsic/calculated/extracted fields.
    #  The "" delimiter instructs 'joinFields' to produce JSON output.
    fields:  #
      - open="["
      - timestamp=dateFormat("+%s.%N")
      - separator=","
      - fields=joinFields("", "text")
      - close="]"

    # 'encodeType' specifies how 'fields' are encoded to stdout. Possible values:
    #    - 'json': formats names and values for the current TenXObject as: {"field":"str","field2":1, "field3": true}
    #    - 'delimited': formats values for the current TenXObject as: "str",1,true
    #                   The value separator (e.g., ',') is set by: 'encodeDelimiter'
    encodeType: delimited

    # 'encodeDelimiter' sets the char delimiter used to separate TenXObject
    #  field values when 'encodeType' is set to: 'delimited'
    encodeDelimiter: ' '

  # ----------------------------- Template Output -----------------------------

    # 'writeTemplates' controls whether to write new templates for TenXObjects written to
    #  this output. To learn more https://doc.log10x.com/run/template
  - writeTemplates: false  # (❗ REQUIRED)

    # -------------------------- Process Options ------------------------------

    # The output below writes the template values of TenXObjects
    # passed to this location. To learn more about TenXTemplates, see:
    # https://doc.log10x.com/run/template

    # 'command' defines the OS-resolvable path of the sub-process to spawn
    command: /opt/fluent-bit/bin/fluent-bit

    args:
      - -i
      - stdin
      - -o
      - stdout

    # 'destroyMode' is set to 'cached' to allow a fluent-bit sub-process to
    #  to be reused across multiple 10x pipeline executions within a host JVM.
    destroyMode: cached

    # 'maxCacheSize' sets the size of the cache to used for caching fluent-bit instances
    maxCacheSize: 1

    # 'lazyLaunch' determines whether the fluent-bit instance will launch immediately
    # during the pipeline initialization, or only on-demand.
    lazyLaunch: true

    # 'startupWaitMs' and 'startupWaitPattern' controls the wait for the fluent-bit instance
    # to be ready for receiving events, before starting to emit to it.
    startupWaitMs: $=parseDuration("5s")
    startupWaitPattern: \[\s*sp\s*]\s+stream\s+processor\s+started\b

http

Configure HTTP outputs to write TenXObject and template values to a log4j2 HTTP socket appender.

Below is the default configuration from: http/config.yaml (* Required Fields).

Edit Online

Edit HTTP output Config Locally 

# 🔟❎ 'run' HTTP log4j2 output configuration

# Configure HTTP outputs to write TenXObject instance and template values.
# To learn more see https://doc.log10x.com/run/output/event/http/

# Set the 10x pipeline to 'run' 
tenx: run

# =============================== Dependencies ================================

include: run/modules/output/event/http

# ============================== HTTP Options =================================

# Multiple HTTP outputs can be defined below
outputHttp:

  # ----------------------------- Object Output ------------------------------

    # 'writeObjects' controls whether to write TenXObject field values to output.
  - writeObjects: false # (❗ REQUIRED)

    # ---------------------------- HTTP Options -----------------------------

    # 'url' sets the target HTTP host to write TenXObject instance/template values.
    #  For example, the URL below posts to the Datadog log intake endpoint:
    #  https://http-intake.logs.datadoghq.com/v1/input?host=myHost&amp;ddsource=tenx&amp;service=loggingApp&ddtags=env:Dev
    url: http://127.0.0.1:8000

    # 'headers' are added to each HTTP request. For example,
    #  'DD-API-KEY=TenXEnv.get("apiKey"),DD-APPLICATION-KEY=TenXEnv.get("appKey")
    #  can be used to authenticate the Datadog intake example above.
    headers: [
       Content-Type=application/json    
    ]

    #'method' sets the HTTP method to use (e.g. GET, POST, DELETE)  
    method: POST

    # --------------------------- Encoding Options ----------------------------

    # 'filter' sets a JavaScript expression that TenXObjects must evaluate as truthy to write to the output.
    #  To learn more see https://doc.log10x.com/run/output/receive/#filter-expressions
    filter: isObject                          

    # 'fields' defines the fields to write for each TenXObject sent to this output.
    #  To learn more see https://doc.log10x.com/run/output/stream/#outputfields 
    fields: 
      - encode() 

    # 'encodeType' specifies how 'fields' are encoded to output. Possible values: [json,delimited].
    #  To learn more see https://doc.log10x.com/run/output/event/outputStream/#outputstreamencodetype
    encodeType: json      

  # ---------------------------- Template Output ----------------------------

    # The HTTP output below writes new template of TenXObjects written to http url. 
    # To learn more about TenXTemplates, see https://doc.log10x.com/run/template

    # ---------------------------General Options -----------------------------

    # 'writeTemplates' controls whether to write new templates to this output. 
  - writeTemplates: false # (❗ REQUIRED)

    # ---------------------------- HTTP Options -------------------------------

    url: http://127.0.0.1:8000

    # 'headers' are added to each HTTP request the output makes. 
    headers: [
       Content-Type=application/json    
    ]

    #'method' sets the HTTP method to use (e.g. GET, POST, DELETE)  
    method: POST

unix

Configure Unix domain socket outputs to write TenXObject and template values to a log4j2 Unix domain socket appender.

Below is the default configuration from: unix/config.yaml (* Required Fields).

Edit Online

Edit Unix domain socket output Config Locally 

# 🔟❎ 'run' Unix socket output configuration

# Configure a UNIX socket output to write TenXObject instance and template field values.
# To learn more see https://doc.log10x.com/run/output/event/unix

# Set the 10x pipeline to 'run' 
tenx: run

# =============================== Dependencies ================================

include: run/modules/output/event/unix

# ============================== UNIX Socket Options ==========================

# Multiple UNIX socket outputs can be defined below
outputUnixSocket:

  # ----------------------------- Object Output ------------------------------

    # 'writeObjects' controls whether to write TenXObject field values to output.
  - writeObjects: false # (❗ REQUIRED)

    # ---------------------------- UNIX Options -----------------------------

    # 'url' sets the target socket address to write TenXObject instance values.
    address: /var/run/socket

    # --------------------------- Encoding Options ----------------------------

    # 'filter' sets a JavaScript expression that TenXObjects must evaluate as truthy to write to the output.
    #  To learn more see https://doc.log10x.com/run/output/receive/#filter-expressions
    filter: isObject                          

    # 'fields' defines the fields to write for each TenXObject sent to this output.
    #  To learn more see https://doc.log10x.com/run/output/stream/#outputfields  
    fields: 
      - encode() 

    # 'encodeType' specifies how 'fields' are encoded to output. Possible values: [json,delimited].
    #  To learn more see https://doc.log10x.com/run/output/event/outputStream/#outputstreamencodetype
    encodeType: json      

  # ---------------------------- Template Output ----------------------------

    # 'writeTemplates' controls whether to write new templates for TenXObjects written to
    #  this output. To learn more see https://doc.log10x.com/run/template
  - writeTemplates: false

    # ---------------------------- UNIX Options -----------------------------

    # 'url' sets the target socket address to write new TenXTemplate values.
    address: /var/run/socket
Metric Outputs

Metric Outputs

Activate a Metric output to write aggregated TenXSummary instances to metric outputs (e.g., Prometheus, Datadog).

log10x

Configure log10x managed Prometheus outputs to publish TenXSummary metrics to the log10x Prometheus backend.

Below is the default configuration from: log10x/config.yaml.

Edit Online

Edit log10x managed Prometheus output Config Locally 

# 🔟❎ 'run' 10x-hosted metrics backend output configuration

# Configuration 10x-hosted metrics backend counter outputs.
# To learn more see https://doc.log10x.com/run/output/metric/log10x/

# Set the 10x pipeline to 'run'
tenx: run

# =============================== Dependencies ================================

include: run/modules/output/metric/log10x

# =========================== Log10x Metrics output ===========================

# Define multiple log10x metric outputs below
log10xMetrics:

    # ---------------------------- Counter Options ----------------------------

    # 'nameField' sets the TenXSummary field whose value identifies the micrometer counter to increment.
    #  The 'inputName' field returns the name of the aggregator that produced the target TenXSummary instance.
    #  To learn more see https://doc.log10x.com/run/aggregate
  - nameField: inputName

    # 'counterFields' specifies the TenXSummary fields whose values are used to increment the counter.
    #  To learn more see https://doc.log10x.com/run/aggregate/#summary-fields
    counterFields:
      - summaryVolume
      - summaryBytes
      - summaryTotals

    # 'tagFields' sets the TenXSummary fields used as the counter tag dimension values.
    #  Defaults to https://doc.log10x.com/run/initialize/#enrichmentFields
    tagFields:
      - $=yield TenXEnv.get("enrichmentFields")

    # 'tagFieldNames' allows setting custom names to aggregated fields for metrics reporting.
    #  Defaults to https://doc.log10x.com/run/initialize/#metricFieldNames
    tagFieldNames:
      - $=yield TenXEnv.get("metricFieldNames")

    # 'filter' sets a JavaScript expression that TenXSummaries must evaluate as truthy to write to the output.
    #  To learn more see https://doc.log10x.com/run/output/receive/#filter-expressions
    filter: isSummary

scrape

Configure Prometheus scrape metrics outputs to publish TenXSummary metrics to Prometheus.

Below is the default configuration from: scrape/config.yaml.

Edit Online

Edit Prometheus scrape metrics output Config Locally 

# 🔟❎ 'run' Prometheus metric output configuration

# Configure Prometheus metric counter outputs.
# To learn more see https://doc.log10x.com/run/output/metric/prometheus/

# Set the 10x pipeline to 'run'
tenx: run

# =============================== Dependencies ================================

include: run/modules/output/metric/prometheus/scrape

# ============================== Prometheus Outputs ===========================

# Define multiple Prometheus metric outputs below
prometheus:

    # --------------------------- Connection Options --------------------------

    # 'port' sets the port number to listen to the Prometheus scraper on
    #  Override via 'PROMETHEUS_SCRAPE_PORT' to avoid collisions in sidecar deployments
    #  (default 9100 collides with node_exporter).
  - port: $=TenXEnv.get("PROMETHEUS_SCRAPE_PORT", 9100)

    # ---------------------------- Counter Options ----------------------------

    # 'nameField' sets the TenXSummary field whose value identifies the micrometer counter to increment. 
    #  The 'inputName' field returns the name of the aggregator that produced the target TenXSummary instance.
    #  To learn more see https://doc.log10x.com/run/aggregate
    nameField: inputName

    # 'counterFields' specifies the TenXSummary fields whose values are used to increment the counter.
    #  To learn more see https://doc.log10x.com/run/aggregate/#summary-fields
    counterFields:
      - summaryVolume
      - summaryBytes
      - summaryTotals

    # 'tagFields' sets the TenXSummary fields used as the counter tag dimension values.
    #  Defaults to https://doc.log10x.com/run/initialize/#enrichmentFields
    tagFields:
      - $=yield TenXEnv.get("enrichmentFields")

    # 'tagFieldNames' allows setting custom names to aggregated fields for metrics reporting.
    #  Defaults to https://doc.log10x.com/run/initialize/#metricFieldNames
    tagFieldNames:
      - $=yield TenXEnv.get("metricFieldNames")

    # 'filter' sets a JavaScript expression that TenXSummaries must evaluate as truthy to write to the output.
    #  To learn more see https://doc.log10x.com/run/output/receive/#filter-expressions
    filter: isSummary

push-gateway

Configure Prometheus Push gateway metrics outputs to publish TenXSummary metrics to Prometheus via a Push gateway.

Below is the default configuration from: push-gateway/config.yaml.

Edit Online

Edit Prometheus Push gateway metrics output Config Locally 

# 🔟❎ 'run' Prometheus push gateway metric output configuration

# Configures Prometheus push gateway metric counter outputs.
# To learn more see https://doc.log10x.com/run/output/metric/prometheus/push-gateway/
# For the Prometheus push gateway see: https://prometheus.io/docs/practices/pushing/

# Set the 10x pipeline to 'run'
tenx: run

# =============================== Dependencies ================================

include: run/modules/output/metric/prometheus/push-gateway

# ========================== Prometheus PG Options ============================

# Define multiple Prometheus push gateway metric outputs below
prometheusPG:

    # --------------------------- Connection Options --------------------------

    # 'host' sets the address and port number of the push gateway
  - host: $=TenXEnv.get("PUSHGATEWAY_URL", "localhost:9091")

    # 'job' sets the logical job name reported to the push gateway
    job: $=TenXEnv.get("PUSHGATEWAY_JOB", "my-10x-job")

    # ---------------------------- Counter Options ----------------------------

    # 'nameField' sets the TenXSummary field whose value identifies the micrometer counter to increment. 
    #  The 'inputName' field returns the name of the aggregator that produced the target TenXSummary instance.
    #  To learn more see https://doc.log10x.com/run/aggregate
    nameField: inputName

    # 'counterFields' specifies the TenXSummary fields whose values are used to increment the counter.
    #  To learn more see https://doc.log10x.com/run/aggregate/#summary-fields
    counterFields:
      - summaryVolume
      - summaryBytes
      - summaryTotals

    # 'tagFields' sets the TenXSummary fields used as the counter tag dimension values.
    #  Defaults to https://doc.log10x.com/run/initialize/#enrichmentFields
    tagFields:
      - $=yield TenXEnv.get("enrichmentFields")

    # 'tagFieldNames' allows setting custom names to aggregated fields for metrics reporting.
    #  Defaults to https://doc.log10x.com/run/initialize/#metricFieldNames
    tagFieldNames:
      - $=yield TenXEnv.get("metricFieldNames")

    # 'filter' sets a JavaScript expression that TenXSummaries must evaluate as truthy to write to the output.
    #  To learn more see https://doc.log10x.com/run/output/receive/#filter-expressions
    filter: isSummary

    # ---------------------------- Publish Options ----------------------------

    # 'step' sets the interval to send metrics Prometheus PG (e.g., '1m').
    #  To learn more see https://micrometer.io/docs/concepts#rate-aggregation
    step: 5s

remote-write

Configure Prometheus RW metrics outputs to publish TenXSummary metrics to Prometheus via the Remote-Write protocol.

Below is the default configuration from: remote-write/config.yaml (* Required Fields).

Edit Online

Edit Prometheus RW metrics output Config Locally 

# 🔟❎ 'run' Prometheus remote write metric output configuration

# Configure Prometheus remote write metric counter outputs.
# To learn more see https://doc.log10x.com/run/output/metric/prometheus/remote-write/
# For Prometheus RW, see https://prometheus.io/docs/concepts/remote_write_spec/

# Set the 10x pipeline to 'run'
tenx: run

# =============================== Dependencies ================================

include: run/modules/output/metric/prometheus/remote-write

# ========================== Prometheus RW Options ============================

# Define multiple Prometheus RW metric outputs below
prometheusRW:

    # --------------------------- Connection Options --------------------------

    # 'host' sets the Prometheus remote write endpoint
    #  'PROMETHEUS_REMOTE_WRITE_URL' works for any remote-write-compatible backend
    #  (Grafana Cloud, Mimir, Thanos, VictoriaMetrics, Cortex).
  - host: $=TenXEnv.get("PROMETHEUS_REMOTE_WRITE_URL", "http://localhost:9090/api/v1/write")

    # 'user' for authentication
    user: $=TenXEnv.get("PROMETHEUS_REMOTE_WRITE_USERNAME") # (❗ EnvVar REQUIRED)

    # 'password' for authentication
    password: $=TenXEnv.get("PROMETHEUS_REMOTE_WRITE_PASSWORD") # (❗ EnvVar REQUIRED)

    # ---------------------------- Counter Options ----------------------------

    # 'nameField' sets the TenXSummary field whose value identifies the micrometer counter to increment. 
    #  The 'inputName' field returns the name of the aggregator that produced the target TenXSummary instance.
    #  To learn more see https://doc.log10x.com/run/aggregate
    nameField: inputName

    # 'counterFields' specifies the TenXSummary fields whose values are used to increment the counter.
    #  To learn more see https://doc.log10x.com/run/aggregate/#summary-fields
    counterFields:
      - summaryVolume
      - summaryBytes
      - summaryTotals

    # 'tagFields' sets the TenXSummary fields used as the counter tag dimension values.
    #  Defaults to https://doc.log10x.com/run/initialize/#enrichmentFields
    tagFields:
      - $=yield TenXEnv.get("enrichmentFields")

    # 'tagFieldNames' allows setting custom names to aggregated fields for metrics reporting.
    #  Defaults to https://doc.log10x.com/run/initialize/#metricFieldNames
    tagFieldNames:
      - $=yield TenXEnv.get("metricFieldNames")

    # 'filter' sets a JavaScript expression that TenXSummaries must evaluate as truthy to write to the output.
    #  To learn more see https://doc.log10x.com/run/output/receive/#filter-expressions
    filter: isSummary

    # ---------------------------- Publish Options ----------------------------

    # 'step' sets the interval to send metrics to Prometheus RW (e.g. '1m').
    #  To learn more see https://micrometer.io/docs/concepts#rate-aggregation
    step: 5s

datadog

Configure Datadog metrics outputs to publish TenXSummary metrics to Datadog.

Below is the default configuration from: datadog/config.yaml (* Required Fields).

Edit Online

Edit Datadog metrics output Config Locally 

# 🔟❎ 'run' Datadog metric output configuration

# Configure Datadog metric counter outputs.
# To learn more see https://doc.log10x.com/run/output/metric/datadog/

# Set the 10x pipeline to 'run'
tenx: run

# =============================== Dependencies ================================

include: run/modules/output/metric/datadog

# ============================== Datadog Options ==============================

# Define multiple Datadog metric outputs below
datadog:

    # --------------------------- Connection Options --------------------------

    # 'apiKey' and 'applicationKey' set the Datadog API and app keys (required).
    #  To learn more see: https://docs.datadoghq.com/account_management/api-app-keys/
  - apiKey: $=TenXEnv.get("DD_API_KEY")         # (❗ EnvVar REQUIRED)

    applicationKey: $=TenXEnv.get("DD_APP_KEY") # (❗ EnvVar REQUIRED)

    # 'uri' sets the Datadog site address to which to connect.
    #  'DD_SITE' is the bare site domain (e.g. 'us5.datadoghq.com', 'datadoghq.eu') , 
    #  matches the Datadog Agent convention so existing cluster secrets work as-is.
    #  To learn more see: https://docs.datadoghq.com/getting_started/site/
    uri: $="https://" + TenXEnv.get("DD_SITE", "us5.datadoghq.com")

    # ---------------------------- Publish Options ----------------------------

    # 'descriptions' sets whether to send meter descriptions to Datadog
    #  Turn this off to minimize the amount of data sent on each write
    descriptions: false

    # 'step' sets the interval to send metrics to CloudWatch (e.g., '1m').
    #  To learn more see: https://micrometer.io/docs/concepts#rate-aggregation
    step: 5s

    # ---------------------------- Counter Options ----------------------------

    # 'nameField' sets the TenXSummary field whose value identifies the micrometer counter to increment. 
    #  The 'inputName' field returns the name of the aggregator that produced the target TenXSummary instance.
    #  To learn more see https://doc.log10x.com/run/aggregate
    nameField: inputName

    # 'counterFields' specifies the TenXSummary fields whose values are used to increment the counter.
    #  To learn more see https://doc.log10x.com/run/aggregate/#summary-fields
    counterFields:
      - summaryVolume
      - summaryBytes
      - summaryTotals

    # 'tagFields' sets the TenXSummary fields used as the counter tag dimension values.
    #  Defaults to https://doc.log10x.com/run/initialize/#enrichmentFields
    tagFields:
      - $=yield TenXEnv.get("enrichmentFields")

    # 'tagFieldNames' allows setting custom names to aggregated fields for metrics reporting.
    #  Defaults to https://doc.log10x.com/run/initialize/#metricFieldNames
    tagFieldNames:
      - $=yield TenXEnv.get("metricFieldNames")

    # 'filter' sets a JavaScript expression that TenXSummaries must evaluate as truthy to write to the output.
    #  To learn more see https://doc.log10x.com/run/output/receive/#filter-expressions
    filter: isSummary

cloudwatch

Configure AWS CloudWatch metrics outputs to publish TenXSummary metrics to AWS CloudWatch metrics.

Below is the default configuration from: cloudwatch/config.yaml (* Required Fields).

Edit Online

Edit AWS CloudWatch metrics output Config Locally 

# 🔟❎ 'run' AWS CloudWatch metric output configuration

# Configure AWS CloudWatch metric counter outputs.
# To learn more see https://doc.log10x.com/run/output/metric/cloudwatch/

# ============================== Dependencies =================================

# Set the 10x pipeline to 'run'
tenx: run

# =============================== Dependencies ================================

include: run/modules/output/metric/cloudwatch

# ============================ AWS CloudWatch Options =========================

# Define multiple AWS CloudWatch metric outputs below

cloudwatch:

    # -------------------------- Connection Options ---------------------------

    # 'awsAccessKeyId' defines the AWS access key (required)
    #  To learn more see https://docs.aws.amazon.com/general/latest/gr/aws-sec-cred-types.html
  - awsAccessKeyId: $=TenXEnv.get("AWS_ACCESS_KEY_ID") # (❗ EnvVar REQUIRED)

    # 'awsSecretKey' defines the target AWS secret key (required)
    #  To learn more see https://docs.aws.amazon.com/general/latest/gr/aws-sec-cred-types.html
    awsSecretKey: $=TenXEnv.get("AWS_SECRET_ACCESS_KEY") # (❗ EnvVar REQUIRED)

    # 'namespace' defines the target AWS CloudWatch namespace (required)
    #  To learn more see https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/cloudwatch_concepts.html#Namespace
    namespace: $=TenXEnv.get("CW_NAMESPACE") # (❗ EnvVar REQUIRED)

    # ---------------------------- Counter Options ----------------------------

    # 'nameField' sets the TenXSummary field whose value identifies the micrometer counter to increment. 
    #  The 'inputName' field returns the name of the aggregator that produced the target TenXSummary instance.
    #  To learn more see https://doc.log10x.com/run/aggregate
    nameField: inputName

    # 'counterFields' specifies the TenXSummary fields whose values are used to increment the counter.
    #  To learn more see https://doc.log10x.com/run/aggregate/#summary-fields
    counterFields:
      - summaryVolume
      - summaryBytes
      - summaryTotals

    # 'tagFields' sets the TenXSummary fields used as the counter tag dimension values.
    #  Defaults to https://doc.log10x.com/run/initialize/#enrichmentFields
    tagFields:
      - $=yield TenXEnv.get("enrichmentFields")

    # 'tagFieldNames' allows setting custom names to aggregated fields for metrics reporting.
    #  Defaults to https://doc.log10x.com/run/initialize/#metricFieldNames
    tagFieldNames:
      - $=yield TenXEnv.get("metricFieldNames")

    # 'filter' sets a JavaScript expression that TenXSummaries must evaluate as truthy to write to the output.
    #  To learn more see https://doc.log10x.com/run/output/receive/#filter-expressions
    filter: isSummary

    # 'step' sets the interval to send metrics to CloudWatch (e.g., '1m').
    #  To learn more see https://micrometer.io/docs/concepts#rate-aggregation
    step: 5s

elastic

Configure ElasticSearch metrics outputs to publish TenXSummary metrics to ElasticSearch.

Below is the default configuration from: elastic/config.yaml (* Required Fields).

Edit Online

Edit ElasticSearch metrics output Config Locally 

# 🔟❎ 'run' ElasticSearch metric output configuration

# Configure ElasticSearch metric counter outputs.
# To learn more see https://doc.log10x.com/run/output/metric/elastic/

# Set the 10x pipeline to 'run'
tenx: run

# =============================== Dependencies ================================

include: run/modules/output/metric/elastic

# =============================== Elastic Options =============================

# Define multiple Elastic metric outputs below

elastic:

    # --------------------------- Connection Options --------------------------

    # 'host' sets the host to send metrics to (e.g., https://<deployment-name>.us-central1.gcp.cloud.es.io).
  - host: $=TenXEnv.get("ELASTICSEARCH_HOST") # (❗ EnvVar REQUIRED)

    # 'apiKeyCredentials' sets base64-encoded credentials string
    #  To learn more see: https://www.elastic.co/guide/en/elasticsearch/reference/current/security-api-create-api-key.html
    apiKeyCredentials: $=TenXEnv.get("ELASTIC_API_KEY")  # (❗ EnvVar REQUIRED)

    # 'userName' sets the the Basic Authentication username if 'apiKeyCredentials' is not set.
    userName: $=TenXEnv.get("ELASTIC_USERNAME")  # (❗ EnvVar REQUIRED)

    # 'password' sets a Basic Authentication password.
    password: $=TenXEnv.get("ELASTIC_PASSWORD")  # (❗ EnvVar REQUIRED)

    # ---------------------------- Counter Options ----------------------------

    # 'nameField' sets the TenXSummary field whose value identifies the micrometer counter to increment. 
    #  The 'inputName' field returns the name of the aggregator that produced the target TenXSummary instance.
    #  To learn more see https://doc.log10x.com/run/aggregate
    nameField: inputName

    # 'counterFields' specifies the TenXSummary fields whose values are used to increment the counter.
    #  To learn more see https://doc.log10x.com/run/aggregate/#summary-fields
    counterFields:
      - summaryVolume
      - summaryBytes
      - summaryTotals

    # 'tagFields' sets the TenXSummary fields used as the counter tag dimension values.
    #  Defaults to https://doc.log10x.com/run/initialize/#enrichmentFields
    tagFields:
      - $=yield TenXEnv.get("enrichmentFields")

    # 'tagFieldNames' allows setting custom names to aggregated fields for metrics reporting.
    #  Defaults to https://doc.log10x.com/run/initialize/#metricFieldNames
    tagFieldNames:
      - $=yield TenXEnv.get("metricFieldNames")

    # 'filter' sets a JavaScript expression that TenXSummaries must evaluate as truthy to write to the output.
    #  To learn more see https://doc.log10x.com/run/output/receive/#filter-expressions
    filter: isSummary

    # ---------------------------- Publish Options ----------------------------

    # 'step' sets the interval for sending metrics to Elastic (e.g., '1m').
    #  To learn more see: https://micrometer.io/docs/concepts#rate-aggregation
    step: 5s

    # ----------------------------- Index Options -----------------------------

    # 'elasticIndex' sets the index name to which to write metrics.
    index: tenx-metrics

    # 'autoCreateIndex' sets whether to create the index automatically if it doesn't exist.
    autoCreateIndex: true

    # 'indexDateFormat' sets the index date format used for rolling indices.
    #  This is appended to the index name, separated by 'elasticIndexDateSeparator'.
    indexDateFormat: 'yyyy-MM'

    # 'indexDateFormat' sets the separator between the index name and the date part
    indexDateSeparator: '-'

    # 'pipeline' sets the ingest pipeline name
    pipeline: null

    # 'timestampFieldName' sets the name of the elastic timestamp field.
    timestampFieldName: '@timestamp'

signalFx

Configure SignalFx metrics outputs to publish TenXSummary metrics to Splunk SignalFx.

Below is the default configuration from: signalFx/config.yaml (* Required Fields).

Edit Online

Edit SignalFx metrics output Config Locally 

# 🔟❎ 'run' SignalFx metric output configuration

# Configures SignalFx remote write metric counter outputs.
# To learn more see https://doc.log10x.com/run/output/metric/signalFx/

# Set the 10x pipeline to 'run'
tenx: run

# =============================== Dependencies ================================

include: run/modules/output/metric/signalFx

# ============================ SignalFx Options ===============================

# Define multiple SignalFx metric outputs below
signalfx:

    # --------------------------- Connection Options --------------------------

    # 'accessToken' sets the SignalFx access token.
    #  To learn more see https://dev.splunk.com/observability/reference/api/org_tokens/latest
  - accessToken: $=TenXEnv.get("SIGNALFX_ACCESS_TOKEN")  # (❗ EnvVar REQUIRED)

    # 'source' sets the value to map to "host" when shipping metrics to SignalFx.
    #  Defaults to local hostname
    source: null

    # 'sourceUri' sets the URI of an internal proxy en route to signalFx.
    #  'SIGNALFX_INGEST_URL' overrides the default ingest endpoint
    #  (e.g. https://ingest.us1.signalfx.com, https://ingest.eu0.signalfx.com).
    sourceUri: $=TenXEnv.get("SIGNALFX_INGEST_URL", "https://ingest.signalfx.com")

    # ---------------------------- Counter Options ----------------------------

    # 'nameField' sets the TenXSummary field whose value identifies the micrometer counter to increment. 
    #  The 'inputName' field returns the name of the aggregator that produced the target TenXSummary instance.
    #  To learn more see https://doc.log10x.com/run/aggregate
    nameField: inputName

    # 'counterFields' specifies the TenXSummary fields whose values are used to increment the counter.
    #  To learn more see https://doc.log10x.com/run/aggregate/#summary-fields
    counterFields:
      - summaryVolume
      - summaryBytes
      - summaryTotals

    # 'tagFields' sets the TenXSummary fields used as the counter tag dimension values.
    #  Defaults to https://doc.log10x.com/run/initialize/#enrichmentFields
    tagFields:
      - $=yield TenXEnv.get("enrichmentFields")

    # 'tagFieldNames' allows setting custom names to aggregated fields for metrics reporting.
    #  Defaults to https://doc.log10x.com/run/initialize/#metricFieldNames
    tagFieldNames:
      - $=yield TenXEnv.get("metricFieldNames")

    # 'filter' sets a JavaScript expression that TenXSummaries must evaluate as truthy to write to the output.
    #  To learn more see https://doc.log10x.com/run/output/receive/#filter-expressions
    filter: isSummary

    # ---------------------------- Publish Options ----------------------------

    # 'step' sets the interval to send metrics to SignalFx (e.g., '1m').
    #  To learn more see https://micrometer.io/docs/concepts#rate-aggregation

    # 'publishCumulativeHistogram' sets whether the SignalFx registry should emit cumulative histogram
    publishCumulativeHistogram: false

log

Configure Log4j2 metrics outputs to publish TenXSummary metrics to a log4j2 appender in Micrometer 'simple' format.

Below is the default configuration from: log/config.yaml.

Edit Online

Edit Log4j2 metrics output Config Locally 

# 🔟❎ 'run' Log4j2 metric output configuration

# Configure log4j2 metric counter outputs.
# To learn more see https://doc.log10x.com/run/output/metric/log/

# Set the 10x pipeline to 'run'
tenx: run

# =============================== Dependencies ================================

include: run/modules/output/metric/log

# ============================== Log4j2 Options ===============================

# Define multiple log4j2 metric outputs below
simple:

    # 'loggerName' sets the log4j2 logger to which registry metric data is periodically logged
    #  metric are logged a 'INFO' messages using micrometer's default 'simple' format.
    #  To learn more see: https://github.com/micrometer-metrics/micrometer/blob/main/micrometer-core/src/main/java/io/micrometer/core/instrument/simple/SimpleMeterRegistry.java
  - loggerName: consoleOut

    # ---------------------------- Counter Options ----------------------------

    # 'nameField' sets the TenXSummary field whose value identifies the micrometer counter to increment. 
    #  The 'inputName' field returns the name of the aggregator that produced the target TenXSummary instance.
    #  To learn more see https://doc.log10x.com/run/aggregate
    nameField: inputName

    # 'counterFields' specifies the TenXSummary fields whose values are used to increment the counter.
    #  To learn more see https://doc.log10x.com/run/aggregate/#summary-fields
    counterFields:
      - summaryVolume
      - summaryBytes
      - summaryTotals

    # 'tagFields' sets the TenXSummary fields used as the counter tag dimension values.
    #  Defaults to https://doc.log10x.com/run/initialize/#enrichmentFields
    tagFields:
      - $=yield TenXEnv.get("enrichmentFields")

    # 'tagFieldNames' allows setting custom names to aggregated fields for metrics reporting.
    #  Defaults to https://doc.log10x.com/run/initialize/#metricFieldNames
    tagFieldNames:
      - $=yield TenXEnv.get("metricFieldNames")

    # 'filter' sets a JavaScript expression that TenXSummaries must evaluate as truthy to write to the output.
    #  To learn more see https://doc.log10x.com/run/output/receive/#filter-expressions
    filter: isSummary

    # ---------------------------- Publish Options ----------------------------

    # 'step' sets the frequency in which metrics are logged to 'loggerName' in
    # ISO-8601 format. To learn more see:
    # https://docs.oracle.com/javase/8/docs/api/java/time/Duration.html#parse-java.lang.CharSequence-
    step: PT1S


This app is defined in dev/app.yaml.

Next: Run