Skip to content

Run

Identify high-cost log events in your analytics platforms (Splunk, Elasticsearch, Datadog, CloudWatch) to pinpoint optimization targets.

Setup Guide

Follow the steps below. Steps that require customization link to the relevant Configuration section where you can edit on github.dev or locally.

Step 1: Install

Install the Cloud binary flavor:

Step 2: Set Environment Variables (Required)

Set these environment variables before running. See path configuration for details.

Variable Description Required
TENX_CONFIG Path to your configuration directory Yes
TENX_API_KEY Your Log10x API key (get one) Yes

API Key Required

Unlike local development apps, the Cloud Reporter requires a valid TENX_API_KEY to run. The app will fail immediately without it.

Analyzer credentials (based on your platform):

Platform Variables Documentation
Splunk SPLUNK_USERNAME, SPLUNK_PASSWORD Splunk Authentication
Elasticsearch ELASTIC_USERNAME, ELASTIC_PASSWORD Elastic Security
Datadog DD_API_KEY, DD_APP_KEY Datadog API Keys
CloudWatch AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY AWS Access Keys 
export TENX_CONFIG=/path/to/your/config
export TENX_API_KEY=your-api-key
export SPLUNK_USERNAME=your-username
export SPLUNK_PASSWORD=your-password

See best practices for managing secrets in production.

Step 3: Enable Analyzer Input

  1. In the app config, uncomment the input line for your analyzer (Splunk, Elasticsearch, Datadog, CloudWatch, Coralogix, or Logz.io)

  2. In the analyzer inputs section below, find your analyzer's tab and set the host, credentials, index/log group, and time range

Step 4: Symbol Library (optional)

Load Symbol library files to transform events into typed TenXObjects.

Place symbol files in the symbolPaths folders specified in the symbol config.

Step 5: Enrichments (optional)

Enrichments add context (severity, HTTP codes) for richer aggregation reports.

To enable:

  1. In the app config, uncomment the desired enrichment include entries
  2. In the initializers section below, configure each enrichment's settings
Step 6: Metrics Output (optional)

Publish aggregated metrics to time-series databases (Prometheus, Datadog).

To enable:

  1. In the app config, uncomment the desired metric output include entry
  2. In the metric outputs section below, configure connection details
Step 7: Run

Best for: Quick local testing and development.

tenx @apps/cloud/reporter
Schedule periodically 
crontab -e

Add (runs every 15 minutes):

*/15 * * * * /usr/local/bin/tenx @apps/cloud/reporter

Best for: Isolated testing with local configuration.

docker run --rm \
  -v $TENX_CONFIG:/etc/tenx/config/ \
  -e TENX_CONFIG=/etc/tenx/config/ \
  -e TENX_API_KEY=${TENX_API_KEY} \
  -e SPLUNK_USERNAME=${SPLUNK_USERNAME} \
  -e SPLUNK_PASSWORD=${SPLUNK_PASSWORD} \
  ghcr.io/log-10x/pipeline-10x:latest \
  @apps/cloud/reporter
Schedule periodically

Create /opt/scripts/cloud-reporter.sh:

#!/bin/bash
docker run --rm \
  -v $TENX_CONFIG:/etc/tenx/config/ \
  -e TENX_CONFIG=/etc/tenx/config/ \
  -e TENX_API_KEY=${TENX_API_KEY} \
  -e SPLUNK_USERNAME=${SPLUNK_USERNAME} \
  -e SPLUNK_PASSWORD=${SPLUNK_PASSWORD} \
  ghcr.io/log-10x/pipeline-10x:latest \
  @apps/cloud/reporter

Schedule:

chmod +x /opt/scripts/cloud-reporter.sh
crontab -e

Add (runs every 15 minutes):

*/15 * * * * /opt/scripts/cloud-reporter.sh

Best for: CI/CD pipelines with version-controlled configuration.

docker run --rm \
  -e TENX_API_KEY=${TENX_API_KEY} \
  -e SPLUNK_USERNAME=${SPLUNK_USERNAME} \
  -e SPLUNK_PASSWORD=${SPLUNK_PASSWORD} \
  ghcr.io/log-10x/pipeline-10x:latest \
  '@github={"token": "${GH_TOKEN}", "repo": "my-user/my-repo"}' \
  @apps/cloud/reporter
Schedule periodically

Create /opt/scripts/cloud-reporter-gitops.sh:

#!/bin/bash
docker run --rm \
  -e TENX_API_KEY=${TENX_API_KEY} \
  -e SPLUNK_USERNAME=${SPLUNK_USERNAME} \
  -e SPLUNK_PASSWORD=${SPLUNK_PASSWORD} \
  ghcr.io/log-10x/pipeline-10x:latest \
  '@github={"token": "${GH_TOKEN}", "repo": "my-user/my-repo"}' \
  @apps/cloud/reporter

Schedule:

chmod +x /opt/scripts/cloud-reporter-gitops.sh
crontab -e

Add (runs every 15 minutes):

*/15 * * * * /opt/scripts/cloud-reporter-gitops.sh
Step 8: Verify

Verify no errors appear in the log file.

Check for Camel route errors:

  1. View the log file for errors:

    tail -f /var/log/tenx/tenx.log | grep -iE "(error|exception|failed)"

  2. If you see connection or authentication errors, enable debug logging for your analyzer's route. See Debugging Camel Routes for detailed instructions.

  3. Quick debug example - add to $TENX_CONFIG/log4j2.yaml:

    loggers:
  logger:
    - name: splunkRoute
      level: trace

    loggers:
  logger:
    - name: elasticsearchRoute
      level: trace

    loggers:
  logger:
    - name: cloudwatchLogsRoute
      level: trace

    loggers:
  logger:
    - name: datadogLogsRoute
      level: trace

View results in the dashboard:

Once running, view your cost analytics in the Cloud Reporter Dashboard.

Configuration

To configure the Cloud Reporter app, Edit these settings:

Main Config

Main Config

Below is the default configuration from: reporter/config.yaml.

Edit Online

Edit config.yaml Locally 

#
# 🔟❎  Cloud reporter app main config

# Loads the required modules for a Cloud reporter app instance.
# To learn more see https://doc.log10x.com/apps/cloud/reporter

# ============================ Bootstrap Runtime ==============================

# Initialize the runtime. To learn more see https://doc.log10x.com/run/bootstrap

tenx: run

runtimeName: $=TenXEnv.get("TENX_RUNTIME_NAME", "myCloudReporter")

# ============================ Load App Modules ===============================

# Uncomment and edit selected config.yaml files (e.g., run/input/analyzer/elastic/config.yaml)

include:

# ------------------------------ App settings ---------------------------------

  # Load app settings:
  - cloud/reporter

  # Load gitops settings for pulling additional assets from Github.
  # Edit the gitops/config.yaml file with specific config
  #
  # For additional info see - https://doc.log10x.com/config/github
  - gitops

# ------------------------------ Select Inputs --------------------------------

  # This application supports multiple possible inputs, at least one of which should be enabled
  #
  # Uncomment and edit to read events from remote log analyzers input(s):
  #
  # - run/input/analyzer/elastic         # https://doc.log10x.com/run/input/analyzer/elasticsearch
  # - run/input/analyzer/coralogix       # https://doc.log10x.com/run/input/analyzer/elasticsearch
  # - run/input/analyzer/logzio          # https://doc.log10x.com/run/input/analyzer/elasticsearch
  # - run/input/analyzer/datadogLogs     # https://doc.log10x.com/run/input/analyzer/datadogLogs
  # - run/input/analyzer/splunk          # https://doc.log10x.com/run/input/analyzer/splunk
  # - run/input/analyzer/cloudwatchLogs  # https://doc.log10x.com/run/input/analyzer/cloudwatchLogs

# --------------------------- Enrich TenXObjects ------------------------------

  # Uncomment and edit config files to enrich TenXObjects with calculated fields:

  - run/initialize/inputField   # https://doc.log10x.com/run/initialize/#inputfield
  - run/initialize/level        # https://doc.log10x.com/run/initialize/level
  - run/initialize/group        # https://doc.log10x.com/run/initialize/group
  - run/initialize/message      # https://doc.log10x.com/run/initialize/message
  - run/initialize/httpCode     # https://doc.log10x.com/run/initialize/httpCode
  - run/initialize/lookup       # https://doc.log10x.com/run/initialize/lookup
  - run/initialize/custom       # https://doc.log10x.com/run/transform/script/object
  - run/initialize/k8s          # https://doc.log10x.com/run/initialize/k8s
  # - run/initialize/geoIP        # https://doc.log10x.com/run/initialize/geoIP

# -------------------------- Select Metric Outputs ----------------------------

  # Uncomment and edit to publish aggregated TenXSummary instances to metric outputs:
  #
  # NOTE - in order to use the public Log10x grafana dashboards, log10x must be enabled

  - run/output/metric/log10x                     # https://doc.log10x.com/run/output/metric/log10x
  # - run/output/metric/prometheus/scrape        # https://doc.log10x.com/run/output/metric/prometheus/scrape
  # - run/output/metric/prometheus/push-gateway  # https://doc.log10x.com/run/output/metric/prometheus/push-gateway
  # - run/output/metric/prometheus/remote-write  # https://doc.log10x.com/run/output/metric/prometheus/remote-write
  # - run/output/metric/cloudwatch               # https://doc.log10x.com/run/output/metric/cloudwatch
  # - run/output/metric/datadog                  # https://doc.log10x.com/run/output/metric/datadog
  # - run/output/metric/elastic                  # https://doc.log10x.com/run/output/metric/elastic
  # - run/output/metric/signalFx                 # https://doc.log10x.com/run/output/metric/signalFx
  # - run/output/metric/log                      # https://doc.log10x.com/run/output/metric/log
Analyzer Inputs

Analyzer Inputs

Activate Log analyzer inputs to read events from log analyzers via REST.

splunk

Configure Splunk inputs to read events from a Splunk Cloud/on-premises deployment.

Below is the default configuration from: splunk/config.yaml (* Required Fields).

Edit Online

Edit Splunk input Config Locally 

# 🔟❎ 'run' Splunk input configuration

# Configure a Splunk event input
# To learn more see https://doc.log10x.com/run/input/analyzer/splunk/

# Set the 10x pipeline to 'run'
tenx: run

# =============================== Dependencies ================================

include: run/modules/input/analyzer/splunk

# =============================== Splunk Options ==============================

# Multiple Splunk inputs can be defined below

splunk:

    # ---------------------------- General Options ----------------------------

    # 'name' sets a unique logical name across all pipeline inputs
  - name: Splunk
    # Disabled by default - configure host/port to enable
    enabled: false

    # --------------------------- Connection Options --------------------------

    # 'host' and 'port' set the Splunk host address to connect to (e.g., '<deployment-name>.splunkcloud.com')
    host: null    # (❗ REQUIRED)
    port: null    # (Not mandatory if the host already encapsulates it)
    protocol: "https"

    # 'username' and 'password' used to authenticate against the Splunk deployment
    #  To learn more see https://docs.splunk.com/Documentation/Splunk/latest/RESTUM/RESTusing#Authentication_and_authorization
    username: $=TenXEnv.get("SPLUNK_USERNAME") # (❗ EnvVar REQUIRED)
    password: $=TenXEnv.get("SPLUNK_PASSWORD") # (❗ EnvVar REQUIRED)

    # ----------------------------- Query Options -----------------------------

    # 'pageSize' sets the number of events to retrieve with each result page
    #  Performance: Increase to 1000-2000 for high-volume environments
    pageSize: 500

    # 'query' sets the Splunk search query to execute for this job
    query: search *

    # --------------------------- Backpressure Options -----------------------

    # 'queryInterval' sets the interval between queries to the remote API
    #  Performance: Increase for rate-limited APIs; decrease for real-time needs
    queryInterval: $=parseDuration("2s")

    # 'totalDuration' sets the max duration to try reading from the the remote input
    #  Performance: Match to your job scheduling interval
    totalDuration: $=parseDuration("5min")

    # 'totalBytesLimit' sets the max total bytes to read from the remote input
    #  Performance: Increase for longer analysis windows (e.g., 200MB for 10min)
    totalBytesLimit: $=parseBytes("50MB")

    # 'totalEventsLimit' sets the max number of events to read the remote input
    #  Performance: Adjust based on memory capacity; each event consumes memory
    totalEventsLimit: 10000

    # --------------------------- Ancillary Options ---------------------------

    # 'printProgress' controls whether to print progress gage to the console
    #  This option helps debug and test the input
    printProgress: $=!TenXEnv.get("quiet")

elastic

Configure ElasticSearch inputs to read events from an ElasticSearch hosted/on-premises cluster.

Below is the default configuration from: elastic/config.yaml (* Required Fields).

Edit Online

Edit ElasticSearch input Config Locally 

# 🔟❎ 'run' Elastic input configuration

# Configure an Elastic event input
# To learn more see https://doc.log10x.com/run/input/analyzer/elasticsearch/

# Set the 10x pipeline to 'run'
tenx: run

# =============================== Dependencies ================================

include: run/modules/input/analyzer/elasticsearch

# =============================== Elastic Options ==============================

# Multiple Elastic inputs can be defined below:
elasticSearch:

    # 'name' sets a unique logical name across all pipeline inputs
  - name: ElasticSearch

    # --------------------------- Connection Options --------------------------

    # 'host' and 'port' set the ElasticSearch API endpoint to connect to
    #  For hosted Elastic (e.g. my-project-ec5189.es.us-central1.gcp.elastic.cloud).
    #  See https://cloud.elastic.co/deployments and https://i.stack.imgur.com/jXjLJ.png
    host: null # (❗ REQUIRED)
    port: "" # (not mandatory if the host already encapsulates it)

    # 'protocol' sets the connection protocol ('https' or 'http' for local testing)
    protocol: https

    # 'token' used to authenticate with. To learn more see:
    # https://www.elastic.co/guide/en/cloud-enterprise/current/ece-restful-api-authentication.html
    token: $=TenXEnv.get("ELASTIC_TOKEN") # (❗ REQUIRED)

    # ----------------------------- Query Options -----------------------------

    # 'targets' lists data streams, indices, and aliases to search.
    #  To search all data streams and indices, use '*' or '_all'.
    targets:
      - _all

    # 'startTime' and 'endTime' set the time frame of the query.
    startTime: now-1h
    endTime: now

    # --------------------------- Backpressure Options -----------------------

  # 'queryInterval' sets the interval between queries to the remote API
    queryInterval: $=parseDuration("2s")

    # 'totalDuration' sets the max duration to try reading from the the remote input
    totalDuration: $=parseDuration("5min")

    # 'totalBytesLimit' sets the max total bytes to read from the remote input
    totalBytesLimit: $=parseBytes("50MB")

    # 'totalEventsLimit' sets the max number of events to read the remote input
    totalEventsLimit: 10000

    # --------------------------- Ancillary Options ---------------------------

    # 'printProgress' controls whether to print a progress gage to the console
    #  This option helps debug and test the input
    printProgress: $=!TenXEnv.get("quiet")

    # --------------------------- Advanced Options ---------------------------

    # 'querySize' sets the number of documents retrieved with each results page.
    querySize: 500

    # 'query' sets the elastic query to request paginated documents from the  '/_search' REST endpoint.
    #  The default query (defined in 'input/analyzer/elasticsearch/query.txt')
    #  retrieves paginated values sorted in desc order by @timestamp
    query: null

logzio

Configure ElasticSearch inputs to read events from an ElasticSearch hosted/on-premises cluster.

Below is the default configuration from: logzio/config.yaml (* Required Fields).

Edit Online

Edit ElasticSearch input Config Locally 

# 🔟❎ 'run' Logz.io ElasticSearch input configuration

# Configures a Logz.io ElasticSearch event input
# To learn more see https://doc.log10x.com/run/input/analyzer/elastic/

# Set the 10x pipeline to 'run'
tenx: run

# =============================== Dependencies ================================

include: run/modules/input/analyzer/elasticsearch

# =============================== Logzio Options ==============================

# Multiple Logz.io inputs can be defined below:
elasticSearch:

    # 'name' sets a unique logical name across all pipeline inputs
  - name: logzioElasticSearch

    # --------------------------- Connection Options --------------------------

    # 'token' must be set to the Logz.io API token.
    #  See: https://docs.logz.io/user-guide/tokens/api-tokens.html
    token: null # (❗ REQUIRED)

    # 'host' and 'port' set the Logz.io service address to connect to
    host: api.logz.io
    port: 443

    # ----------------------------- Query Options -----------------------------

    # 'targets' lists data streams, indices, and aliases to search.
    #  To search all data streams and indices, use '*' or '_all'.
    targets:
      - v1

    # 'startTime' and 'endTime' set the time frame of the query.
    startTime: now-5m
    endTime: now

    # --------------------------- Backpressure Options -----------------------

    # 'queryInterval' sets the interval between queries to the remote API
    queryInterval: $=parseDuration("2s")

    # 'totalDuration' sets the max duration to try reading from the the remote input 
    totalDuration: $=parseDuration("5min")

    # 'totalBytesLimit' sets the max total bytes to read from the remote input
    totalBytesLimit: $=parseBytes("50MB")

    # 'totalEventsLimit' sets the max number of events to read the remote input
    totalEventsLimit: 10000

    # --------------------------- Ancillary Options ---------------------------

    # 'printProgress' controls whether to print a progress gage to the console
    #  This option helps debug and test the input
    printProgress: $=!TenXEnv.get("quiet")

    # --------------------------- Advanced Options ---------------------------

    # 'querySize' sets the number of documents retrieved with each results page.
    querySize: 500

    # 'query' sets the elastic query to request paginated documents from the  '/_search' REST endpoint.
    #  The default query (defined in 'input/analyzer/elasticsearch/query.txt')
    #  retrieves paginated values sorted in desc order by @timestamp
    query: null

    # 'endpoint' sets the REST API endpoint to query
    endpoint: search

    # 'messageField' defines the name of the field to extract from the retrieved
    # documents containing the log message.
    messageField: message

coralogix

Configure ElasticSearch inputs to read events from an ElasticSearch hosted/on-premises cluster.

Below is the default configuration from: coralogix/config.yaml (* Required Fields).

Edit Online

Edit ElasticSearch input Config Locally 

# 🔟❎ 'run' Coralogix ElasticSearch input configuration

# This configuration defines a Coralogix ElasticSearch input from
# which events are read and transformed into TenXObjects for aggregation and reporting.

# Set the 10x pipeline to 'run'
tenx: run

# =============================== Dependencies ================================

include: run/modules/input/analyzer/elasticsearch

# ============================== Coralogix Options ============================

# Multiple Coralogix inputs can be defined below
elasticSearch:

    # 'name' sets a unique logical name across all pipeline inputs
  - name: coralogixElasticSearch

    # ------------------------- Connection Options ----------------------------

    # 'token' must be set to the Coralogix send data API key
    #  To learn more see: https://coralogix.com/docs/user-guides/account-management/api-keys/send-your-data-api-key/
    token: null # (❗ REQUIRED)

    # 'authorizationKeyName' must be set to match Coralogix auth type
    #  To learn more see: https://coralogix.com/docs/user-guides/visualizations/hosted-opensearch-view/opensearch-api/
    authorizationKeyName: Bearer

    # 'host' and 'port' set the Coralogix service address to connect to
    host: api.coralogix.us/data/os-api

    # --------------------------- Query Options -------------------------------

    # 'targets' lists data streams, indices, and aliases to search.
    #  To search all data streams and indices, use '*' or '_all.'
    targets:
      - '*'

    # 'startTime' and 'endTime' set the time frame of the query.
    startTime: now-5m
    endTime: now

    # --------------------------- Backpressure Options -----------------------

    # 'queryInterval' sets the interval between queries to the remote API
    queryInterval: $=parseDuration("2s")

    # 'totalDuration' sets the max duration to try reading from the the remote input 
    totalDuration: $=parseDuration("5min")

    # 'totalBytesLimit' sets the max total bytes to read from the remote input
    totalBytesLimit: $=parseBytes("50MB")

    # 'totalEventsLimit' sets the max number of events to read the remote input
    totalEventsLimit: 10000

    # --------------------------- Ancillary Options ---------------------------

    # 'printProgress' controls whether to print a progress gage to the console
    #  This option helps debug and test the input
    printProgress: $=!TenXEnv.get("quiet")

    # --------------------------- Advanced Options ----------------------------

    # 'querySize' sets the number of documents retrieved with each results page.
    querySize: 500

    # 'query' sets the elastic query to request paginated documents from the  '/_search' REST endpoint.
    #  The default query (defined in 'input/analyzer/elasticsearch/query.txt')
    #  retrieves paginated values sorted in desc order by @timestamp
    query: null

    # 'timestampField' specifies the name of the field to get the event timestamp from.
    timestampField: coralogix.timestamp

    # 'messageField' defines the name of the field to extract from the retrieved
    #  documents containing the log message.
    messageField: log

cloudwatchLogs

Configure AWS Cloudwatch Logs inputs to read events from AWS CloudWatch Logs.

Below is the default configuration from: cloudwatchLogs/config.yaml (* Required Fields).

Edit Online

Edit AWS Cloudwatch Logs input Config Locally 

# 🔟❎ 'run' AWS CloudWatch Logs input configuration

# Configure an AWS CloudWatch Logs event input
# To learn more see https://doc.log10x.com/run/input/analyzer/cloudwatchLogs/

# Set the 10x pipeline to 'run'
tenx: run

# =============================== Dependencies ================================

include: run/modules/input/analyzer/cloudwatchLogs

# ============================ CloudWatch Options =============================

# Multiple CloudWatch inputs can be defined below
cloudwatchLogs:

    # 'name' sets a unique logical name across all pipeline inputs
  - name: CloudwatchLogs

    # --------------------------- Connection Options --------------------------

    # 'awsAccessKeyID' sets the AWS access key (e.g. 'AKIAIOSFODNN7EXAMPLE')
    #  To learn more see: https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html
    awsAccessKeyID: $=TenXEnv.get("AWS_ACCESS_KEY_ID") # (❗ EnvVar REQUIRED)

    # 'awsSecretKey' sets the AWS access secret key  (e.g. 'wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY')
    #  To learn more see: https://docs.aws.amazon.com/general/latest/gr/aws-sec-cred-types.html
    awsSecretKey: $=TenXEnv.get("AWS_SECRET_ACCESS_KEY") # (❗ EnvVar REQUIRED)

    # 'groupName' sets the log group name (e.g., 'my-log-group').
    #  To learn more see: https://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_GetLogEvents.html#CWL-GetLogEvents-request-logGroupName
    groupName: "" # (❗ REQUIRED)

    # 'streamName' sets the log stream name (e.g., 'my-log-stream').
    #  To learn more see: https://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_GetLogEvents.html#CWL-GetLogEvents-request-logStreamName
    streamName: "" # (❗ REQUIRED)

    # 'awsRegion' sets the region hosting the CloudWatch Logs stream
    #  To learn more see: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html
    awsRegion: $=TenXEnv.get("AWS_DEFAULT_REGION", "us-east-1")

    # ----------------------------- Query Options -----------------------------

    # 'limit' sets the number of documents retrieved with each results page.
    #  To learn more see: https://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_GetLogEvents.html#CWL-GetLogEvents-request-limit
    limit: 500

    # 'startFromHead' controls the direction of the search (old <-> new events).
    #  To learn more see: https://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_GetLogEvents.html#CWL-GetLogEvents-request-startFromHead
    startFromHead: true

    # 'startTime' sets the start of the time range, expressed as the number of milliseconds after Jan 1, 1970 00:00:00 UTC.
    #  The search includes events with a timestamp equal to or later than this value.
    #  The search excludes events with a timestamp earlier than this value. 
    #  To learn more see: https://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_GetLogEvents.html#CWL-GetLogEvents-request-startTime
    startTime: $=now("-240h")

    # 'endTime' sets the end of the time range, expressed as the number of milliseconds after Jan 1, 1970 00:00:00 UTC.
    #  The search excludes events with a timestamp equal to or later than this value.
    #  To learn more see: https://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_GetLogEvents.html#CWL-GetLogEvents-request-endTime
    endTime: $=now()

    # --------------------------- Backpressure Options -----------------------

    # 'queryInterval' sets the interval between queries to the remote API
    queryInterval: $=parseDuration("2s")

    # 'totalDuration' sets the max duration to try reading from the the remote input 
    totalDuration: $=parseDuration("5min")

    # 'totalBytesLimit' sets the max total bytes to read from the remote input
    totalBytesLimit: $=parseBytes("50MB")

    # 'totalEventsLimit' sets the max number of events to read the remote input
    totalEventsLimit: 10000

    # --------------------------- Ancillary Options ---------------------------

    # 'printProgress' controls whether to print a gage to the console
    #  This option helps debug and test the input
    printProgress: $=!TenXEnv.get("quiet")

    # 'messageField' sets the JSON field to extract from CloudWatch log messages
    #  For K8s/Docker logs with 10x encoding, use 'log' to extract the encoded log field
    messageField: log

datadogLogs

Configure Datadog Logs inputs to read events from Datadog Logs.

Below is the default configuration from: datadogLogs/config.yaml (* Required Fields).

Edit Online

Edit Datadog Logs input Config Locally 

# 🔟❎ 'run' Datadog Logs input configuration

# Configure a Datadog Logs event input
# To learn more see https://doc.log10x.com/run/input/analyzer/datadogLogs/

# Set the 10x pipeline to 'run'
tenx: run

# =============================== Dependencies ================================

include: run/modules/input/analyzer/datadogLogs

# ============================== Datadog Inputs ===============================

# Multiple Datadog inputs can be defined below
datadogLogs:

    # 'name' sets a unique logical name across all pipeline inputs
  - name: DatadogLogs

    # -------------------------- Connection Options ---------------------------

    # 'apiKey' and 'appKey' must be set to authenticate with the logs endpoint
    #  To learn more see: https://docs.datadoghq.com/account_management/api-app-keys/
    apiKey: $=TenXEnv.get("DD_API_KEY") # (❗ REQUIRED)

    appKey: $=TenXEnv.get("DD_APP_KEY") # (❗ REQUIRED)

    # 'host' points to the target Datadog site
    #  To learn more see: https://docs.datadoghq.com/getting_started/site/
    host: us5.datadoghq.com

    # ----------------------------- Query Options -----------------------------

    # 'query' sets the Datadog query used to request paginated documents.
    #  The default query (defined in 'run/modules/input/analyzer/datadog/query.txt') retrieves
    #  values sorted in desc order by @timestamp, starting from the value of 'searchFrom'
    query: null

    # 'searchFrom sets the initial value used to request the first page of results.
    searchFrom: now-1000h

    # 'querySize' sets the number of documents to retrieve with each paginated request
    querySize: 500

    # 'indexes' sets the lists of data indices to search in
    # If not provided, will use a default of '*'
    indexes: null

    # --------------------------- Backpressure Options -----------------------

    # 'queryInterval' sets the interval between queries to the remote API
    queryInterval: $=parseDuration("5s")

    # 'totalDuration' sets the max duration to try reading from the the remote input 
    totalDuration: $=parseDuration("5min")

    # 'totalBytesLimit' sets the max total bytes to read from the remote input
    totalBytesLimit: $=parseBytes("50MB")

    # 'totalEventsLimit' sets the max number of events to read the remote input
    totalEventsLimit: 10000

    # --------------------------- Ancillary Options ---------------------------

    # 'printProgress' controls whether to print a progress gage to the console
    #  This option helps test the input
    printProgress: $=!TenXEnv.get("quiet")
Initializers

Initializers

Activate an Object initializer to enrich TenXObjects with additional context.

message

Configure the Symbol Message Calculator to enrich TenXObjects with logical message symbol sequence and origin values.

Below is the default configuration from: message/config.yaml.

Edit Online

Edit Symbol Message Calculator Config Locally 

# 🔟❎ 'run' symbol lookup configuration

# Configure a symbol origin lookup to enrich TenXObjects
# To learn more see https://doc.log10x.com/run/initialize/message/

# Set the 10x pipeline to 'run'
tenx: run

# =============================== Dependencies ================================

include: run/modules/initialize/message

# ============================== Symbol Options ===============================

symbol:

  # 'types' specifies the types of symbols to search for; if the first type does not yield a result, the next one is tried etc.
  #  Supported values (case insensitive), PACKAGE, CLASS, METHOD, LOG, ENUM, CONST, TEXT, EXEC, ANY
  #  To learn more see https://doc.log10x.com/run/transform/symbol/#contexts
  contexts: log,exec

  # 'messageField' specifies the field name to assign the source/binary origin file name to target TenXObjects
  messageField: message_pattern

  # 'maxLen' specifies the max char len of the output messageField
  maxLen: 70

level

Configure the Level Classifier to classify TenXObjects with a severity level.

Below is the default configuration from: level/config.yaml.

Edit Online

Edit Level Classifier Config Locally 

# 🔟❎ 'run' level classifier configuration

# Configuration level classifier to enrich TenXObjects
# To learn more see https://doc.log10x.com/run/initialize/level/

# Set the 10x pipeline to 'run'
tenx: run

# =============================== Dependencies ================================

include: run/modules/initialize/level

# =============================== Level Options ===============================

level:

  # 'field' specify the field name to assign with the inferred severity level
  field: severity_level

  # 'terms' specify severity level classification, start-of-line matching
  #  Used to identify the severity level (TRACE, DEBUG, INFO, WARN, ERROR, CRITICAL) of a log line, typically marking
  #  the start of a log event (head). Sorted by likelihood and uniqueness. Used in a Trie for O(m) matching.
  terms:
    - 'trace=TRACE'          # Common in JavaScript (Winston) or custom logs; e.g., "trace Variable x = 5"
    - 'Trace=TRACE'          # Mixed case; e.g., "Trace Variable x = 5"
    - 'TRACE=TRACE'          # Uppercase, less common; e.g., "TRACE Message"
    - 'debug=DEBUG'          # Common in JavaScript (Winston) or custom logs; e.g., "debug Entering method foo"
    - 'Debug=DEBUG'          # Mixed case; e.g., "Debug Entering method foo"
    - 'DEBUG=DEBUG'          # Uppercase; e.g., "DEBUG Message"
    - 'DBG=DEBUG'            # Abbreviation, less common; e.g., "DBG Entering method foo"
    - 'info=INFO'            # Common in JavaScript (Winston) or custom logs; e.g., "info Application started"
    - 'Info=INFO'            # Mixed case; e.g., "Info Application started"
    - 'INFO=INFO'            # Uppercase; e.g., "INFO Message"
    - 'notice=INFO'          # Common in custom logs or lowercase syslog; e.g., "notice Configuration updated"
    - 'Notice=INFO'          # Mixed case; e.g., "Notice Configuration updated"
    - 'NOTICE=INFO'          # Uppercase, syslog; e.g., "NOTICE Message"
    - 'warn=WARN'            # Common in JavaScript (Winston) or custom logs; e.g., "warn Low disk space"
    - 'Warn=WARN'            # Mixed case; e.g., "Warn Low disk space"
    - 'WARN=WARN'            # Uppercase; e.g., "WARN Message"
    - 'warning=WARN'         # Common in custom logs or verbose frameworks; e.g., "warning Low disk space"
    - 'Warning=WARN'         # Mixed case; e.g., "Warning Low disk space"
    - 'WARNING=WARN'         # Uppercase; e.g., "WARNING Message"
    - 'error=ERROR'          # Common in JavaScript (Winston) or custom logs; e.g., "error Failed to connect"
    - 'Error=ERROR'          # Mixed case; e.g., "Error Failed to connect"
    - 'ERROR=ERROR'          # Uppercase; e.g., "ERROR Message"
    - 'err=ERROR'            # Lowercase abbreviation; e.g., "err Failed to connect"
    - 'ERR=ERROR'            # Uppercase abbreviation; e.g., "ERR Message"
    - 'TypeError=ERROR'      # JavaScript, stack trace; e.g., "TypeError: undefined is not a function"
    - 'ReferenceError=ERROR' # JavaScript, stack trace; e.g., "ReferenceError: x is not defined"
    - 'ValueError=ERROR'     # Python, stack trace; e.g., "ValueError: invalid literal"
    - 'PHP_Warning=ERROR'    # PHP, warning message; e.g., "PHP Warning: Undefined variable"
    - 'fatal=CRITICAL'       # Common in custom logs; e.g., "fatal System crash"
    - 'Fatal=CRITICAL'       # Mixed case; e.g., "Fatal System crash"
    - 'FATAL=CRITICAL'       # Uppercase; e.g., "FATAL Message"
    - 'critical=CRITICAL'    # Common in custom logs; e.g., "critical System crash"
    - 'Critical=CRITICAL'    # Mixed case; e.g., "Critical System crash"
    - 'CRITICAL=CRITICAL'    # Uppercase; e.g., "CRITICAL Message"
    - 'crit=CRITICAL'        # Lowercase abbreviation, syslog; e.g., "crit System crash"
    - 'CRIT=CRITICAL'        # Uppercase abbreviation, syslog; e.g., "CRIT Message"
    - 'alert=CRITICAL'       # Common in custom logs or syslog; e.g., "alert High CPU usage"
    - 'Alert=CRITICAL'       # Mixed case; e.g., "Alert High CPU usage"
    - 'ALERT=CRITICAL'       # Uppercase, syslog; e.g., "ALERT Message"
    - 'emerg=CRITICAL'       # Common in custom logs or lowercase syslog; e.g., "emerg System unusable"
    - 'Emerg=CRITICAL'       # Mixed case; e.g., "Emerg System unusable"
    - 'EMERG=CRITICAL'       # Uppercase, syslog; e.g., "EMERG Message"
    - 'panic=CRITICAL'       # Go panic, first line; e.g., "panic: runtime error: index out of range"
    - 'fatal_error=CRITICAL' # Go fatal error, first line; e.g., "fatal error: all goroutines are asleep"
    - 'thread_main_panicked_at=CRITICAL' # Rust panic, first line; e.g., "thread 'main' panicked at 'explicit panic'"
    - 'Segmentation_fault=CRITICAL' # C/C++ memory violation, first line; e.g., "Segmentation fault (core dumped)"
    - 'Aborted=CRITICAL'     # C/C++ termination, first line; e.g., "Aborted (core dumped)"
    - 'Assertion_failed=CRITICAL' # C/C++ assertion failure, first line; e.g., "Assertion failed: x > 0"
    - 'PHP_Fatal_error=CRITICAL' # PHP fatal error, first line; e.g., "PHP Fatal error: Out of memory"
    - 'Traceback_most_recent_call_last=CRITICAL' # Python stack trace, first line; e.g., "Traceback (most recent call last)"
    - 'Stack_trace=CRITICAL' # Generic stack trace, first line; e.g., "Stack trace"
    - 'Error=CRITICAL'       # JavaScript, stack trace; e.g., "Error: Something went wrong"
    - 'Fatal_error=CRITICAL' # PHP, stack trace; e.g., "Fatal error: Uncaught Exception"
    - 'Unhandled_exception=CRITICAL' # C#/Dart, stack trace; e.g., "Unhandled exception: System.NullReferenceException"
    - 'Exception_in_thread=CRITICAL' # Java, stack trace; e.g., "Exception in thread 'main' java.lang.RuntimeException"
    - 'goroutine=CRITICAL'   # Go, stack trace; e.g., "goroutine 1 [running]:"
    - 'failed=ERROR'         # ERROR-level entries in application, system, or web server logs

  # 'timestampPatterns' patterns for LogEvent head classification, start-of-line matching
  #  Used to identify timestamp formats in log lines and infer their implied severity level based on the first character.
  #  Formats are typical of Kubernetes kube-apiserver logs with varying microsecond precision. The first character
  #  ('I', 'W', 'E', 'F') determines the severity: I=INFO, W=WARN, E=ERROR, F=CRITICAL. Sorted by precision.
  timestampPatterns:
    - "'I'MMdd HH:mm:ss.S=INFO"          # Used in Kubernetes kube-apiserver logs with INFO prefix, 1-digit microsecond precision
    - "'I'MMdd HH:mm:ss.SS=INFO"         # Used in Kubernetes kube-apiserver logs with INFO prefix, 2-digit microsecond precision
    - "'I'MMdd HH:mm:ss.SSS=INFO"        # Used in Kubernetes kube-apiserver logs with INFO prefix, 3-digit microsecond precision
    - "'I'MMdd HH:mm:ss.SSSS=INFO"       # Used in Kubernetes kube-apiserver logs with INFO prefix, 4-digit microsecond precision
    - "'I'MMdd HH:mm:ss.SSSSS=INFO"      # Used in Kubernetes kube-apiserver logs with INFO prefix, 5-digit microsecond precision
    - "'I'MMdd HH:mm:ss.SSSSSS=INFO"     # Used in Kubernetes kube-apiserver logs with INFO prefix, 6-digit microsecond precision
    - "'W'MMdd HH:mm:ss.S=WARN"          # Used in Kubernetes kube-apiserver logs with WARNING prefix, 1-digit microsecond precision
    - "'W'MMdd HH:mm:ss.SS=WARN"         # Used in Kubernetes kube-apiserver logs with WARNING prefix, 2-digit microsecond precision
    - "'W'MMdd HH:mm:ss.SSS=WARN"        # Used in Kubernetes kube-apiserver logs with WARNING prefix, 3-digit microsecond precision
    - "'W'MMdd HH:mm:ss.SSSS=WARN"       # Used in Kubernetes kube-apiserver logs with WARNING prefix, 4-digit microsecond precision
    - "'W'MMdd HH:mm:ss.SSSSS=WARN"      # Used in Kubernetes kube-apiserver logs with WARNING prefix, 5-digit microsecond precision
    - "'W'MMdd HH:mm:ss.SSSSSS=WARN"     # Used in Kubernetes kube-apiserver logs with WARNING prefix, 6-digit microsecond precision
    - "'E'MMdd HH:mm:ss.S=ERROR"         # Used in Kubernetes kube-apiserver logs with ERROR prefix, 1-digit microsecond precision
    - "'E'MMdd HH:mm:ss.SS=ERROR"        # Used in Kubernetes kube-apiserver logs with ERROR prefix, 2-digit microsecond precision
    - "'E'MMdd HH:mm:ss.SSS=ERROR"       # Used in Kubernetes kube-apiserver logs with ERROR prefix, 3-digit microsecond precision
    - "'E'MMdd HH:mm:ss.SSSS=ERROR"      # Used in Kubernetes kube-apiserver logs with ERROR prefix, 4-digit microsecond precision
    - "'E'MMdd HH:mm:ss.SSSSS=ERROR"     # Used in Kubernetes kube-apiserver logs with ERROR prefix, 5-digit microsecond precision
    - "'E'MMdd HH:mm:ss.SSSSSS=ERROR"    # Used in Kubernetes kube-apiserver logs with ERROR prefix, 6-digit microsecond precision
    - "'F'MMdd HH:mm:ss.S=CRITICAL"      # Used in Kubernetes kube-apiserver logs with FATAL prefix, 1-digit microsecond precision
    - "'F'MMdd HH:mm:ss.SS=CRITICAL"     # Used in Kubernetes kube-apiserver logs with FATAL prefix, 2-digit microsecond precision
    - "'F'MMdd HH:mm:ss.SSS=CRITICAL"    # Used in Kubernetes kube-apiserver logs with FATAL prefix, 3-digit microsecond precision
    - "'F'MMdd HH:mm:ss.SSSS=CRITICAL"   # Used in Kubernetes kube-apiserver logs with FATAL prefix, 4-digit microsecond precision
    - "'F'MMdd HH:mm:ss.SSSSS=CRITICAL"  # Used in Kubernetes kube-apiserver logs with FATAL prefix, 5-digit microsecond precision
    - "'F'MMdd HH:mm:ss.SSSSSS=CRITICAL" # Used in Kubernetes kube-apiserver logs with FATAL prefix, 6-digit microsecond precision

group

Configure the Group initializer to combine multi-line events into TenXObject group instances.

Below is the default configuration from: group/config.yaml.

Edit Online

Edit Group initializer Config Locally 

# 🔟❎ 'run' event grouping configuration

# Group  sequences of TenXObjects to filter, aggregate and output as a single logical unit.
# To learn more see https://doc.log10x.com/run/transform/group/

# Set the 10x pipeline to 'run'
tenx: run

# =============================== Dependencies ================================

include: run/modules/initialize/group

# =============================== Group Options ===============================


# 'indicators'  specifies a list of `value:state` pairs that determine if a log line's start marks a group head (`true`) or child (`false`).
group:

  # 'indicators' specifies a list of strings that, when matched at the start of a log line's text, designate it as a group head.
  # Unmatched lines default to false, indicating they are group children.
  indicators:
    - '192.'              # Indicates a private IP address, common in web server logs (e.g., "192.168.1.1 - - [...]")
    - '10.'               # Indicates a private IP range, often in Kubernetes or internal network logs (e.g., "10.244.0.125 - - [...]")
    - '172.'              # Indicates a private IP range, typical in enterprise network logs (e.g., "172.16.0.1 - - [...]")
    - '127.'              # Indicates localhost, a common web server log initiator (e.g., "127.0.0.1 - - [...]")
    - 'GET '              # Indicates an HTTP GET request, marking the start of a web transaction log (e.g., "GET /index.html HTTP/1.1")
    - 'POST '             # Indicates an HTTP POST request, marking the start of a web transaction log (e.g., "POST /api HTTP/1.1")
    - 'PUT '              # Indicates an HTTP PUT request, marking the start of a web transaction log (e.g., "PUT /resource HTTP/1.1")
    - 'DELETE '           # Indicates an HTTP DELETE request, marking the start of a web transaction log (e.g., "DELETE /resource HTTP/1.1")
    - 'HEAD '             # Indicates an HTTP HEAD request, marking the start of a web transaction log (e.g., "HEAD /index.html HTTP/1.1")
    - 'OPTIONS '          # Indicates an HTTP OPTIONS request, marking the start of a web transaction log (e.g., "OPTIONS /api HTTP/1.1")
    - 'HTTP/'             # Indicates an HTTP protocol version, marking the start of a web transaction log (e.g., "GET /index.html HTTP/1.1")
    - 'kernel:'           # Indicates a Linux kernel log entry, marking the start of a system event (e.g., "kernel: [0.123456] Device initialized")
    - 'sshd['             # Indicates an SSH daemon log entry, marking the start of a security event (e.g., "sshd[1234]: Accepted password ...")
    - 'systemd['          # Indicates a systemd service log entry, marking the start of a system service event (e.g., "systemd[1]: Started service ...")
    - 'cron['             # Indicates a cron daemon log entry, marking the start of a scheduled task event (e.g., "cron[1234]: Running job ...")
    - 'syslog:'           # Indicates a syslog message, marking the start of a system log event (e.g., "syslog: Message ...")
    - 'rsyslogd:'         # Indicates an rsyslog daemon log entry, marking the start of a logging system event (e.g., "rsyslogd: Log started ...")
    - 'auditd['           # Indicates an audit daemon log entry, marking the start of a security audit event (e.g., "auditd[1234]: Audit event ...")
    - 'daemon:'           # Indicates a syslog daemon facility log, marking the start of a system service event (e.g., "daemon: Service started ...")
    - 'user:'             # Indicates a syslog user facility log, marking the start of a user-related event (e.g., "user: User logged in ...")
    - 'local0:'           # Indicates a syslog local facility log, marking the start of a custom system event (e.g., "local0: Custom message ...")
    - 'local1:'           # Indicates a syslog local facility log, marking the start of a custom system event (e.g., "local1: Custom message ...")
    - 'level='            # Indicates a structured log key, marking the start of a key-value log entry (e.g., "level=info msg=Started")
    - 'msg='              # Indicates a structured log key, marking the start of a message log entry (e.g., "msg=Application started")
    - 'message='          # Indicates a structured log key, marking the start of a message log entry (e.g., "message=Application started")
    - 'event='            # Indicates a structured log key, marking the start of an event log entry (e.g., "event=Service startup")
    - 'thread='           # Indicates a structured log key, marking the start of a thread-specific log entry (e.g., "thread=main Processing ...")
    - 'Starting '         # Indicates the start of a process initiation log (e.g., "Starting server on port 8080")
    - 'Stopping '         # Indicates the start of a process termination log (e.g., "Stopping service ...")
    - 'Running '          # Indicates the start of a process status log (e.g., "Running task ...")
    - 'Listening '        # Indicates the start of a network service log (e.g., "Listening on port 8080 ...")
    - 'Connecting '       # Indicates the start of a connection attempt log (e.g., "Connecting to database ...")
    - 'Connected '        # Indicates the start of a successful connection log (e.g., "Connected to database ...")
    - 'Disconnected '     # Indicates the start of a disconnection log (e.g., "Disconnected from server ...")
    - 'Processing '       # Indicates the start of a task processing log (e.g., "Processing request ...")
    - 'Received '         # Indicates the start of a data reception log (e.g., "Received message ...")
    - 'Sent '             # Indicates the start of a data transmission log (e.g., "Sent response ...")
    - 'User '             # Indicates the start of a user action log (e.g., "User logged in ...")
    - 'Authentication '   # Indicates the start of an authentication log (e.g., "Authentication successful ...")
    - 'Authorized '       # Indicates the start of an authorization log (e.g., "Authorized user access ...")
    - 'Failed '           # Indicates the start of a failure log (e.g., "Failed login attempt ...")
    - 'kubelet:=true'     # Indicates a Kubernetes kubelet log entry, marking the start of a node event (e.g., "kubelet: Starting kubelet")
    - 'pod:=true'         # Indicates a Kubernetes pod log entry, marking the start of a pod event (e.g., "pod: Starting container")
    - 'container:=true'   # Indicates a Kubernetes container log entry, marking the start of a container event (e.g., "container: Started")
    - 'namespace:=true'   # Indicates a Kubernetes namespace log entry, marking the start of a namespace event (e.g., "namespace: Created")
    - 'Traceback '        # Indicates the start of a Python stack trace (e.g., "Traceback (most recent call last):")
    - 'File "'            # Indicates a Python stack trace line (e.g., "File "/script.py", line 10")
    - 'Trace:'            # Indicates a Node.js console trace (e.g., "Trace: Show me")
    - 'Error:'            # Indicates a Node.js error log (e.g., "Error: Something went wrong")
    - 'Warning:'          # Indicates a Node.js warning log (e.g., "Warning: Deprecated method")
    - 'Exception in thread ' # Indicates a Java exception header (e.g., "Exception in thread 'main'")
    - 'goroutine '        # Indicates a Go goroutine stack trace (e.g., "goroutine 1 [running]:")
    - 'panic:'            # Indicates a Go panic (e.g., "panic: runtime error")
    - 'thread '           # Indicates a Rust thread panic (e.g., "thread 'main' panicked at")
    - 'stack backtrace:'  # Indicates a Rust backtrace header (e.g., "stack backtrace:")
    - 'PHP Warning:'      # Indicates a PHP warning (e.g., "PHP Warning: Undefined variable")
    - 'PHP Fatal error:'  # Indicates a PHP fatal error (e.g., "PHP Fatal error: Out of memory")
    - 'in /'              # Indicates a PHP/Ruby file path in stack trace (e.g., "in /path/to/file.php:10")
    - 'Exception:'        # Indicates a C# exception (e.g., "Exception: Invalid operation")
    - 'terminate called'  # Indicates a C++ termination (e.g., "terminate called after throwing an instance of 'std::exception'")
    - 'syslog:'           # Indicates a Linux syslog message (e.g., "syslog: Message ...")
    - 'IN='               # Indicates an iptables firewall input log (e.g., "IN=eth0 OUT=")
    - 'OUT='              # Indicates an iptables firewall output log (e.g., "OUT=eth0 SRC=")
    - 'SRC='              # Indicates an iptables firewall source IP log (e.g., "SRC=192.168.1.1 DST=")
    - 'DST='              # Indicates an iptables firewall destination IP log (e.g., "DST=10.0.0.1 LEN=")
    - 'KafkaServer:'      # Indicates a Kafka server log (e.g., "KafkaServer: Starting Kafka server")
    - 'Redis:'            # Indicates a Redis server log (e.g., "Redis: Server initialized")

  # 'negators' specifies patterns that mark a line as a continuation of a previous event (e.g., stack trace lines).
  # Lines matching these patterns are NOT standalone events and typically don't need their own message pattern.
  # Events that are NOT negators (isStandalone=true) will get a message even without timestamp/severity.
  negators:
    # Java/Kotlin/Scala stack traces
    - "\tat "             # Java stack frame (e.g., "\tat com.example.MyClass.method(MyClass.java:42)")
    - "    at "           # Java stack frame with spaces (e.g., "    at com.example.MyClass.method")
    - "\t... "            # Java truncated stack (e.g., "\t... 15 more")
    - "    ... "          # Java truncated stack with spaces
    - "Caused by: "       # Java chained exception (e.g., "Caused by: java.io.IOException")
    - "Suppressed: "      # Java suppressed exception (e.g., "Suppressed: java.lang.Exception")

    # .NET/C# stack traces
    - "   at "            # .NET stack frame (e.g., "   at MyNamespace.MyClass.Method()")
    - "--- End of "       # .NET inner exception marker (e.g., "--- End of inner exception stack trace ---")
    - " ---> "            # .NET inner exception chain

    # Python stack traces
    - "  File \""         # Python stack frame (e.g., '  File "/path/to/script.py", line 10')
    - "    raise "        # Python raise statement in traceback
    - "    return "       # Python return statement in traceback

    # Node.js/JavaScript stack traces
    - "    at "           # Node.js stack frame (e.g., "    at Object.<anonymous> (/path/file.js:10:15)")
    - "    at new "       # Node.js constructor stack frame
    - "    at Module."    # Node.js module stack frame
    - "    at Object."    # Node.js object stack frame

    # Go stack traces
    - "\t/"               # Go stack frame path (e.g., "\t/go/src/main.go:42 +0x123")
    - "created by "       # Go goroutine creator (e.g., "created by main.main")

    # Ruby stack traces
    - "\tfrom "           # Ruby stack frame (e.g., "\tfrom /path/to/file.rb:10:in `method'")
    - "        from "     # Ruby stack frame with spaces

    # Rust stack traces
    - "             at "  # Rust source location (13 spaces + at)
    - "  --> "            # Rust error pointer

    # PHP stack traces
    - "#0 "               # PHP stack frame (e.g., "#0 /path/to/file.php(10): function()")
    - "#1 "               # PHP stack frame
    - "#2 "               # PHP stack frame
    - "#3 "               # PHP stack frame
    - "#4 "               # PHP stack frame
    - "#5 "               # PHP stack frame
    - "  thrown in "      # PHP exception location

httpCode

Configure the HTTP Status Code Extractor to enrich TenXObjects with an extracted HTTP status code.

Below is the default configuration from: httpCode/config.yaml.

Edit Online

Edit HTTP Status Code Extractor Config Locally 

# 🔟❎ 'run' HTTP code classifier configuration

# The HTTP code classifier enriches TenXObjects with an extracted HTTP numeric response code (e.g., 404) field.

# Set the 10x pipeline to 'run'
tenx: run

# =============================== Dependencies ================================

include: run/modules/initialize/httpCode

# =============================== GeoIP Options ===============================

httpCode:

  # 'field' specifies the name of the field in which to assign an extracted HTTP status code, if found
  field: http_code

  # 'precludersBefore' specifies an array of characters or patterns that, when appearing immediately before a candidate HTTP status code in tokenized logs, indicate it is not a valid HTTP code.
  #  Used to preclude false positives like negative numbers ('-200'), floats ('.200'), or variables ('_200').
  #  Entries are single characters or short strings for O(1) lookup efficiency in the heuristic.
  precludersBefore: ['+', '$', '#', '@', '(', '[', '~', '-', '.', ':']

  # 'precludersAfter' specifies an array of characters or patterns that, when appearing immediately after a candidate HTTP status code in tokenized logs, indicate it is not a valid HTTP code.
  #  Used to preclude false positives like percentages ('200%'), floats ('200.'), or expressions ('200+').
  #  Entries are single characters or short strings for O(1) lookup efficiency in the heuristic.
  precludersAfter: ['%', ')', ']', '=', '!', '?', '+', '-', '.']

  # 'keywords' specifies an array of keywords indicating HTTP-related context in log events.
  #  Used to filter logs likely containing HTTP status codes, reducing false positives.
  #  Keywords cover HTTP methods, common log fields, and format-specific terms from sources like Apache, Nginx, HAProxy, Squid, IIS, Spring Boot.
  keywords:
    - HTTP/             # Indicates HTTP protocol version in request lines (e.g., "GET / HTTP/1.1").
    - GET               # HTTP method for retrieving resources.
    - POST              # HTTP method for submitting data.
    - PUT               # HTTP method for updating resources.
    - DELETE            # HTTP method for deleting resources.
    - HEAD              # HTTP method for retrieving headers only.
    - OPTIONS           # HTTP method for describing communication options.
    - TRACE             # HTTP method for diagnostic purposes.
    - PATCH             # HTTP method for partial updates.
    - request           # Common in log fields referring to HTTP requests (e.g., "requestMethod").
    - Request           # Title case variant for HTTP requests
    - response          # Common in log fields referring to HTTP responses (e.g., "responseSize").
    - Response          # Title case variant for HTTP responses
    - status            # Common key for HTTP status codes (e.g., "status":200).
    - Status            # Title case variant for HTTP status
    - code              # Shorthand for status code (e.g., "code":404).
    - Code              # Title case variant for status code
    - upstream_status   # Nginx-specific for backend status codes.
    - Upstream_status   # Title case variant for Nginx backend status
    - Upstream_Status   # Title case variant for Nginx backend status
    - http_status       # JSON key for HTTP status (e.g., "http_status":500).
    - Http_status       # Title case variant for JSON key for HTTP status
    - Http_Status       # Title case variant for JSON key for HTTP status
    - status_code       # Alternative key for status codes (e.g., "status_code":429).
    - statusCode        # Camel case variant for status codes (e.g., "statusCode":200)
    - Completed         # Spring Boot DispatcherServlet log indicator (e.g., "Completed 200 OK").
    - access            # Indicates access logs (e.g., Apache access log).
    - Access            # Title case variant for access logs
    - error             # Indicates error logs containing HTTP errors.
    - Error             # Title case variant for error logs
    - method            # Refers to HTTP request method (e.g., "method":"GET").
    - Method            # Title case variant for HTTP method
    - path              # Refers to request path or URI (e.g., "path":"/api").
    - Path              # Title case variant for request path
    - url               # Refers to request URL (e.g., "url":"http://example.com").
    - Url               # Title case variant for request URL
    - URL               # Title case variant for request URL
    - referrer          # HTTP referrer header field.
    - referrer          # Title case variant for referrer header field.
    - Referer           # Alternative spelling for HTTP referrer
    - Referer           # Title case variant for alternative spelling
    - user-agent        # HTTP user-agent header field.
    - User-agent        # Title case variant for user-agent
    - User-Agent        # Title case variant for user-agent
    - latency           # Common in logs for request/response time (e.g., HAProxy, ELB).
    - Latency           # Title case variant for latency
    - frontend          # HAProxy term for incoming connections.
    - Frontend          # Title case variant for HAProxy frontend
    - backend           # HAProxy term for upstream servers.
    - Backend           # Title case variant for HAProxy backend
    - TCP_MISS          # Squid hierarchy code prefix (e.g., "TCP_MISS/200").
    - W3SVC             # IIS log prefix (e.g., "W3SVC1").
    - DispatcherServlet # Spring Boot servlet class for HTTP handling.

  # 'validValues' specifies an array of all standard HTTP status codes as defined by the IANA HTTP Status Code Registry and relevant RFCs (e.g., RFC 9110, RFC 6585, RFC 7231-7235, RFC 7538, RFC 7725, RFC 8297, RFC 9111).
  #  Used to validate potential HTTP status codes in log events, ensuring only recognized codes are extracted.
  #  Each entry includes the numeric code followed by a comment describing its meaning and purpose.
  validValues:
    - '200' # OK: Standard response for successful HTTP requests.
    - '304' # Not Modified: Resource has not been modified since last requested.
    - '404' # Not Found: Server cannot find the requested resource.
    - '301' # Moved Permanently: Resource has been moved permanently to a new URI.
    - '302' # Found: Resource temporarily located at a different URI.
    - '403' # Forbidden: Client does not have access rights to the content.
    - '400' # Bad Request: Server cannot process due to client error.
    - '500' # Internal Server Error: Generic server error.
    - '206' # Partial Content: Server is delivering only part of the resource due to a range header.
    - '503' # Service Unavailable: Server temporarily unable to handle request.
    - '401' # Unauthorized: Authentication required and has failed or not provided.
    - '204' # No Content: Server processed request but no content is returned.
    - '502' # Bad Gateway: Server received an invalid response from upstream.
    - '201' # Created: Request has been fulfilled, resulting in new resource creation.
    - '429' # Too Many Requests: Client has sent too many requests in a given time.
    - '308' # Permanent Redirect: Resource permanently at another URI, method unchanged.
    - '307' # Temporary Redirect: Resource temporarily at another URI, method unchanged.
    - '303' # See Other: Response can be found under a different URI using GET.
    - '405' # Method Not Allowed: Request method is not supported for the resource.
    - '406' # Not Acceptable: Server cannot produce a response matching client’s Accept headers.
    - '408' # Request Timeout: Server timed out waiting for the request.
    - '409' # Conflict: Request conflicts with current state of the resource.
    - '410' # Gone: Resource is permanently unavailable.
    - '413' # Content Too Large: Request entity is larger than server limits.
    - '414' # URI Too Long: Request-URI is longer than the server can handle.
    - '415' # Unsupported Media Type: Media format of the requested data is not supported.
    - '416' # Range Not Satisfiable: Requested range cannot be fulfilled.
    - '422' # Unprocessable Content: Request is well-formed but semantically incorrect.
    - '504' # Gateway Timeout: Server, as gateway, did not get timely upstream response.
    - '505' # HTTP Version Not Supported: Server does not support the HTTP version.
    - '100' # Continue: Client should continue with request.
    - '101' # Switching Protocols: Server is switching protocols as requested.
    - '202' # Accepted: Request accepted for processing, but processing is not complete.
    - '203' # Non-Authoritative Information: Server is a transforming proxy with non-authoritative metadata.
    - '205' # Reset Content: Client should reset the document view.
    - '300' # Multiple Choices: Indicates multiple options for the resource.
    - '411' # Length Required: Content-Length header is required but not provided.
    - '412' # Precondition Failed: Server does not meet one of the preconditions.
    - '417' # Expectation Failed: Server cannot meet Expect header requirements.
    - '421' # Misdirected Request: Request sent to server unable to produce a response.
    - '423' # Locked: Resource is locked (WebDAV).
    - '424' # Failed Dependency: Request failed due to failure of a previous request.
    - '425' # Too Early: Server unwilling to risk processing a request that might be replayed.
    - '426' # Upgrade Required: Client should switch to a different protocol.
    - '428' # Precondition Required: Server requires conditional request headers.
    - '431' # Request Header Fields Too Large: Headers exceed server limits.
    - '451' # Unavailable For Legal Reasons: Resource access blocked for legal reasons.
    - '501' # Not Implemented: Server does not support the requested functionality.
    - '506' # Variant Also Negotiates: Server configuration error in content negotiation.
    - '507' # Insufficient Storage: Server cannot store the representation (WebDAV).
    - '508' # Loop Detected: Server detected an infinite loop in request processing.
    - '510' # Not Extended: Further extensions required for request fulfillment.
    - '511' # Network Authentication Required: Client needs to authenticate for network access.
    - '102' # Processing: Server has received and is processing the request.
    - '103' # Early Hints: Used to return some response headers before final HTTP message.
    - '207' # Multi-Status: Conveys information about multiple resources in WebDAV.
    - '208' # Already Reported: Used inside a DAV binding to avoid enumerating bindings repeatedly.
    - '226' # IM Used: Server has fulfilled a request for the resource using instance-manipulations.
    - '305' # Use Proxy: Resource is available only through a proxy.
    - '306' # (Unused): Previously used for Switch Proxy, now reserved.
    - '402' # Payment Required: Reserved for future use, e.g., digital payments.
    - '407' # Proxy Authentication Required: Client must authenticate with proxy.
    - '418' # (Unused): Previously “I’m a teapot” (RFC 2324, joke).

lookup

Configure the File Lookup to enrich TenXObjects with lookup table values.

Below is the default configuration from: lookup/config.yaml.

Edit Online

Edit File Lookup Config Locally 

# 🔟❎ 'run' lookup file configuration

# Configuration text lookup to enrich TenXObjects
# To learn more see https://doc.log10x.com/run/initialize/lookup/

# Set the 10x pipeline to 'run'
tenx: run

# =============================== Dependencies ================================

include: run/modules/initialize/lookup

# ============================== Lookup Options ===============================

lookup:

  # 'file' specifies the .csv/.tsv lookup table file
  file: $=path("pipelines/run/initialize/lookup/http.csv")

  # 'keyField' specifies the field whose value is query lookup key.
  #  The calculated code field below captures the penultimate HTTP code value from a web log event.
  #  For example, 200 for: Mar 10 12:34:56 webserver nginx: 192.168.1.10 - - [10/Mar/2025:12:34:56 +0000] "GET /index.html HTTP/1.1" 200 1024
  keyField: $=yield TenXEnv.get("httpCodeField")

  # 'valueField' specifies the name of the result field to assign to the lookup table value
  valueField: http_message

k8s

Configure the k8s Context Extractor to enrich TenXObjects with k8s pod and container information.

Below is the default configuration from: k8s/config.yaml.

Edit Online

Edit k8s Context Extractor Config Locally 

# 🔟❎ 'run' k8s lookup configuration

# Configure a k8s field extractors to enrich TenXObjects
# To learn more see https://doc.log10x.com/run/initialize/k8s/

# Set the 10x pipeline to 'run'
tenx: run

# =============================== Dependencies ================================

include: run/modules/initialize/k8s

# ============================== Symbol Options ===============================

k8s:

  # 'extractorName' specifies which extractor (e.g., fluentK8s/filebeatK8s) to use for k8s metadata extraction
  extractorName: fluentK8s # extract k8s context using Fluent schema

  # 'namespaceNameField' specifies the name of the field in which to assign an extracted k8 namespace name status code, if found
  namespaceNameField: k8s_namespace

  # 'containerNameField' specifies the name of the field in which to assign an extracted k8 container name status code, if found
  containerNameField: k8s_container

  # 'podNameField' specifies the name of the field in which to assign an extracted k8 pod name status code, if found
  podNameField: k8s_pod

geoIP

Configure the GeoIP lookup to geo-reference TenXObjects via their ipAddress value.

Below is the default configuration from: geoIP/config.yaml.

Edit Online

Edit GeoIP lookup Config Locally 

# 🔟❎ 'run' GeoIP lookup configuration

# The GeoIP lookup geo-references TenXObjects based on their ipAddress field value.
# To learn more see https://doc.log10x.com/api/js/#TenXObject+ipAddress

# Set the 10x pipeline to 'run'
tenx: run

# =============================== Dependencies ================================

include: run/modules/initialize/geoIP

# =============================== GeoIP Options ===============================

geoIP:

  # 'file' specifies the MaxMind GeoIP DB (.mmdb) file to load.
  # To learn more see https://doc.log10x.com/api/js/#TenXLookup.loadGeoIPDB
  file: data/run/lookup/geo.mmdb

  # 'countryField' specifies the field name for country geo-reference data.
  # Defaults to "country". Set to "" to disable country enrichment.
  countryField: country

  # Uncomment the following fields to enable additional geo-reference data as needed:
  # continentField: continent
  # subdivisionField: subdivision
  # cityField: city
  # postalField: postal
  # latitudeField: latitude
  # longitudeField: longitude
Metric Outputs

Metric Outputs

Activate a Metric output to write aggregated TenXSummary instances to metric outputs (e.g., Prometheus, Datadog).

log10x

Configure log10x managed Prometheus outputs to publish TenXSummary metrics to the log10x Prometheus backend.

Below is the default configuration from: log10x/config.yaml.

Edit Online

Edit log10x managed Prometheus output Config Locally 

# 🔟❎ 'run' 10x-hosted metrics backend output configuration

# Configuration 10x-hosted metrics backend counter outputs.
# To learn more see https://doc.log10x.com/run/output/metric/log10x/

# Set the 10x pipeline to 'run'
tenx: run

# =============================== Dependencies ================================

include: run/modules/output/metric/log10x

# =========================== Log10x Metrics output ===========================

# Define multiple log10x metric outputs below
log10xMetrics:

    # ---------------------------- Counter Options ----------------------------

    # 'nameField' sets the TenXSummary field whose value identifies the micrometer counter to increment.
    #  The 'inputName' field returns the name of the aggregator that produced the target TenXSummary instance.
    #  To learn more see https://doc.log10x.com/run/aggregate
  - nameField: inputName

    # 'counterFields' specifies the TenXSummary fields whose values are used to increment the counter.
    #  To learn more see https://doc.log10x.com/run/aggregate/#summary-fields
    counterFields:
      - summaryVolume
      - summaryBytes
      - summaryTotals

    # 'tagFields' sets the TenXSummary fields used as the counter tag dimension values.
    #  Defaults to https://doc.log10x.com/run/initialize/#enrichmentFields
    tagFields:
      - $=yield TenXEnv.get("enrichmentFields")

    # 'tagFieldNames' allows setting custom names to aggregated fields for metrics reporting.
    #  Defaults to https://doc.log10x.com/run/initialize/#metricFieldNames
    tagFieldNames:
      - $=yield TenXEnv.get("metricFieldNames")

    # 'filter' sets a JavaScript expression that TenXSummaries must evaluate as truthy to write to the output.
    #  To learn more see https://doc.log10x.com/run/output/regulate/#filter-expressions
    filter: isSummary

scrape

Configure Prometheus scrape metrics outputs to publish TenXSummary metrics to Prometheus.

Below is the default configuration from: scrape/config.yaml.

Edit Online

Edit Prometheus scrape metrics output Config Locally 

# 🔟❎ 'run' Prometheus metric output configuration

# Configure Prometheus metric counter outputs.
# To learn more see https://doc.log10x.com/run/output/metric/prometheus/

# Set the 10x pipeline to 'run'
tenx: run

# =============================== Dependencies ================================

include: run/modules/output/metric/prometheus/scrape

# ============================== Prometheus Outputs ===========================

# Define multiple Prometheus metric outputs below
prometheus:

    # --------------------------- Connection Options --------------------------

    # 'port' sets the port number to listen to the Prometheus scraper on
  - port: 9100

    # ---------------------------- Counter Options ----------------------------

    # 'nameField' sets the TenXSummary field whose value identifies the micrometer counter to increment. 
    #  The 'inputName' field returns the name of the aggregator that produced the target TenXSummary instance.
    #  To learn more see https://doc.log10x.com/run/aggregate
    nameField: inputName

    # 'counterFields' specifies the TenXSummary fields whose values are used to increment the counter.
    #  To learn more see https://doc.log10x.com/run/aggregate/#summary-fields
    counterFields:
      - summaryVolume
      - summaryBytes
      - summaryTotals

    # 'tagFields' sets the TenXSummary fields used as the counter tag dimension values.
    #  Defaults to https://doc.log10x.com/run/initialize/#enrichmentFields
    tagFields:
      - $=yield TenXEnv.get("enrichmentFields")

    # 'tagFieldNames' allows setting custom names to aggregated fields for metrics reporting.
    #  Defaults to https://doc.log10x.com/run/initialize/#metricFieldNames
    tagFieldNames:
      - $=yield TenXEnv.get("metricFieldNames")

    # 'filter' sets a JavaScript expression that TenXSummaries must evaluate as truthy to write to the output.
    #  To learn more see https://doc.log10x.com/run/output/regulate/#filter-expressions
    filter: isSummary

push-gateway

Configure Prometheus Push gateway metrics outputs to publish TenXSummary metrics to Prometheus via a Push gateway.

Below is the default configuration from: push-gateway/config.yaml (* Required Fields).

Edit Online

Edit Prometheus Push gateway metrics output Config Locally 

# 🔟❎ 'run' Prometheus push gateway metric output configuration

# Configures Prometheus push gateway metric counter outputs.
# To learn more see https://doc.log10x.com/run/output/metric/prometheus/push-gateway/
# For the Prometheus push gateway see: https://prometheus.io/docs/practices/pushing/

# Set the 10x pipeline to 'run'
tenx: run

# =============================== Dependencies ================================

include: run/modules/output/metric/prometheus/push-gateway

# ========================== Prometheus PG Options ============================

# Define multiple Prometheus push gateway metric outputs below
prometheusPG:

    # --------------------------- Connection Options --------------------------

    # 'host' sets the address and port number of the push gateway
  - host: localhost:9091 # (❗ REQUIRED)

    # 'job' sets the logical job name reported to the push gateway
    job: my-10x-job

    # ---------------------------- Counter Options ----------------------------

    # 'nameField' sets the TenXSummary field whose value identifies the micrometer counter to increment. 
    #  The 'inputName' field returns the name of the aggregator that produced the target TenXSummary instance.
    #  To learn more see https://doc.log10x.com/run/aggregate
    nameField: inputName

    # 'counterFields' specifies the TenXSummary fields whose values are used to increment the counter.
    #  To learn more see https://doc.log10x.com/run/aggregate/#summary-fields
    counterFields:
      - summaryVolume
      - summaryBytes
      - summaryTotals

    # 'tagFields' sets the TenXSummary fields used as the counter tag dimension values.
    #  Defaults to https://doc.log10x.com/run/initialize/#enrichmentFields
    tagFields:
      - $=yield TenXEnv.get("enrichmentFields")

    # 'tagFieldNames' allows setting custom names to aggregated fields for metrics reporting.
    #  Defaults to https://doc.log10x.com/run/initialize/#metricFieldNames
    tagFieldNames:
      - $=yield TenXEnv.get("metricFieldNames")

    # 'filter' sets a JavaScript expression that TenXSummaries must evaluate as truthy to write to the output.
    #  To learn more see https://doc.log10x.com/run/output/regulate/#filter-expressions
    filter: isSummary

    # ---------------------------- Publish Options ----------------------------

    # 'step' sets the interval to send metrics Prometheus PG (e.g., '1m').
    #  To learn more see https://micrometer.io/docs/concepts#rate-aggregation
    step: 5s

remote-write

Configure Prometheus RW metrics outputs to publish TenXSummary metrics to Prometheus via the Remote-Write protocol.

Below is the default configuration from: remote-write/config.yaml (* Required Fields).

Edit Online

Edit Prometheus RW metrics output Config Locally 

# 🔟❎ 'run' Prometheus remote write metric output configuration

# Configure Prometheus remote write metric counter outputs.
# To learn more see https://doc.log10x.com/run/output/metric/prometheus/remote-write/
# For Prometheus RW, see https://prometheus.io/docs/concepts/remote_write_spec/

# Set the 10x pipeline to 'run'
tenx: run

# =============================== Dependencies ================================

include: run/modules/output/metric/prometheus/remote-write

# ========================== Prometheus RW Options ============================

# Define multiple Prometheus RW metric outputs below
prometheusRW:

    # --------------------------- Connection Options --------------------------

    # 'host' sets the Prometheus remote write endpoint
  - host: http://localhost:9090/api/v1/write # (❗ REQUIRED)

    # 'user' for authentication
    user: null # (❗ REQUIRED)

    # 'password' for authentication
    password: null # (❗ REQUIRED)

    # ---------------------------- Counter Options ----------------------------

    # 'nameField' sets the TenXSummary field whose value identifies the micrometer counter to increment. 
    #  The 'inputName' field returns the name of the aggregator that produced the target TenXSummary instance.
    #  To learn more see https://doc.log10x.com/run/aggregate
    nameField: inputName

    # 'counterFields' specifies the TenXSummary fields whose values are used to increment the counter.
    #  To learn more see https://doc.log10x.com/run/aggregate/#summary-fields
    counterFields:
      - summaryVolume
      - summaryBytes
      - summaryTotals

    # 'tagFields' sets the TenXSummary fields used as the counter tag dimension values.
    #  Defaults to https://doc.log10x.com/run/initialize/#enrichmentFields
    tagFields:
      - $=yield TenXEnv.get("enrichmentFields")

    # 'tagFieldNames' allows setting custom names to aggregated fields for metrics reporting.
    #  Defaults to https://doc.log10x.com/run/initialize/#metricFieldNames
    tagFieldNames:
      - $=yield TenXEnv.get("metricFieldNames")

    # 'filter' sets a JavaScript expression that TenXSummaries must evaluate as truthy to write to the output.
    #  To learn more see https://doc.log10x.com/run/output/regulate/#filter-expressions
    filter: isSummary

    # ---------------------------- Publish Options ----------------------------

    # 'step' sets the interval to send metrics to Prometheus RW (e.g. '1m').
    #  To learn more see https://micrometer.io/docs/concepts#rate-aggregation
    step: 5s

datadog

Configure Datadog metrics outputs to publish TenXSummary metrics to Datadog.

Below is the default configuration from: datadog/config.yaml (* Required Fields).

Edit Online

Edit Datadog metrics output Config Locally 

# 🔟❎ 'run' Datadog metric output configuration

# Configure Datadog metric counter outputs.
# To learn more see https://doc.log10x.com/run/output/metric/datadog/

# Set the 10x pipeline to 'run'
tenx: run

# =============================== Dependencies ================================

include: run/modules/output/metric/datadog

# ============================== Datadog Options ==============================

# Define multiple Datadog metric outputs below
datadog:

    # --------------------------- Connection Options --------------------------

    # 'apiKey' and 'applicationKey' set the Datadog API and app keys (required).
    #  To learn more see: https://docs.datadoghq.com/account_management/api-app-keys/
  - apiKey: $=TenXEnv.get("DD_API_KEY")         # (❗ EnvVar REQUIRED)

    applicationKey: $=TenXEnv.get("DD_APP_KEY") # (❗ EnvVar REQUIRED)

    # 'uri' sets the Datadog site address to which to connect.
    #  To learn more see: https://docs.datadoghq.com/getting_started/site/
    uri: https://us5.datadoghq.com # (or https://api.us3.datadoghq.com)

    # ---------------------------- Publish Options ----------------------------

    # 'descriptions' sets whether to send meter descriptions to Datadog
    #  Turn this off to minimize the amount of data sent on each write
    descriptions: false

    # 'step' sets the interval to send metrics to CloudWatch (e.g., '1m').
    #  To learn more see: https://micrometer.io/docs/concepts#rate-aggregation
    step: 5s

    # ---------------------------- Counter Options ----------------------------

    # 'nameField' sets the TenXSummary field whose value identifies the micrometer counter to increment. 
    #  The 'inputName' field returns the name of the aggregator that produced the target TenXSummary instance.
    #  To learn more see https://doc.log10x.com/run/aggregate
    nameField: inputName

    # 'counterFields' specifies the TenXSummary fields whose values are used to increment the counter.
    #  To learn more see https://doc.log10x.com/run/aggregate/#summary-fields
    counterFields:
      - summaryVolume
      - summaryBytes
      - summaryTotals

    # 'tagFields' sets the TenXSummary fields used as the counter tag dimension values.
    #  Defaults to https://doc.log10x.com/run/initialize/#enrichmentFields
    tagFields:
      - $=yield TenXEnv.get("enrichmentFields")

    # 'tagFieldNames' allows setting custom names to aggregated fields for metrics reporting.
    #  Defaults to https://doc.log10x.com/run/initialize/#metricFieldNames
    tagFieldNames:
      - $=yield TenXEnv.get("metricFieldNames")

    # 'filter' sets a JavaScript expression that TenXSummaries must evaluate as truthy to write to the output.
    #  To learn more see https://doc.log10x.com/run/output/regulate/#filter-expressions
    filter: isSummary

cloudwatch

Configure AWS CloudWatch metrics outputs to publish TenXSummary metrics to AWS CloudWatch metrics.

Below is the default configuration from: cloudwatch/config.yaml (* Required Fields).

Edit Online

Edit AWS CloudWatch metrics output Config Locally 

# 🔟❎ 'run' AWS CloudWatch metric output configuration

# Configure AWS CloudWatch metric counter outputs.
# To learn more see https://doc.log10x.com/run/output/metric/cloudwatch/

# ============================== Dependencies =================================

# Set the 10x pipeline to 'run'
tenx: run

# =============================== Dependencies ================================

include: run/modules/output/metric/cloudwatch

# ============================ AWS CloudWatch Options =========================

# Define multiple AWS CloudWatch metric outputs below

cloudwatch:

    # -------------------------- Connection Options ---------------------------

    # 'awsAccessKeyId' defines the AWS access key (required)
    #  To learn more see https://docs.aws.amazon.com/general/latest/gr/aws-sec-cred-types.html
  - awsAccessKeyId: $=TenXEnv.get("AWS_ACCESS_KEY_ID") # (❗ EnvVar REQUIRED)

    # 'awsSecretKey' defines the target AWS secret key (required)
    #  To learn more see https://docs.aws.amazon.com/general/latest/gr/aws-sec-cred-types.html
    awsSecretKey: $=TenXEnv.get("AWS_SECRET_ACCESS_KEY") # (❗ EnvVar REQUIRED)

    # 'namespace' defines the target AWS CloudWatch namespace (required)
    #  To learn more see https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/cloudwatch_concepts.html#Namespace
    namespace: null # (❗ REQUIRED)

    # ---------------------------- Counter Options ----------------------------

    # 'nameField' sets the TenXSummary field whose value identifies the micrometer counter to increment. 
    #  The 'inputName' field returns the name of the aggregator that produced the target TenXSummary instance.
    #  To learn more see https://doc.log10x.com/run/aggregate
    nameField: inputName

    # 'counterFields' specifies the TenXSummary fields whose values are used to increment the counter.
    #  To learn more see https://doc.log10x.com/run/aggregate/#summary-fields
    counterFields:
      - summaryVolume
      - summaryBytes
      - summaryTotals

    # 'tagFields' sets the TenXSummary fields used as the counter tag dimension values.
    #  Defaults to https://doc.log10x.com/run/initialize/#enrichmentFields
    tagFields:
      - $=yield TenXEnv.get("enrichmentFields")

    # 'tagFieldNames' allows setting custom names to aggregated fields for metrics reporting.
    #  Defaults to https://doc.log10x.com/run/initialize/#metricFieldNames
    tagFieldNames:
      - $=yield TenXEnv.get("metricFieldNames")

    # 'filter' sets a JavaScript expression that TenXSummaries must evaluate as truthy to write to the output.
    #  To learn more see https://doc.log10x.com/run/output/regulate/#filter-expressions
    filter: isSummary

    # 'step' sets the interval to send metrics to CloudWatch (e.g., '1m').
    #  To learn more see https://micrometer.io/docs/concepts#rate-aggregation
    step: 5s

elastic

Configure ElasticSearch metrics outputs to publish TenXSummary metrics to ElasticSearch.

Below is the default configuration from: elastic/config.yaml (* Required Fields).

Edit Online

Edit ElasticSearch metrics output Config Locally 

# 🔟❎ 'run' ElasticSearch metric output configuration

# Configure ElasticSearch metric counter outputs.
# To learn more see https://doc.log10x.com/run/output/metric/elastic/

# Set the 10x pipeline to 'run'
tenx: run

# =============================== Dependencies ================================

include: run/modules/output/metric/elastic

# =============================== Elastic Options =============================

# Define multiple Elastic metric outputs below

elastic:

    # --------------------------- Connection Options --------------------------

    # 'host' sets the host to send metrics to (e.g., https://<deployment-name>.us-central1.gcp.cloud.es.io).
  - host: "" # (❗ REQUIRED)

    # 'apiKeyCredentials' sets base64-encoded credentials string
    #  To learn more see: https://www.elastic.co/guide/en/elasticsearch/reference/current/security-api-create-api-key.html
    apiKeyCredentials: $=TenXEnv.get("ELASTIC_API_KEY")  # (❗ EnvVar REQUIRED)

    # 'userName' sets the the Basic Authentication username if 'apiKeyCredentials' is not set.
    userName: $=TenXEnv.get("ELASTIC_USERNAME")  # (❗ EnvVar REQUIRED)

    # 'password' sets a Basic Authentication password.
    password: $=TenXEnv.get("ELASTIC_PASSWORD")  # (❗ EnvVar REQUIRED)

    # ---------------------------- Counter Options ----------------------------

    # 'nameField' sets the TenXSummary field whose value identifies the micrometer counter to increment. 
    #  The 'inputName' field returns the name of the aggregator that produced the target TenXSummary instance.
    #  To learn more see https://doc.log10x.com/run/aggregate
    nameField: inputName

    # 'counterFields' specifies the TenXSummary fields whose values are used to increment the counter.
    #  To learn more see https://doc.log10x.com/run/aggregate/#summary-fields
    counterFields:
      - summaryVolume
      - summaryBytes
      - summaryTotals

    # 'tagFields' sets the TenXSummary fields used as the counter tag dimension values.
    #  Defaults to https://doc.log10x.com/run/initialize/#enrichmentFields
    tagFields:
      - $=yield TenXEnv.get("enrichmentFields")

    # 'tagFieldNames' allows setting custom names to aggregated fields for metrics reporting.
    #  Defaults to https://doc.log10x.com/run/initialize/#metricFieldNames
    tagFieldNames:
      - $=yield TenXEnv.get("metricFieldNames")

    # 'filter' sets a JavaScript expression that TenXSummaries must evaluate as truthy to write to the output.
    #  To learn more see https://doc.log10x.com/run/output/regulate/#filter-expressions
    filter: isSummary

    # ---------------------------- Publish Options ----------------------------

    # 'step' sets the interval for sending metrics to Elastic (e.g., '1m').
    #  To learn more see: https://micrometer.io/docs/concepts#rate-aggregation
    step: 5s

    # ----------------------------- Index Options -----------------------------

    # 'elasticIndex' sets the index name to which to write metrics.
    index: tenx-metrics

    # 'autoCreateIndex' sets whether to create the index automatically if it doesn't exist.
    autoCreateIndex: true

    # 'indexDateFormat' sets the index date format used for rolling indices.
    #  This is appended to the index name, separated by 'elasticIndexDateSeparator'.
    indexDateFormat: 'yyyy-MM'

    # 'indexDateFormat' sets the separator between the index name and the date part
    indexDateSeparator: '-'

    # 'pipeline' sets the ingest pipeline name
    pipeline: null

    # 'timestampFieldName' sets the name of the elastic timestamp field.
    timestampFieldName: '@timestamp'

signalFx

Configure SignalFx metrics outputs to publish TenXSummary metrics to Splunk SignalFx.

Below is the default configuration from: signalFx/config.yaml (* Required Fields).

Edit Online

Edit SignalFx metrics output Config Locally 

# 🔟❎ 'run' SignalFx metric output configuration

# Configures SignalFx remote write metric counter outputs.
# To learn more see https://doc.log10x.com/run/output/metric/signalFx/

# Set the 10x pipeline to 'run'
tenx: run

# =============================== Dependencies ================================

include: run/modules/output/metric/signalFx

# ============================ SignalFx Options ===============================

# Define multiple SignalFx metric outputs below
signalfx:

    # --------------------------- Connection Options --------------------------

    # 'accessToken' sets the SignalFx access token.
    #  To learn more see https://dev.splunk.com/observability/reference/api/org_tokens/latest
  - accessToken: $=TenXEnv.get("SIGNALFX_ACCESS_TOKEN")  # (❗ EnvVar REQUIRED)

    # 'source' sets the value to map to "host" when shipping metrics to SignalFx.
    #  Defaults to local hostname
    source: null

    # 'sourceUri' sets the URI of an internal proxy en route to signalFx.
    sourceUri: https://ingest.signalfx.com

    # ---------------------------- Counter Options ----------------------------

    # 'nameField' sets the TenXSummary field whose value identifies the micrometer counter to increment. 
    #  The 'inputName' field returns the name of the aggregator that produced the target TenXSummary instance.
    #  To learn more see https://doc.log10x.com/run/aggregate
    nameField: inputName

    # 'counterFields' specifies the TenXSummary fields whose values are used to increment the counter.
    #  To learn more see https://doc.log10x.com/run/aggregate/#summary-fields
    counterFields:
      - summaryVolume
      - summaryBytes
      - summaryTotals

    # 'tagFields' sets the TenXSummary fields used as the counter tag dimension values.
    #  Defaults to https://doc.log10x.com/run/initialize/#enrichmentFields
    tagFields:
      - $=yield TenXEnv.get("enrichmentFields")

    # 'tagFieldNames' allows setting custom names to aggregated fields for metrics reporting.
    #  Defaults to https://doc.log10x.com/run/initialize/#metricFieldNames
    tagFieldNames:
      - $=yield TenXEnv.get("metricFieldNames")

    # 'filter' sets a JavaScript expression that TenXSummaries must evaluate as truthy to write to the output.
    #  To learn more see https://doc.log10x.com/run/output/regulate/#filter-expressions
    filter: isSummary

    # ---------------------------- Publish Options ----------------------------

    # 'step' sets the interval to send metrics to SignalFx (e.g., '1m').
    #  To learn more see https://micrometer.io/docs/concepts#rate-aggregation

    # 'publishCumulativeHistogram' sets whether the SignalFx registry should emit cumulative histogram
    publishCumulativeHistogram: false

log

Configure Log4j2 metrics outputs to publish TenXSummary metrics to a log4j2 appender in Micrometer 'simple' format.

Below is the default configuration from: log/config.yaml.

Edit Online

Edit Log4j2 metrics output Config Locally 

# 🔟❎ 'run' Log4j2 metric output configuration

# Configure log4j2 metric counter outputs.
# To learn more see https://doc.log10x.com/run/output/metric/log/

# Set the 10x pipeline to 'run'
tenx: run

# =============================== Dependencies ================================

include: run/modules/output/metric/log

# ============================== Log4j2 Options ===============================

# Define multiple log4j2 metric outputs below
simple:

    # 'loggerName' sets the log4j2 logger to which registry metric data is periodically logged
    #  metric are logged a 'INFO' messages using micrometer's default 'simple' format.
    #  To learn more see: https://github.com/micrometer-metrics/micrometer/blob/main/micrometer-core/src/main/java/io/micrometer/core/instrument/simple/SimpleMeterRegistry.java
  - loggerName: consoleOut

    # ---------------------------- Counter Options ----------------------------

    # 'nameField' sets the TenXSummary field whose value identifies the micrometer counter to increment. 
    #  The 'inputName' field returns the name of the aggregator that produced the target TenXSummary instance.
    #  To learn more see https://doc.log10x.com/run/aggregate
    nameField: inputName

    # 'counterFields' specifies the TenXSummary fields whose values are used to increment the counter.
    #  To learn more see https://doc.log10x.com/run/aggregate/#summary-fields
    counterFields:
      - summaryVolume
      - summaryBytes
      - summaryTotals

    # 'tagFields' sets the TenXSummary fields used as the counter tag dimension values.
    #  Defaults to https://doc.log10x.com/run/initialize/#enrichmentFields
    tagFields:
      - $=yield TenXEnv.get("enrichmentFields")

    # 'tagFieldNames' allows setting custom names to aggregated fields for metrics reporting.
    #  Defaults to https://doc.log10x.com/run/initialize/#metricFieldNames
    tagFieldNames:
      - $=yield TenXEnv.get("metricFieldNames")

    # 'filter' sets a JavaScript expression that TenXSummaries must evaluate as truthy to write to the output.
    #  To learn more see https://doc.log10x.com/run/output/regulate/#filter-expressions
    filter: isSummary

    # ---------------------------- Publish Options ----------------------------

    # 'step' sets the frequency in which metrics are logged to 'loggerName' in
    # ISO-8601 format. To learn more see:
    # https://docs.oracle.com/javase/8/docs/api/java/time/Duration.html#parse-java.lang.CharSequence-
    step: PT1S

Advanced Settings

To configure advanced options (optional) for the Cloud Reporter app, Edit these settings:

Bootstrap

Bootstrap

Configure the Pipeline Bootstrapper to authenticate the log10x account and launch a target pipeline.

Below is the default configuration from: bootstrap/config.yaml.

Edit Online

Edit Pipeline Bootstrapper Config Locally 

# 🔟❎ run bootstrap configuration

# This config file specifies bootstrap options for the run pipeline.
# To learn more see https://doc.log10x.com/run/bootstrap

# To learn more see https://doc.log10x.com/config/

tenx: run

# =============================== Launch Settings ==============================

# 'apiKey' specifies an api key used authenticating against the 10x service
apiKey: $=TenXEnv.get("TENX_API_KEY", "NO-API-KEY")

# 'includePaths' specifies folders on disk for resolving relative config file/folder references in addition to the working folder
includePaths: []

# 'quiet' disables printing version information to the console.
# quiet: true

# 'jarFiles' specifies .jar files to dynamically load for use by compile, input and output API extensions.
jarFiles: []

# 'metricEndpoint' specifies the Prometheus endpoint to report usage/health metrics (enterprise version only).
# metricEndpoint: https://prometheus.log10x.com/api/v1/write

# 'disabledArgs' specifies a list of launch arguments that are disallowed from either command line or user config files.
disabledArgs: []

# 'debugEnvVars' list environment variables to debug
debugEnvVars: []
Github Sync

Github Sync

Below is the default configuration from: gitops/config.yaml (* Required Fields).

Edit Online

Edit config.yaml Locally 

# 🔟❎ GitOps configuration

# This config file pulls symbol, config and lookup files from a remote repo to
# enable centralized configuration management via GitHub.

# To learn more see https://doc.log10x.com/config/github

# The target 10x pipeline to run (e.g., run, compile) is specified in the config files pulled from the repo
tenx: any

include:

  - source: github

    options:
      # 'enabled' controls whether to enable this pull config
      enabled: false # (❗ REQUIRED)

      # 'token' specifies an access token for accessing the Github repo.
      #  To learn more see https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens
      token: $=TenXEnv.get("GH_TOKEN")

      # 'repo' specifies the GitHub user/repo from which to pull
      repo: null # (❗ REQUIRED)

      # 'branch' on 'repo' (if null, uses default repo default branch)
      branch: main

       # 'paths' specifies Glob patterns within 'repo' to pull and extract to a temp folder.
      paths:
        - test
        # - config/*.yaml      # https://doc.log10x.com/config/yaml
        # - symbols/*.10x.tar  # https://doc.log10x.com/run/symbol
        # - lookups/*.csv      # https://doc.log10x.com/api/js/#TenXLookup
        # - scripts/*.js       # https://doc.log10x.com/api/js

      # 'syncInterval' specifies an interval by which check to check the remote branch for updates (null to disable)
      #  This interval works in conjunction with: https://doc.log10x.com/run/reload
      syncInterval: 5min
Symbols

Symbols

Below is the default configuration from: symbol/config.yaml.

Edit Online

Edit config.yaml Locally 

# 🔟❎ 'run' symbol file configuration

# Loads symbol library files to transform events into well-defined TenXObjects.
# To learn more see https://doc.log10x.com/run/symbol

# Set the 10x pipeline to 'run'
tenx: run

# ============================ Symbol Options =================================

symbol:

  # 'paths' specifies the file/folder locations to scan for symbol library files.
  #  To learn more see https://doc.log10x.com/run/symbol/#symbolpaths
  paths:
    - $=path("data/shared/symbols", false)
    - $=path("<TENX_SYMBOLS_PATH>",  false)

  literals: []
Aggregators

Aggregators

Configure Aggregators to aggregate and summarize TenXObjects to publish as metrics.

Below is the default configuration from: report/config.yaml.

Edit Online

Edit Aggregator Config Locally 

# 🔟❎ 'run' aggregator config for the reporter app.

# This file configures aggregators for tracking emitted (non-dropped) events.
# To learn more see https://doc.log10x.com/run/aggregate

# Set the 10x pipeline to 'run'
tenx: run

# ============================== Aggregator Options ===========================

# Multiple aggregators can be defined below
aggregator:

  # Defines an aggregator for tracking emitted (non-dropped) events
  # This enables an output metric to show an accurate (lossless) metric of the
  # data actually emitted to downstream systems.
  - name: emitted_events

    # 'filter' sets a JavaScript expression that TenXObjects must evaluate as truthy to be aggregated.
    #  To learn more see https://doc.log10x.com/run/output/regulate/#filter-expressions
    #
    # This filter excludes dropped events by checking the 'isDropped' property,
    # ensuring only emitted events are tracked.
    #
    # For more info, see https://doc.log10x.com/api/js/#TenXBaseObject+dropped
    filter: "this.isObject && !this.isDropped"

    # 'fields' set the names of intrinsic/extracted/calculated fields whose combined values
    #  identify a unique row within the aggregator's in-mem table for
    #  each object passed to it. This aggregator defaults to enrichment fields.
    #  To learn more see https://doc.log10x.com/run/initialize/#enrichmentFields
    fields:
      - $=yield TenXEnv.get("enrichmentFields")

    # 'totalFields' specifies fields to extract and tally into generated TenXSummaries
    totalFields: []

    # 'flushThreshold' sets the max number of TenXObjects whose values to upsert into
    #  the aggregator's in-mem table. Once exceeded, the aggregator instantiates TenXSummaries
    #  from the table rows to send to output and reset the table.
    flushThreshold: 10000

    # 'flushInterval' sets a periodic interval, after which the aggregator
    #  flushes the rows of its in-mem table into the pipeline as TenXSummary instances.
    flushInterval: 2s

    # 'maxCardinality' sets the maximum number of rows for the aggregator's
    #  in-mem table. If an TenXObjects whose 'values' are not contained in the table
    #  is passed to the aggregator, 'defaultValue' will be used for it instead
    #  until the aggregator flushes its in-mem table after 'flushInterval' or 'flushThreshold'
    #  are exceeded.
    maxCardinality: 65536

    # 'defaultValue' sets a constant value that is used instead of 'values' when
    #  'maxCardinality' is exceeded within 'flushInterval'
    defaultValue: unknown
Template

Template

Below is the default configuration from: template/config.yaml.

Edit Online

Edit config.yaml Locally 

# 🔟❎ 'run' template file configuration

# Load TenXTemplates .json files that define the structure/schema of TenXObjects.
# To learn more see https://doc.log10x.com/run/template/

# Set the 10x pipeline to 'run'
tenx: run

# =============================== Template ===================================

template:
  # 'files' specifies GLOB pattern for finding JSON-encoded TenXTemplates.
  files:
    - $=path("data/templates/*.json")
    - $=path("data/sample/output/*.json")

  # 'cacheSize' controls the maximum total byte size of templates held
  #  in the in-memory cache vs. on disk. Set to 0 to disable pruning.
  cacheSize: $=parseBytes("10MB")

# =============================== Variable ===================================

var:
  # 'placeholder' specifies a character to use when encoding a TenXTemplate
  #  to signify the location of a runtime variable value.
  placeholder: "$"

  # 'maxRecurIndexes' controls the maximum number of variable values to reuse.
  maxRecurIndexes: 10

# =============================== Timestamp ==================================

timestamp:
  # 'prefix' specifies a prefix for a TenXTemplate's timestamp tokens.
  prefix: (

  # 'postfix' specifies a postfix for a TenXTemplate's timestamp tokens.
  postfix: )
Transform

Transform

Configure the Transform to transform log and trace events into well-defined TenXObjects.

timestamp

Configure the Timestamp parser to extract alphanumeric and epoch timestamp values from input events.

Below is the default configuration from: timestamp/config.yaml.

Edit Online

Edit Timestamp parser Config Locally 

# 🔟❎ 'run' timestamp parser configuration

# Identify unix/alphanumeric timestamp structures within TenXTemplates.
# https://doc.log10x.com/run/transform/timestamp/

# Set the 10x pipeline to 'run'
tenx: run

# ============================= Timestamp Options =============================

timestamp:

  # 'maxPerObject' controls the max number of timestamps to add into an TenXObject's
  # 'timestamp' array. Set to 0 for unlimited.
  maxPerObject: 0

  # 'searchDirection' controls the direction(s) from with timestamps are searched for
  #  within the object's 'text' field. Possible values: 

  #  - fromStart: search for 'maxPerObject' from the start of 'text',  
  #    limiting search to 'searchLengthLimitFromStart' characters  

  #  - fromEnd: search backward for 'maxPerObject' from the end of 'text',
  #    limiting search to 'searchLengthLimitFromEnd' characters  

  #  - fromStartAndEnd: search for 'maxPerObject' from both the start and end of 'text'
  #    limiting search to 'searchLengthLimitFromStart' and 'searchLengthLimitFromEnd' respectively  

  #  - none: do not parse timestamps

  searchDirection: fromStartAndEnd

  # 'searchLengthLimitFromStart' limits the number of characters to search for
  #  timestamps from the beginning of the object's 'text' field. 0 for unlimited
  searchLengthLimitFromStart: 0

  # 'searchLengthLimitFromStart' limits the number of characters to search for
  #  timestamps from the end of the object's 'text' field. 0 for unlimited
  searchLengthLimitFromEnd: 0

  # 'zone' controls the timezone for formatting timestamp epoch
  #  values to string. The 'java.time.ZoneId.of(String zoneId)' 
  #  is used to obtain the timezone from the 'zone' value.
  #  If set to null, the host OS timezone is used. 
  zone: null

  # 'literals' contains an array of strings to treat as a part 
  #  of any timestamp candidate found when structuring an TenXObject.
  literals:
    - T
    - Z
    - I # Go INFO
    - E # Go ERROR
    - W # Go WARN

  # 'patterns' specifies an array of date-time formats to attempt when parsing timestamps from input events.
  #  Timestamp formats that appear frequently within an input stream can be 'bumped' higher within the list below.

  patterns:
  # Most common formats
  - "'I'MMdd HH:mm:ss.S"                       # Used in Kubernetes kube-apiserver logs with INFO prefix, 1-digit microsecond precision.
  - "'I'MMdd HH:mm:ss.SS"                      # Used in Kubernetes kube-apiserver logs with INFO prefix, 2-digit microsecond precision.
  - "'I'MMdd HH:mm:ss.SSS"                     # Used in Kubernetes kube-apiserver logs with INFO prefix, 3-digit microsecond precision.
  - "'I'MMdd HH:mm:ss.SSSS"                    # Used in Kubernetes kube-apiserver logs with INFO prefix, 4-digit microsecond precision.
  - "'I'MMdd HH:mm:ss.SSSSS"                   # Used in Kubernetes kube-apiserver logs with INFO prefix, 5-digit microsecond precision.
  - "'I'MMdd HH:mm:ss.SSSSSS"                  # Used in Kubernetes kube-apiserver logs with INFO prefix, 6-digit microsecond precision.
  - "'W'MMdd HH:mm:ss.S"                       # Used in Kubernetes kube-apiserver logs with WARNING prefix, 1-digit microsecond precision.
  - "'W'MMdd HH:mm:ss.SS"                      # Used in Kubernetes kube-apiserver logs with WARNING prefix, 2-digit microsecond precision.
  - "'W'MMdd HH:mm:ss.SSS"                     # Used in Kubernetes kube-apiserver logs with WARNING prefix, 3-digit microsecond precision.
  - "'W'MMdd HH:mm:ss.SSSS"                    # Used in Kubernetes kube-apiserver logs with WARNING prefix, 4-digit microsecond precision.
  - "'W'MMdd HH:mm:ss.SSSSS"                   # Used in Kubernetes kube-apiserver logs with WARNING prefix, 5-digit microsecond precision.
  - "'W'MMdd HH:mm:ss.SSSSSS"                  # Used in Kubernetes kube-apiserver logs with WARNING prefix, 6-digit microsecond precision.
  - "'E'MMdd HH:mm:ss.S"                       # Used in Kubernetes kube-apiserver logs with ERROR prefix, 1-digit microsecond precision.
  - "'E'MMdd HH:mm:ss.SS"                      # Used in Kubernetes kube-apiserver logs with ERROR prefix, 2-digit microsecond precision.
  - "'E'MMdd HH:mm:ss.SSS"                     # Used in Kubernetes kube-apiserver logs with ERROR prefix, 3-digit microsecond precision.
  - "'E'MMdd HH:mm:ss.SSSS"                    # Used in Kubernetes kube-apiserver logs with ERROR prefix, 4-digit microsecond precision.
  - "'E'MMdd HH:mm:ss.SSSSS"                   # Used in Kubernetes kube-apiserver logs with ERROR prefix, 5-digit microsecond precision.
  - "'E'MMdd HH:mm:ss.SSSSSS"                  # Used in Kubernetes kube-apiserver logs with ERROR prefix, 6-digit microsecond precision.
  - "'F'MMdd HH:mm:ss.S"                       # Used in Kubernetes kube-apiserver logs with FATAL prefix, 1-digit microsecond precision.
  - "'F'MMdd HH:mm:ss.SS"                      # Used in Kubernetes kube-apiserver logs with FATAL prefix, 2-digit microsecond precision.
  - "'F'MMdd HH:mm:ss.SSS"                     # Used in Kubernetes kube-apiserver logs with FATAL prefix, 3-digit microsecond precision.
  - "'F'MMdd HH:mm:ss.SSSS"                    # Used in Kubernetes kube-apiserver logs with FATAL prefix, 4-digit microsecond precision.
  - "'F'MMdd HH:mm:ss.SSSSS"                   # Used in Kubernetes kube-apiserver logs with FATAL prefix, 5-digit microsecond precision.
  - "'F'MMdd HH:mm:ss.SSSSSS"                  # Used in Kubernetes kube-apiserver logs with FATAL prefix, 6-digit microsecond precision.
  - "yyyy-MM-dd HH:mm:ss"                      # Widely used in Java applications, databases (e.g., MySQL, PostgreSQL), and application servers (e.g., Tomcat, JBoss).
  - "yyyy-MM-dd'T'HH:mm:ss.SSSZ"               # Common in web services, APIs, Java (with DateTimeFormatter.ISO_OFFSET_DATE_TIME), Python (with datetime.isoformat()), and systems requiring precise timestamps with timezone information.
  - "MMM dd HH:mm:ss"                          # Frequently seen in syslog, Unix-based systems, network devices (e.g., Cisco routers), and web servers (e.g., Apache, Nginx).
  - "dd/MMM/yyyy:HH:mm:ss Z"                   # Standard syslog format, used in firewalls (e.g., Cisco ASA, Palo Alto), network monitoring tools, and Unix-based systems.
  - "yyyy-MM-dd HH:mm:ss,SSS"                  # Common in Java logging frameworks like Log4j and Logback.
  - "yyyy-MM-dd'T'HH:mm:ss,SSS"                # Common in Java logging with ISO 8601 date-time, comma-separated milliseconds.
  - "MM/dd/yyyy HH:mm:ss"                      # Used in Windows event logs, .NET applications, and U.S.-based systems.
  - "EEE MMM dd HH:mm:ss yyyy"                 # Human-readable format used in various logs, including some web servers and application logs.
  - "yyyy-MM-dd'T'HH:mm:ss"                    # Simplified ISO 8601 format, used in many modern applications and frameworks.
  - "HH:mm:ss"                                 # Time-only format, used when the date is implied or provided separately, common in embedded systems and some programming languages.
  - "yyyyMMdd HH:mm:ss"                        # Compact format used in some legacy systems and batch processing logs.
  - "MMM dd, yyyy h:mm:ss a"                   # Human-readable format with 12-hour clock, used in application logs and some U.S.-based systems.
  - "MMM dd, yyyy hh:mm:ss a"                  # Human-readable format with 12-hour clock (padded hour), used in application logs and some U.S.-based systems.
  # Common formats with slight variations
  - "yyyy-MM-dd HH:mm:ss.SSS"                  # Extended precision format used in Java applications and databases requiring millisecond accuracy.
  - "dd-MMM-yyyy HH:mm:ss.SSS"                 # Common in Java logging frameworks (e.g., Log4j, SLF4J) and application logs requiring human-readable dates with millisecond precision.
  - "yyyy-MM-dd'T'HH:mm:ss.S'Z'"               # Microsecond precision ISO 8601 format, 1-digit precision.
  - "yyyy-MM-dd'T'HH:mm:ss.SS'Z'"              # Microsecond precision ISO 8601 format, 2-digit precision.
  - "yyyy-MM-dd'T'HH:mm:ss.SSS'Z'"             # Microsecond precision ISO 8601 format, 3-digit precision.
  - "yyyy-MM-dd'T'HH:mm:ss.SSSS'Z'"            # Microsecond precision ISO 8601 format, 4-digit precision.
  - "yyyy-MM-dd'T'HH:mm:ss.SSSSS'Z'"           # Microsecond precision ISO 8601 format, 5-digit precision.
  - "yyyy-MM-dd'T'HH:mm:ss.SSSSSS'Z'"          # Microsecond precision ISO 8601 format, 6-digit precision.
  - "yyyy-MM-dd'T'HH:mm:ss.S'Z'"               # Nanosecond precision ISO 8601 format, 1-digit precision.
  - "yyyy-MM-dd'T'HH:mm:ss.SS'Z'"              # Nanosecond precision ISO 8601 format, 2-digit precision.
  - "yyyy-MM-dd'T'HH:mm:ss.SSS'Z'"             # Nanosecond precision ISO 8601 format, 3-digit precision.
  - "yyyy-MM-dd'T'HH:mm:ss.SSSS'Z'"            # Nanosecond precision ISO 8601 format, 4-digit precision.
  - "yyyy-MM-dd'T'HH:mm:ss.SSSSS'Z'"           # Nanosecond precision ISO 8601 format, 5-digit precision.
  - "yyyy-MM-dd'T'HH:mm:ss.SSSSSS'Z'"          # Nanosecond precision ISO 8601 format, 6-digit precision.
  - "yyyy-MM-dd'T'HH:mm:ss.SSSSSSS'Z'"         # Nanosecond precision ISO 8601 format, 7-digit precision.
  - "yyyy-MM-dd'T'HH:mm:ss.SSSSSSSS'Z'"        # Nanosecond precision ISO 8601 format, 8-digit precision.
  - "yyyy-MM-dd'T'HH:mm:ss.SSSSSSSSS'Z'"       # Nanosecond precision ISO 8601 format, 9-digit precision.
  - "yyyy-MM-dd'T'HH:mm:ssZZZZZ"               # ISO 8601 with timezone offset, used in systems where timezone information is critical.
  - "yyyy-MM-dd HH:mm:ss.SSSZZZZZ"             # Similar to above but without the 'T' separator, used in database logs and application servers.
  - "yyyy-MM-dd HH:mm:ss.S"                    # Microsecond precision format, 1-digit precision.
  - "yyyy-MM-dd HH:mm:ss.SS"                   # Microsecond precision format, 2-digit precision.
  - "yyyy-MM-dd HH:mm:ss.SSS"                  # Microsecond precision format, 3-digit precision.
  - "yyyy-MM-dd HH:mm:ss.SSSS"                 # Microsecond precision format, 4-digit precision.
  - "yyyy-MM-dd HH:mm:ss.SSSSS"                # Microsecond precision format, 5-digit precision.
  - "yyyy-MM-dd HH:mm:ss.SSSSSS"               # Microsecond precision format, 6-digit precision.
  - "MM/dd/yyyy*HH:mm:ss*SSS"                  # Used in some U.S.-based systems, particularly in legacy applications or specific logging frameworks.
  - "M/d/yyyy h:mm:ss a:SSS"                   # Common in systems using 12-hour time format, such as some Windows applications or older logging systems.
  - "M/d/yyyy hh:mm:ss a:SSS"                  # Common in systems using 12-hour time format with milliseconds and padded hour, such as some Windows applications.
  - "M/dd/yyyy hh:mm:ss a"                     # Similar to above, used in systems where millisecond precision is not needed.
  - "yyyy-MM-dd'T'HH:mm:ss.SSSX"               # ISO 8601 with basic timezone offset format, used in systems that require standardized timestamps.
  - "yyyy-MM-dd'T'HH:mm:ss.SSSz"               # ISO 8601 with timezone name, used in applications that need to display timezone information.
  - "yyyy-MM-dd'T'HH:mm:ss'Z'"                 # ISO 8601 format assuming UTC, used in systems where all times are in UTC.
  - "yy/MM/dd HH:mm:ss"                        # Two-digit year format, used in compact logs or older systems.
  - "MMM dd HH:mm:ss ZZZZ"                     # Used in some Unix-based systems and web servers, includes timezone offset.
  - "HH:mm:ss,SSS"                             # Time-only format with milliseconds, used in performance logs or systems where date is provided separately.
  - "yyyy-MM-dd*HH:mm:ss"                      # Used in some application logs where the separator is a space or asterisk.
  - "yyyy MMM dd HH:mm:ss.SSS"                 # Human-readable format with milliseconds, used in some application logs.
  - "dd/MMM/yyyy HH:mm:ss"                     # Used in European systems and some web applications.
  - "yyyy-MM-dd'T'HH:mm:ss.SSS''Z''"           # ISO 8601 with milliseconds and literal 'Z', used in systems where 'Z' is explicitly included.
  - "MMM dd yyyy HH:mm:ss"                     # Human-readable format, used in various logs.
  - "yyyy-MM-dd HH:mm:ss ZZZZ"                 # Format with timezone offset, used in systems requiring timezone information.
  - "yyyy-MM-dd HH:mm:ssZZZZZ"                 # Similar to above, used in application logs.
  - "dd MMM yyyy HH:mm:ss"                     # European format, used in some web applications and databases.
  - "MMdd_HH:mm:ss"                            # Compact format without separators, used in file names or space-constrained logs.
  - "yyyy-MM-dd HH:mm:ss,SSSZZZZZ"             # Java logging format with timezone offset.
  - "yyyyMMdd HH:mm:ss.SSS"                    # Compact format with milliseconds, used in some legacy systems.
  - "yyyy/MM/dd HH:mm:ss"                      # Format used in some Asian systems, particularly in Japan.
  - "dd/MM/yyyy HH:mm:ss"                      # Common in European systems, including some web applications and databases.
  - "MM-dd-yyyy HH:mm:ss"                      # U.S. format variant, used in some older systems.
  - "yyyyMMddHHmmss"                           # Compact format for file names or database timestamps, used in systems where space is a concern.
  - "EEE, dd MMM yyyy HH:mm:ss zzz"            # RFC 1123 format, used in HTTP headers, web servers, and email systems.

fields

Configure the Field parser to scan TenXTemplates for JSON and KV fields.

Below is the default configuration from: fields/config.yaml.

Edit Online

Edit Field parser Config Locally 

# 🔟❎ 'run' TenXTemplate field extract configuration

# Configure how to extract JSON objects and KV structures from TenXTemplates.
# To learn more see https://doc.log10x.com/run/transform/fields/

# Set the 10x pipeline to 'run'
tenx: run

# =============================== Extract Options =============================

field:

  # 'extract' controls whether to scan TenXTemplate for  JSON
  #  objects or key-value lists (e.g., 'X=Y'). 
  extract: true

  # 'nameBreaks' controls which characters found to the left of a token
  #  whose is a candidate for being a 'key' in a KV field formation should serve 
  #  as a terminator for the search. For example, for an object whose 'text' field
  #  contains the following entry ',tx_result=OK' the desired key name should be 'tx_result'
  #  and as such, the name terminator character should be ',' vs. '_' (in which case
  #  the key would have been named 'result').
  nameBreaks: ', /\{}.()[]'

  # 'valueBreaks' controls which characters found to the right of a token
  #  whose is a candidate for being a 'value' in a KV field formation should serve 
  #  as a terminator for the search. For example, for an object whose 'text' field
  #  contains the following entry: 'status=RESULT_SUCCESS' the desired KV value 
  #  should be 'RESULT_SUCCESS', and as such, the value terminator character 
  #  should be ',' vs. '_' (in which case the value would be 'RESULT').
  valueBreaks: ', /\{}()[]'

symbol

Configure the Origin selector to select the source code/binary executable origin of TenXTemplate symbol values.

Below is the default configuration from: symbol/config.yaml.

Edit Online

Edit Origin selector Config Locally 

# 🔟❎ 'run' symbol origin configuration

# Identify the origin of symbol values within TenXTemplates.
# To learn more see https://doc.log10x.com/run/transform/symbol/

# Set the 10x pipeline to 'run'
tenx: run

# =============================== Origin Options ==============================

symbol:

  # 'maxOrigins' controls the number of source/binary origins to list 
  #  per symbol token sequence. As a series of tokens within 
  #  a target TenXTemplate (e.g., 'ERROR', 'could not connect to "{}') may appear 
  #  in multiple source/binary files within the pipeline's loaded library.
  #  The sorting algorithm configured below is used to select the 'maxSymbolOrigins' 
  #  topmost entries to list.
  maxOrigins: 64

  # 'sequenceReserved' defines a list of terms to ignore when searching
  #  for the symbol tokens that constitute an TenXTemplate's 'message' portion.
  #  For example, for an event whose text contains: 'connect success = true',
  #  the value 'true' will not be considered a part of the event's message,
  #  as the 'true' value is mostly likely the result of a variable boolean state.
  #  For more information, see: https://doc.log10x.com/api/js/#TenXObject+symbolSequence
  sequenceReserved:
    - "null"
    - "nil"
    - "true"
    - "false"
    - "to"
    - "the"
    - "a"
    - "at"
    - "for"
    - "log"
    - "info"
    - "http"  
    - TRACE
    - DEBUG
    - INFO
    - NOTICE
    - WARN
    - ERROR
    - CRITICAL
    - ALERT
    - EMERGENCY

  # ----------------------------- Debug Options -------------------------------

  debug:

    # 'symbol' debugging allows for verbose printing of the selection process
    #  for symbol tokens from the pipeline's 10x symbol files used to produce the
    #  results of 10x reflection functions.
    #  For more information, see: https://doc.log10x.com/api/js/#TenXObject+symbol


    # 'origins' outputs information about the origin selection process for an TenXTemplate symbol.
    #  For example, setting 'units' to 'foo.js' will output information about if/how it 
    #  was selected as the origin of TenXTemplate symbols.
    #  In other words, if a symbol (e.g., 'MyClass') has 'foo.java' as the source file
    #  from which it originated, adding 'foo.js' to 'units' will emit information
    #  about the selection process.
    #  Specifying '*' will emit information for all source /binary files that have
    #  been selected as the origin of any TenXTemplate objects within the pipeline.
    origins: [
    #  '*'
    ]

    # 'symbols' logs the origin selection process for symbol tokens within an TenXTemplate.
    #  For example, set 'symbols' to 'Could not connect to' to log how the 10x JavaScript 'symbolSequence' 
    #  determines the origin source code/binary file within the pipeline's symbol library. 

    #  Specifying '*' will emit information for all source code /binary files that have
    #  been selected as the origin of any TenXTemplate objects within the pipeline.
    symbols: [
     # '*'
    ]

group

Configure the Group sequencer to group and sequence TenXObjects.

Below is the default configuration from: group/config.yaml.

Edit Online

Edit Group sequencer Config Locally 

# 🔟❎ 'run' event grouping configuration

# Group  sequences of TenXObjects to filter, aggregate and output as a single logical unit.
# To learn more see https://doc.log10x.com/run/transform/group/

# Set the 10x pipeline to 'run'
tenx: run

# =============================== Group Options ===============================

group:

  # 'filters' specify JavaScript expressions an TenXObject instance/group must 
  #  evaluate as truthy against to be written to output
  filters: []

  # 'maxSize' defines the maximum number of TenXObjects to group
  #  before the group is sealed and forwarded into the pipeline. 
  #  Subsequent TenXObjects can form a new group.
  maxSize: 20000

  # 'flushTimeout' defines the max interval (e.g., 10s) to wait for 
  #  new events to be read from an input stream before it flushes any
  #  pending TenXObjects group into the pipeline.
  #  This mechanism is designed to avoid latencies in dispatching pending event
  #  groups to output destinations.
  flushTimeout: $=parseDuration("5s")

  # 'async' specifies whether to sequence and group TenXObjects in a dedicated thread
  async: true

parallelize

Configure the Parallel processor to distribute event parsing and transformation workloads across multiple cores..

Below is the default configuration from: parallelize/config.yaml.

Edit Online

Edit Parallel processor Config Locally 

# 🔟❎ 'run' event parallel processing configuration

# Transform log/trace events read from inputs into well-defined TenXObjects using multiple cores.
# To learn more see https://doc.log10x.com/run/transform/parallelize/

# Set the 10x pipeline to 'run'
tenx: run

# =============================== Parallel Options ============================

parallelEvent:

  # 'threadPoolSize' specifies the number of threads allocated to transform events into TenXObjects concurrently.
  #  If the value is -
  #  - = 0: transform events into TenXObjects synchronously using their input stream calling thread.
  #  - < 1: interpreted as a percentage of the number of available cores (e.g., 0.5 = use up to 50% of available cores)
  #  - = 1: allocate a single dedicated to transform events.
  #  - > 1: interpreted as a fixed number of threads (e.g., 10 = 10 threads)
  threadPoolSize: "0.5"

  # 'batchSize' specifies the maximum number of events to queue for concurrent processing before flushing.
  #  If 'threadPoolSize' is 1, this value is unused, and events are transformed into TenXObject synchronously 
  #  to their calling input's thread.  If 0, flush pending events after 'parallelEventFlushInterval' expires or the
  #  source input reaches end-of-file.
  batchSize: 1000

  # 'flushInterval'  specifies the maximum wait duration before flushing queued events 
  #  If 'parallelThreadPoolSize' is 1, this value is unused, and events are transformed into TenXObject synchronously 
  #  to their input thread. If 0, no wait flush interval is applied. 
  flushInterval: 2s

  # 'processingTimeout' specifies the maximum wait duration before dropping un-processed queued events 
  #  This value provides a backstop for overflowing the heap if the pipeline cannot dequeue 
  #  pending events to transform into TenXObjects. If 0, no timeout is applied.
  processingTimeout: 30s


This app is defined in reporter/app.yaml.

Next: Deploy