Skip to content

Symbol Message Calculator

Extracts consistent message identifiers from log events for accurate log-to-metrics conversion and cost control.

Raw log events contain high-cardinality variable data (timestamps, IDs, values) mixed with constant, low-cardinality symbols.

The message initializer uses symbol libraries to isolate stable message patterns from each event, enabling accurate classification of event instances by their logical type.

Message Extraction

The initializer identifies the core message pattern by selecting the longest sequence of consecutive symbols from a TenXTemplate originating from the same source code or binary file.

The inputField parameter limits searches to specific JSON fields. Setting inputField: log searches only within the log field content.

Building on this process, here's how it applies to real events:

Kubernetes Example:

{
  "stream": "stderr",
  "log": "2025-04-17 14:32:40,287 INFO [main] [recommendation_server.py:47] - Receive ListRecommendations for product ids:['L9ECAV7KIM', '0PUK6V6EV0']",
  "docker": {
    "container_id": "9c04355088aa168abb1a074b696ad15366c254602be8cbb69299e1e87d3bcffb"
  },
  "kubernetes": {
    "container_name": "recommendationservice",
    "namespace_name": "default"
  }
}

Extracted Message:

Receive_ListRecommendations_for_product_ids

Kafka Controller Event:

{
  "stream": "stdout",
  "log": "[2025-08-01 22:19:30,905] INFO [controller-1-to-controller-registration-channel-manager]: Recorded new controller, from now on will use node 0.0.0.0:9093 (id: 1 rack: null) (kafka.server.NodeToControllerRequestThread)",
  "docker": {
    "container_id": "79af0d7ce5f3c159411c6a15ee2d9044f3559bd2fe1630f8a6640d4c2cc87771"
  },
  "kubernetes": {
    "container_name": "kafka",
    "namespace_name": "default",
    "pod_name": "kafka-549545757c-2lmxv",
    "container_image": "ghcr.io/open-telemetry/demo:2.0.2-kafka"
  }
}

Extracted Message:

channel_manager_Recorded_new_controller_from_now_on_will_use_node_id_rack

OpenSearch PeerFinder Event:

{
  "stream": "stdout",
  "log": "[2025-08-01T22:19:24,590][INFO ][o.o.d.PeerFinder         ] [opensearch-0] setting findPeersInterval to [1s] as node commission status = [true] for local node [{opensearch-0}{N_KuFBFGRmSnettsBzOX3Q}{3XUyt5iPRMKvzuHPCaPFyg}{192.168.57.56}{192.168.57.56:9300}{dimr}{shard_indexing_pressure_enabled=true}]",
  "docker": {
    "container_id": "b6f244ebdaa72d7565b8944a1aad79cd5ac06ac767e4e603145a5e4bfd121883"
  },
  "kubernetes": {
    "container_name": "opensearch",
    "namespace_name": "default",
    "pod_name": "opensearch-0",
    "container_image": "docker.io/opensearchproject/opensearch:2.19.0"
  }
}

Extracted Message:

commission_status_local_node_opensearch_shard_indexing_pressure_enabled

HTTP Access Log Event:

{
  "stream": "stdout",
  "log": "192.168.43.96 - - [01/Aug/2025:22:21:50 +0000] \"GET /products/LensCleaningKit.jpg HTTP/1.1\" 200 101928 \"http://frontend-proxy:8080/\" \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/133.0.0.0 Safari/537.36\"",
  "docker": {
    "container_id": "ac37d50d39857193f5d2ff92872f1c022d3b746fa721233eccd1b4aae7d26a8b"
  },
  "kubernetes": {
    "container_name": "image-provider",
    "namespace_name": "default",
    "pod_name": "image-provider-58c6f8444-q4c8p",
    "container_image": "ghcr.io/open-telemetry/demo:2.0.2-image-provider"
  }
}

Extracted Message:

frontend_proxy_Mozilla_X11_Linux_x86_AppleWebKit_KHTML_like_Gecko_Safari

See the JavaScript implementation of this module on Github.

Applications

💰 Cost tracking: Identifies high-volume event types consuming log budgets with the Dev app app

📈 Cost control: Apply intelligent filtering using the Receiver app to prevent over-billing

🤖 Multi-platform analytics: Feed patterns into AIOps and monitoring systems via metric outputs for Datadog, CloudWatch, SignalFx, and Prometheus

🔄 Automatic adaptation: Updates automatically with code changes using symbol libraries. No manual regex pattern configuration and maintenance

Config Files

To configure the Symbol Message Calculator module, Edit these files.

Below is the default configuration from: message/config.yaml.

Edit Online

Edit config.yaml Locally 

# 🔟❎ 'run' symbol lookup configuration

# Configure a symbol origin lookup to enrich TenXObjects
# To learn more see https://doc.log10x.com/run/initialize/message/

# Set the 10x pipeline to 'run'
tenx: run

# =============================== Dependencies ================================

include: run/modules/initialize/message

# ============================== Symbol Options ===============================

symbol:

  # 'types' specifies the types of symbols to search for; if the first type does not yield a result, the next one is tried etc.
  #  Supported values (case insensitive), PACKAGE, CLASS, METHOD, LOG, ENUM, CONST, TEXT, EXEC, ANY
  #  To learn more see https://doc.log10x.com/run/transform/symbol/#contexts
  contexts: log,exec

  # 'messageField' specifies the field name to assign the source/binary origin file name to target TenXObjects
  messageField: message_pattern

  # 'messageHashField' specifies the field name to assign a stable, URL-safe pattern hash (xxHash64 of the messageField value, base64url, 11 chars). Safe as a SIEM query term or forwarder filter key. Set empty to disable.
  messageHashField: tenx_hash

  # 'maxLen' specifies the max char len of the output messageField
  maxLen: 120

Options

Specify the options below to configure the Symbol Message Calculator:

Name Description
symbolContexts Types of symbols to search
symbolMaxLen Max number of output chars
symbolMessageField Field name to assign with symbol sequence value
symbolMessageHashField Field name to assign with the hash of the symbol sequence value
symbolOriginField Field name to assign with symbol message origin file

symbolContexts

Types of symbols to search.

Type Default
String log,exec

Specifies a comma delimited list of symbol contexts values.

This is argument is passed as the symbolContexts argument value for the symbolSequence function.

symbolMaxLen

Max number of output chars.

Type Default
Number 0

Specifies the max number of chars of the symbolMessageField. Set 0 to unlimited.

symbolMessageField

Field name to assign with symbol sequence value.

Type Default
String symbolMessage

Specifies the name of the field to assign the return value of symbolSequence function when invoking it with the a symbolContext argument value of symbolContexts.

For an example, see symbolInputField.

symbolMessageHashField

Field name to assign with the hash of the symbol sequence value.

Type Default
String tenx_hash

Specifies the name of the field to assign a stable, URL-safe hash of the symbolMessageField value, computed via TenXString.patternHash.

The value is an 11-character base64url identifier (xxHash64 of the UTF-8 bytes, big-endian, no padding). It is one-way and reproduces byte-for-byte outside the engine, so it is safe to use directly as a deduplication key, metric dimension, SIEM query term, or forwarder filter key.

Defaults to tenx_hash. Set to an empty value to disable.

symbolOriginField

Field name to assign with symbol message origin file.

Type Default
String

Specifies the name of the field to assign the return value of symbolOrigin function.


This module is defined in message/module.yaml.