Symbol Message Calculator
Extracts consistent message identifiers from log events for accurate log-to-metrics conversion and cost control.
Raw log events contain high-cardinality variable data (timestamps, IDs, values) mixed with constant, low-cardinality symbols.
The message initializer uses symbol libraries to isolate stable message patterns from each event, enabling accurate classification of event instances by their logical type.
Message Extraction
The initializer identifies the core message pattern by selecting the longest sequence of consecutive symbols from a TenXTemplate originating from the same source code or binary file.
The inputField parameter limits searches to specific JSON fields. Setting inputField: log searches only within the log field content.
Pattern identity: pattern vs template
Four terms are easy to conflate. They are distinct:
- Pattern (
symbolMessage), the selection described above: a subset of representing tokens chosen from the template, not the whole line. Short and legible (e.g.Receive ListRecommendations for product ids). It is the unit of cost attribution. pattern_hash(alias:tenx_hash), the hash of thesymbolMessage(thesymbolMessageHashField, defaulttenx_hash). This is the stable, user-facing identity that tools and metrics key on. It is stable because it keys on the representing subset: it stays constant across deploys, restarts, pod renames, and format drift, and many template variants that share the representing tokens collapse to the samepattern_hash.- Template, the full
$-marked structural shape of the line (every token, with variable slots marked$). A single pattern sits over a set of templates, one per format variant present in the data. template_hash, the engine-internal fingerprint of a template's field-set. It exists only to join encoded events back to their entry intemplates.jsonat decode time. It is not the stable identity, it is many-to-one with the pattern, and it should never be surfaced to a user or agent as the identifier. Usepattern_hashfor that.
Building on this process, here's how it applies to real events:
Kubernetes Example:
{
"stream": "stderr",
"log": "2025-04-17 14:32:40,287 INFO [main] [recommendation_server.py:47] - Receive ListRecommendations for product ids:['L9ECAV7KIM', '0PUK6V6EV0']",
"docker": {
"container_id": "9c04355088aa168abb1a074b696ad15366c254602be8cbb69299e1e87d3bcffb"
},
"kubernetes": {
"container_name": "recommendationservice",
"namespace_name": "default"
}
}
Extracted Message:
Receive_ListRecommendations_for_product_ids
Kafka Controller Event:
{
"stream": "stdout",
"log": "[2025-08-01 22:19:30,905] INFO [controller-1-to-controller-registration-channel-manager]: Recorded new controller, from now on will use node 0.0.0.0:9093 (id: 1 rack: null) (kafka.server.NodeToControllerRequestThread)",
"docker": {
"container_id": "79af0d7ce5f3c159411c6a15ee2d9044f3559bd2fe1630f8a6640d4c2cc87771"
},
"kubernetes": {
"container_name": "kafka",
"namespace_name": "default",
"pod_name": "kafka-549545757c-2lmxv",
"container_image": "ghcr.io/open-telemetry/demo:2.0.2-kafka"
}
}
Extracted Message:
channel_manager_Recorded_new_controller_from_now_on_will_use_node_id_rack
OpenSearch PeerFinder Event:
{
"stream": "stdout",
"log": "[2025-08-01T22:19:24,590][INFO ][o.o.d.PeerFinder ] [opensearch-0] setting findPeersInterval to [1s] as node commission status = [true] for local node [{opensearch-0}{N_KuFBFGRmSnettsBzOX3Q}{3XUyt5iPRMKvzuHPCaPFyg}{192.168.57.56}{192.168.57.56:9300}{dimr}{shard_indexing_pressure_enabled=true}]",
"docker": {
"container_id": "b6f244ebdaa72d7565b8944a1aad79cd5ac06ac767e4e603145a5e4bfd121883"
},
"kubernetes": {
"container_name": "opensearch",
"namespace_name": "default",
"pod_name": "opensearch-0",
"container_image": "docker.io/opensearchproject/opensearch:2.19.0"
}
}
Extracted Message:
commission_status_local_node_opensearch_shard_indexing_pressure_enabled
HTTP Access Log Event:
{
"stream": "stdout",
"log": "192.168.43.96 - - [01/Aug/2025:22:21:50 +0000] \"GET /products/LensCleaningKit.jpg HTTP/1.1\" 200 101928 \"http://frontend-proxy:8080/\" \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/133.0.0.0 Safari/537.36\"",
"docker": {
"container_id": "ac37d50d39857193f5d2ff92872f1c022d3b746fa721233eccd1b4aae7d26a8b"
},
"kubernetes": {
"container_name": "image-provider",
"namespace_name": "default",
"pod_name": "image-provider-58c6f8444-q4c8p",
"container_image": "ghcr.io/open-telemetry/demo:2.0.2-image-provider"
}
}
Extracted Message:
frontend_proxy_Mozilla_X11_Linux_x86_AppleWebKit_KHTML_like_Gecko_Safari
See the JavaScript implementation of this module on Github.
Applications
💰 Cost tracking: Identifies high-volume event types consuming log budgets with the Dev app app
📈 Cost control: Apply intelligent filtering using the Receiver app to prevent over-billing
🤖 Multi-platform analytics: Feed patterns into AIOps and monitoring systems via metric outputs for Datadog, CloudWatch, SignalFx, and Prometheus
🔄 Automatic adaptation: Updates automatically with code changes using symbol libraries. No manual regex pattern configuration and maintenance
Config Files
To configure the Symbol Message Calculator module, Edit these files.
Below is the default configuration from: message/config.yaml.
ewogICJ0eXBlIiA6ICJvYmplY3QiLAogICJwcm9wZXJ0aWVzIiA6IHsKICAgICJpbmNsdWRlIiA6IHsKICAgICAgInR5cGUiIDogInN0cmluZyIKICAgIH0sCiAgICAidGVueCIgOiB7CiAgICAgICJ0eXBlIiA6ICJzdHJpbmciCiAgICB9LAogICAgInN5bWJvbCIgOiB7CiAgICAgICJ0eXBlIiA6ICJvYmplY3QiLAogICAgICAiYWRkaXRpb25hbFByb3BlcnRpZXMiIDogZmFsc2UsCiAgICAgICJwcm9wZXJ0aWVzIiA6IHsKICAgICAgICAiY29udGV4dHMiIDogewogICAgICAgICAgInR5cGUiIDogWwogICAgICAgICAgICAic3RyaW5nIiwKICAgICAgICAgICAgIm51bGwiCiAgICAgICAgICBdLAogICAgICAgICAgIm1hcmtkb3duRGVzY3JpcHRpb24iIDogIlR5cGVzIG9mIHN5bWJvbHMgdG8gc2VhcmNoXG5cblNwZWNpZmllcyBhIGNvbW1hIGRlbGltaXRlZCBsaXN0IG9mIFtzeW1ib2wgY29udGV4dHNdKGh0dHBzOi8vZG9jLmxvZzEweC5jb20vcnVuL3RyYW5zZm9ybS9zeW1ib2wvI2NvbnRleHRzKSB2YWx1ZXMuIFRoaXMgaXMgYXJndW1lbnQgaXMgcGFzc2VkIGFzIHRoZSBgc3ltYm9sQ29udGV4dHNgIGFyZ3VtZW50IHZhbHVlIGZvciB0aGUgW3N5bWJvbFNlcXVlbmNlXShodHRwczovL2RvYy5sb2cxMHguY29tL2FwaS9qcy8jVGVuWE9iamVjdCtzeW1ib2xTZXF1ZW5jZSkgZnVuY3Rpb24uIChEZWZhdWx0OiBsb2csZXhlYykiLAogICAgICAgICAgImRlZmF1bHQiIDogImxvZyxleGVjIgogICAgICAgIH0sCiAgICAgICAgIm1heExlbiIgOiB7CiAgICAgICAgICAidHlwZSIgOiBbCiAgICAgICAgICAgICJudW1iZXIiLAogICAgICAgICAgICAic3RyaW5nIiwKICAgICAgICAgICAgIm51bGwiCiAgICAgICAgICBdLAogICAgICAgICAgIm1hcmtkb3duRGVzY3JpcHRpb24iIDogIk1heCBudW1iZXIgb2Ygb3V0cHV0IGNoYXJzXG5cblNwZWNpZmllcyB0aGUgbWF4IG51bWJlciBvZiBjaGFycyBvZiB0aGUgW3N5bWJvbE1lc3NhZ2VGaWVsZF0oaHR0cHM6Ly9kb2MubG9nMTB4LmNvbS9ydW4vaW5pdGlhbGl6ZS9tZXNzYWdlLyNzeW1ib2xtZXNzYWdlZmllbGQpLiBTZXQgMCB0byB1bmxpbWl0ZWQuIChBY2NlcHRzIG51bWJlciBvciBzdHJpbmcgd2l0aCAkPSBwcmVmaXggZm9yIHJ1bnRpbWUgZXZhbHVhdGlvbikgKERlZmF1bHQ6IDApIiwKICAgICAgICAgICJkZWZhdWx0IiA6IDAKICAgICAgICB9LAogICAgICAgICJtZXNzYWdlIiA6IHsKICAgICAgICAgICJ0eXBlIiA6ICJvYmplY3QiLAogICAgICAgICAgImFkZGl0aW9uYWxQcm9wZXJ0aWVzIiA6IGZhbHNlLAogICAgICAgICAgInByb3BlcnRpZXMiIDogewogICAgICAgICAgICAiZmllbGQiIDogewogICAgICAgICAgICAgICJ0eXBlIiA6IFsKICAgICAgICAgICAgICAgICJzdHJpbmciLAogICAgICAgICAgICAgICAgIm51bGwiCiAgICAgICAgICAgICAgXSwKICAgICAgICAgICAgICAibWFya2Rvd25EZXNjcmlwdGlvbiIgOiAiRmllbGQgbmFtZSB0byBhc3NpZ24gd2l0aCBzeW1ib2wgc2VxdWVuY2UgdmFsdWVcblxuU3BlY2lmaWVzIHRoZSBuYW1lIG9mIHRoZSBmaWVsZCB0byBhc3NpZ24gdGhlIHJldHVybiB2YWx1ZSBvZiBbc3ltYm9sU2VxdWVuY2VdKGh0dHBzOi8vZG9jLmxvZzEweC5jb20vYXBpL2pzLyNUZW5YT2JqZWN0K3N5bWJvbFNlcXVlbmNlKSBmdW5jdGlvbiB3aGVuIGludm9raW5nIGl0IHdpdGggdGhlIGEgc3ltYm9sQ29udGV4dCBhcmd1bWVudCB2YWx1ZSBvZiBbc3ltYm9sQ29udGV4dHNdKGh0dHBzOi8vZG9jLmxvZzEweC5jb20vcnVuL2luaXRpYWxpemUvbWVzc2FnZS8jc3ltYm9sY29udGV4dHMpLiAgRm9yIGFuIGV4YW1wbGUsIHNlZSBbc3ltYm9sSW5wdXRGaWVsZF0oaHR0cHM6Ly9kb2MubG9nMTB4LmNvbS9ydW4vaW5pdGlhbGl6ZS9tZXNzYWdlLyNzeW1ib2xpbnB1dGZpZWxkKS4gKERlZmF1bHQ6IHN5bWJvbE1lc3NhZ2UpIiwKICAgICAgICAgICAgICAiZGVmYXVsdCIgOiAic3ltYm9sTWVzc2FnZSIKICAgICAgICAgICAgfSwKICAgICAgICAgICAgImhhc2hGaWVsZCIgOiB7CiAgICAgICAgICAgICAgInR5cGUiIDogWwogICAgICAgICAgICAgICAgInN0cmluZyIsCiAgICAgICAgICAgICAgICAibnVsbCIKICAgICAgICAgICAgICBdLAogICAgICAgICAgICAgICJtYXJrZG93bkRlc2NyaXB0aW9uIiA6ICJGaWVsZCBuYW1lIHRvIGFzc2lnbiB3aXRoIHRoZSBoYXNoIG9mIHRoZSBzeW1ib2wgc2VxdWVuY2UgdmFsdWVcblxuU3BlY2lmaWVzIHRoZSBuYW1lIG9mIHRoZSBmaWVsZCB0byBhc3NpZ24gYSBzdGFibGUsIFVSTC1zYWZlIGhhc2ggb2YgdGhlIFtzeW1ib2xNZXNzYWdlRmllbGRdKGh0dHBzOi8vZG9jLmxvZzEweC5jb20vcnVuL2luaXRpYWxpemUvbWVzc2FnZS8jc3ltYm9sbWVzc2FnZWZpZWxkKSB2YWx1ZSwgY29tcHV0ZWQgdmlhIFtUZW5YU3RyaW5nLmhhc2hdKGh0dHBzOi8vZG9jLmxvZzEweC5jb20vYXBpL2pzLyNUZW5YU3RyaW5nLmhhc2gpLiAgVGhlIHZhbHVlIGlzIGFuIDExLWNoYXJhY3RlciBiYXNlNjR1cmwgaWRlbnRpZmllciAoeHhIYXNoNjQgb2YgdGhlIFVURi04IGJ5dGVzLCBiaWctZW5kaWFuLCBubyBwYWRkaW5nKS4gSXQgaXMgb25lLXdheSBhbmQgcmVwcm9kdWNlcyBieXRlLWZvci1ieXRlIG91dHNpZGUgdGhlIGVuZ2luZSwgc28gaXQgaXMgc2FmZSB0byB1c2UgZGlyZWN0bHkgYXMgYSBkZWR1cGxpY2F0aW9uIGtleSwgbWV0cmljIGRpbWVuc2lvbiwgYW5hbHl6ZXIgcXVlcnkgdGVybSwgb3IgZm9yd2FyZGVyIGZpbHRlciBrZXkuICBEZWZhdWx0cyB0byBgdGVueF9oYXNoYC4gU2V0IHRvIGFuIGVtcHR5IHZhbHVlIHRvIGRpc2FibGUuIChEZWZhdWx0OiB0ZW54X2hhc2gpIiwKICAgICAgICAgICAgICAiZGVmYXVsdCIgOiAidGVueF9oYXNoIgogICAgICAgICAgICB9CiAgICAgICAgICB9CiAgICAgICAgfSwKICAgICAgICAib3JpZ2luRmllbGQiIDogewogICAgICAgICAgInR5cGUiIDogWwogICAgICAgICAgICAic3RyaW5nIiwKICAgICAgICAgICAgIm51bGwiCiAgICAgICAgICBdLAogICAgICAgICAgIm1hcmtkb3duRGVzY3JpcHRpb24iIDogIkZpZWxkIG5hbWUgdG8gYXNzaWduIHdpdGggc3ltYm9sIG1lc3NhZ2Ugb3JpZ2luIGZpbGVcblxuU3BlY2lmaWVzIHRoZSBuYW1lIG9mIHRoZSBmaWVsZCB0byBhc3NpZ24gdGhlIHJldHVybiB2YWx1ZSBvZiBbc3ltYm9sT3JpZ2luXShodHRwczovL2RvYy5sb2cxMHguY29tL2FwaS9qcy8jVGVuWE9iamVjdCtzeW1ib2xPcmlnaW4pIGZ1bmN0aW9uLiAoRGVmYXVsdDogKSIsCiAgICAgICAgICAiZGVmYXVsdCIgOiAiIgogICAgICAgIH0KICAgICAgfQogICAgfSwKICAgICJtZXNzYWdlTmVnYXRvcnMiIDogewogICAgICAidHlwZSIgOiBbCiAgICAgICAgImFycmF5IiwKICAgICAgICAic3RyaW5nIiwKICAgICAgICAibnVsbCIKICAgICAgXSwKICAgICAgIm1hcmtkb3duRGVzY3JpcHRpb24iIDogIkxpc3Qgb2Ygc3RyaW5ncyB0aGF0LCB3aGVuIG1hdGNoZWQgYXQgdGhlIHN0YXJ0IG9mIGEgbG9nIGxpbmUncyB0ZXh0LCBtYXJrIGl0IGFzIGxvdy12YWx1ZSBjb250aW51YXRpb24vYm9pbGVycGxhdGUgdGhhdCBkb2VzIE5PVCByZWNlaXZlIGl0cyBvd24gbWVzc2FnZSBwYXR0ZXJuLlxuXG5BIGxpc3Qgb2Ygc3RyaW5ncyB0aGF0LCB3aGVuIG1hdGNoZWQgYXQgdGhlIHN0YXJ0IG9mIGEgbG9nIGxpbmUncyB0ZXh0LCBtYXJrIGl0IGFzIGEga25vd24gY29udGludWF0aW9uIHBhdHRlcm4gKGUuZy4sIHN0YWNrIHRyYWNlIGZyYW1lLCBpbmRlbnRlZCBkZXRhaWwsIGJhbm5lcikgdGhhdCBzaG91bGQgbm90IHJlY2VpdmUgaXRzIG93biBtZXNzYWdlIHBhdHRlcm4uICBOZWdhdG9ycyBhcmUgZXZhbHVhdGVkIGJ5IHRoZSBtZXNzYWdlIHRlbXBsYXRlJ3MgZ2F0ZSB3aGVuIHRoZSBldmVudCBpcyBuZWl0aGVyIHBhcnQgb2YgYSBtdWx0aS1ldmVudCBncm91cCAoZ3JvdXBTaXplID4gMSkgbm9yIGEgZ3JvdXAgaGVhZDogaW4gdGhhdCBjYXNlLCBhbiBldmVudCBzdGlsbCByZWNlaXZlcyBhIHBhdHRlcm4gdW5sZXNzIGl0IG1hdGNoZXMgYSBuZWdhdG9yLiBUaGlzIGlzIHRoZSBcIkhlbGxvIFdvcmxkIHZzIG9ycGhhbiBzdGFjayBmcmFtZVwiIGRpc2NyaW1pbmF0b3IuICAtIGAgYCAtIGFueSBpbmRlbnRlZCBsaW5lICh1bml2ZXJzYWwgY29udGludWF0aW9uIHNpZ25hbCkuIC0gYFx0YCAtIGFueSB0YWItcHJlZml4ZWQgbGluZSAoSmF2YS9HbyBzdGFjayBmcmFtZXMgZXRjLikuIC0gYENhdXNlZCBieTogYCAtIEphdmEgY2hhaW5lZCBleGNlcHRpb24gKGNvbHVtbiAwKS4gLSBgU3VwcHJlc3NlZDogYCAtIEphdmEgc3VwcHJlc3NlZCBleGNlcHRpb24gKGNvbHVtbiAwKS4gLSBgLS0tIEVuZCBvZiBgIC0gLk5FVCBpbm5lciBleGNlcHRpb24gbWFya2VyLiAtIGBjcmVhdGVkIGJ5IGAgLSBHbyBnb3JvdXRpbmUgY3JlYXRvci4gLSBgIzAgYC4uYCM1IGAgLSBQSFAgc3RhY2sgZnJhbWVzIChkZXB0aCAwLTUpLiIsCiAgICAgICJpdGVtcyIgOiB7CiAgICAgICAgInR5cGUiIDogInN0cmluZyIKICAgICAgfQogICAgfQogIH0sCiAgImFkZGl0aW9uYWxQcm9wZXJ0aWVzIiA6IHRydWUKfQ==
# 🔟❎ 'run' symbol lookup configuration
# Configure a symbol origin lookup to enrich TenXObjects
# To learn more see https://doc.log10x.com/run/initialize/message/
# Set the 10x pipeline to 'run'
tenx: run
# =============================== Dependencies ================================
include: run/modules/initialize/message
# ============================== Symbol Options ===============================
symbol:
# 'types' specifies the types of symbols to search for; if the first type does not yield a result, the next one is tried etc.
# Supported values (case insensitive), PACKAGE, CLASS, METHOD, LOG, ENUM, CONST, TEXT, EXEC, ANY
# To learn more see https://doc.log10x.com/run/transform/symbol/#contexts
contexts: log,exec
# 'messageField' specifies the field name to assign the source/binary origin file name to target TenXObjects
messageField: message_pattern
# 'messageHashField' specifies the field name to assign a stable, URL-safe pattern hash (xxHash64 of the messageField value, base64url, 11 chars). Safe as a SIEM query term or forwarder filter key. Set empty to disable.
messageHashField: tenx_hash
# 'maxLen' specifies the max char len of the output messageField
maxLen: 120
# ============================== Message Options ==============================
# 'negators' specifies patterns that mark a log line as boilerplate/continuation (stack trace frames,
# indented detail, banners) so the message-template skips its own message-pattern compute for them.
# The check fires only for templates that don't already pass the (groupSize > 1) || groupHead arms;
# i.e., for single-event templates with no head signal. A non-negator line in that case (e.g., bare
# "Hello World", "Heartbeat received from node-7") still gets a pattern; a negator-matching orphan
# (e.g., a lone "\tat com.foo.Bar(...)") does not.
message:
negators:
- " " # any indented line (stack frames, indented app detail, multi-line JSON, banners)
- "\t" # real tab byte, for msgpack/forward inputs that preserve raw whitespace
- "\\t" # literal backslash+t, for file/JSON inputs that keep escape sequences raw
- "Caused by: " # Java chained exception (column 0)
- "Suppressed: " # Java suppressed exception (column 0)
- "--- End of " # .NET inner exception marker (column 0)
- "created by " # Go goroutine creator (column 0)
- "#0 " # PHP stack frame depth 0
- "#1 " # PHP stack frame depth 1
- "#2 " # PHP stack frame depth 2
- "#3 " # PHP stack frame depth 3
- "#4 " # PHP stack frame depth 4
- "#5 " # PHP stack frame depth 5
Options
Specify the options below to configure the Symbol Message Calculator:
| Name | Description |
|---|---|
| symbolContexts | Types of symbols to search |
| symbolMaxLen | Max number of output chars |
| symbolMessageField | Field name to assign with symbol sequence value |
| symbolMessageHashField | Field name to assign with the hash of the symbol sequence value |
| symbolOriginField | Field name to assign with symbol message origin file |
| messageNegators | List of strings that, when matched at the start of a log line's text, mark it as low-value continuation/boilerplate that does NOT receive its own message pattern. |
symbolContexts
Types of symbols to search.
| Type | Default |
|---|---|
| String | log,exec |
Specifies a comma delimited list of symbol contexts values.
This is argument is passed as the symbolContexts argument value for the symbolSequence function.
symbolMaxLen
Max number of output chars.
| Type | Default |
|---|---|
| Number | 0 |
Specifies the max number of chars of the symbolMessageField. Set 0 to unlimited.
symbolMessageField
Field name to assign with symbol sequence value.
| Type | Default |
|---|---|
| String | symbolMessage |
Specifies the name of the field to assign the return value of symbolSequence function when invoking it with the a symbolContext argument value of symbolContexts.
For an example, see symbolInputField.
symbolMessageHashField
Field name to assign with the hash of the symbol sequence value.
| Type | Default |
|---|---|
| String | tenx_hash |
Specifies the name of the field to assign a stable, URL-safe hash of the symbolMessageField value, computed via TenXString.hash.
The value is an 11-character base64url identifier (xxHash64 of the UTF-8 bytes, big-endian, no padding). It is one-way and reproduces byte-for-byte outside the engine, so it is safe to use directly as a deduplication key, metric dimension, analyzer query term, or forwarder filter key.
Defaults to tenx_hash. Set to an empty value to disable.
symbolOriginField
Field name to assign with symbol message origin file.
| Type | Default |
|---|---|
| String |
Specifies the name of the field to assign the return value of symbolOrigin function.
messageNegators
List of strings that, when matched at the start of a log line's text, mark it as low-value continuation/boilerplate that does NOT receive its own message pattern.
| Type | Default |
|---|---|
| List | [] |
A list of strings that, when matched at the start of a log line's text, mark it as a known continuation pattern (e.g., stack trace frame, indented detail, banner) that should not receive its own message pattern.
Negators are evaluated by the message template's gate when the event is neither part of a multi-event group (groupSize > 1) nor a group head: in that case, an event still receives a pattern unless it matches a negator. This is the "Hello World vs orphan stack frame" discriminator.
- any indented line (universal continuation signal).- any tab-prefixed line (Java/Go stack frames etc.).Caused by:- Java chained exception (column 0).Suppressed:- Java suppressed exception (column 0).--- End of- .NET inner exception marker.created by- Go goroutine creator.#0..#5- PHP stack frames (depth 0-5).
This module is defined in message/module.yaml.