Skip to content

Symbols

Identify the source code/binary executable origin of symbol values within TenXTemplate hidden classes.

Identifying where each symbol value originated from within TenXTemplates enables reflective programming wherein TenXObjects can inspect their structure and member types and source code/executable file origin.

Each symbol lists its file of origin (e.g., foo.js, sudo, acme.dll) and the context in which it appears in it (e.g., class/func/enum name, string format..).

Source code/executable origin information is accessible at run time via the symbol, symbolSequence, and symbolOrigin functions.

this.class = this.symbol("class");
this.sequence = this.symbolSequence("log,exec"); 
this.origin = this.symbolOrigin("log,exec");

For the following Apache Spark log event:

17/06/09 20:10:42 INFO executor.CoarseGrainedExecutorBackend: Connecting to driver: spark://CoarseGrainedScheduler@10.10.34.11:48069

Field Value
class CoarseGrainedExecutorBackend
sequence Connecting to driver
origin CoarseGrainedExecutorBackend.scala

For the following Unix log event:

Jun 14 15:16:02 combo sshd(pam_unix)[19937]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=218.188.2.4

Field Value
class ""
sequence authentication failure
origin sshd

Contexts

Compile scanners assigns each extracted symbol value within a target source code/binary file a context which describes its meaning and position within the file.

Supported contexts include:

package

Source code package/namespace name. For example:

package com.acme.client.core;

  public class AcmeLogic {
  ...  
  }

namespace MyCompany.Proj1
{
    class MyClass1
    {
      ...
    }
}

class

Source code class/interface definition. For example:

class AcmeClient:
...

public interface AcmeServer {...}

The scanner will capture AcmeServer, AcmeClient as class symbols.

The scanner will capture com.acme.client.core, MyCompany.Proj1 as package symbols.

method

Source code method/function name. For example:

function hostError(host) {...}

void Person::get_name() {

The scanner will capture hostError, get_name as method symbols

enum

Source code enum literal value. For example:

from enum import Enum

class STATUS(Enum):
OK = 1
FAIL = 2
RETRY = 3

The scanner will capture:

  • OK, FAIL, RETRY as enum symbols
  • STATUS as a class symbol

log

Source code constant arg within a logging method. For example:

function hostError(host) {
  //The "error connecting" symbol passes as an indirect arg to the logger
  var msg = fmtError("error connecting", host);
  console.error(msg); 
}

The scanner will capture error connecting as a log symbol.

const

Source code constant value assigned to a variable, parameter or code annotation. For example:

@JsonElement(key = "price")
public String price;

report("state: " + state);

this.key = "key";

The scanner will capture price, state, key as a const symbol, unless the enclosing method logs or throws, in which case its literals are captured as log symbols. Const symbols are not linked into the library by default.

exec

Symbol captured from a custom program or OS command via the executable symbol scanner (e.g., strings).

For example:

strings -a /usr/bin/sudo

The scanner will capture Authentication failure message %s as an exec symbol.

text

Config field name/value captured from file (e.g., .xml, .json) via the text symbol scanner.

For example:

spring:
  name: test
  environment: testing

The scanner will capture the field names spring, name, and environment as text symbols.

The test and testing field values are only captured as text if the textScanFieldValues is set to true.

Config Files

To configure the Origin selector unit, Edit these files.

Below is the default configuration from: symbol/config.yaml.

Edit Online

Edit config.yaml Locally 

# 🔟❎ 'run' symbol origin configuration

# Identify the origin of symbol values within TenXTemplates.
# To learn more see https://doc.log10x.com/run/transform/symbol/

# Set the 10x pipeline to 'run'
tenx: run

# =============================== Origin Options ==============================

symbol:

  # 'maxOrigins' is the most places in code to compare for one phrase. A phrase
  #  such as 'could not connect to' can appear in many files. The first ones in
  #  library order, by file name and then by unit name, are compared, so the same
  #  library picks the same origin on every node.
  maxOrigins: 64

  # 'sequenceReserved' lists words that do not count as evidence of where a line
  #  comes from. A match made only of these words cannot pick a source. Before the
  #  message they are skipped, and a key from this list is skipped with its value
  #  ('level=info'). Words of the statement itself stay in the name as written.
  #  To learn more see https://doc.log10x.com/run/transform/symbol/#symbolsequencereserved
  sequenceReserved:
    # literals
    - "null"
    - "nil"
    - "true"
    - "false"
    # stopwords (articles, conjunctions, prepositions)
    - "a"
    - "an"
    - "the"
    - "to"
    - "of"
    - "in"
    - "on"
    - "at"
    - "by"
    - "for"
    - "from"
    - "with"
    - "and"
    - "or"
    - "as"
    - "is"
    # generic log words
    - "log"
    - "logger"
    - "logs"
    - "http"
    - "https"
    # severity levels, uppercase
    - TRACE
    - DEBUG
    - INFO
    - NOTICE
    - WARN
    - WARNING
    - ERROR
    - ERR
    - CRITICAL
    - CRIT
    - ALERT
    - EMERGENCY
    - EMERG
    - FATAL
    - PANIC
    # severity levels, lowercase (engine matching is case-sensitive).
    #  'error' and 'emergency' are not reserved: an 'error' key often
    #  carries the message itself, and 'emergency' appears as prose.
    - "trace"
    - "debug"
    - "info"
    - "notice"
    - "warn"
    - "warning"
    - "err"
    - "critical"
    - "crit"
    - "alert"
    - "fatal"
    - "panic"
    # severity-equivalent envelope keys
    - "level"
    - "levelname"
    - "severity"
    # envelope time keys
    - "time"
    - "timestamp"
    - "ts"
    # 'msg' and 'message' are message keys (see 'preamble.messageKeys')
    # OpenTelemetry / collector envelope
    - "otel"
    - "otelcol"
    - "opentelemetry"
    - "collector"
    - "contrib"
    - "processor"
    - "processors"
    - "receiver"
    - "receivers"
    - "exporter"
    - "exporters"
    - "pipeline"
    - "pipelines"
    # structured / envelope keys, singular and plural (matching is exact)
    - "service"
    - "services"
    - "instance"
    - "instances"
    - "component"
    - "components"
    - "signal"
    - "signals"
    - "resource"
    - "resources"
    - "scope"
    - "scopes"
    - "attribute"
    - "attributes"
    - "traces"
    - "span"
    - "spans"
    - "flag"
    - "flags"
    - "context"
    - "contexts"
    - "baggage"
    # Kubernetes API / audit envelope
    - "metadata"
    - "namespace"
    - "verb"
    - "uid"
    - "identity"
    - "protocol"
    - "priority"
    - "encoding"
    - "headers"
    - "duration"
    # HTTP / request envelope
    - "path"
    - "status"
    # generic identity keys (very common envelope tokens in JSON logs)
    - "id"
    - "name"
    - "kind"
    - "version"
    - "type"
    # source-location envelope (Go / Java / Python loggers print these)
    - "caller"
    - "func"
    - "function"
    - "file"
    - "line"
    - "module"
    - "package"
    - "class"
    - "method"
    - "stack"
    - "stacktrace"
    # process / runtime context. 'host' and 'app' are not reserved:
    #  'host' is message text in 'no such host', and 'app' is the Flask
    #  logger name that tells one application's statements from another's.
    - "pid"
    - "tid"
    - "process"
    - "thread"
    - "hostname"
    - "application"
    # package-qualifier prefixes (Go / Java loggers embed these in
    # source-path tokens like `com.foo.bar.MyLogger`)
    - "com"
    - "org"
    - "io"
    - "net"
    # tokenizer artifacts
    - "t"

  # 'preamble' describes the text a line carries before its message, such as a
  #  thread name in brackets or 'level=info'. That text stays out of the name.
  #  To learn more see https://doc.log10x.com/run/transform/symbol/#symbolpreambleenclosures
  preamble:

    # 'enclosures' lists bracket pairs. Text inside them before the message stays
    #  out of the name ('main' in '[main] Starting worker'). An open bracket with
    #  no close in the same field is ignored.
    enclosures: ['[]', '()']

    # 'assigners' lists characters that join a key to its value when the key or
    #  the value is quoted ('"level":"info"'). Without quotes, ':' joins nothing,
    #  so 'Error: 13 INTERNAL' and 'INFO:root:Starting' keep their words.
    assigners: ['=', ':']

    # 'bareAssigners' lists characters that also join a key to its value without
    #  quotes, as logfmt writes them ('level=info').
    bareAssigners: ['=']

    # 'joiners' lists separators that chain words into one name without spaces
    #  ('org.apache.kafka.log.LocalLog'). Such a name stays in an event's name. A
    #  line of a multi-line event that holds only such names does not name it.
    joiners: ['.', '::']

    # 'messageKeys' lists keys whose value holds the message. The name starts at
    #  the value's first word outside brackets ('starting' in 'msg=starting').
    messageKeys: [msg, message, body]

  # 'groupLead' picks the line of a multi-line event that names it.
  #  'firstWithMessage' uses the first line that holds message text, so the .NET
  #  logger's 'info: cart.cartstore.ValkeyCartStore[0]' line leaves the name to
  #  the line after it. 'head' always uses the first line.
  groupLead: firstWithMessage

  # ----------------------------- Debug Options -------------------------------

  debug:

    # 'debug' logs how the origin of a line is picked.
    #  To learn more see https://doc.log10x.com/run/transform/symbol/#symboldebugorigins

    # 'loggerName' is the log4j logger the debug output goes to.
    loggerName: "symbolDebugger"

    # 'origins' lists source files in the symbol library, such as 'foo.java', for
    #  which to log how they take part in picking an origin. '*' logs all files.
    origins: [
    #  '*'
    ]

    # 'symbols' lists words of a line, such as 'Could not connect to', for which
    #  to log how their origin is picked. '*' logs all of them.
    symbols: [
    #  '*'
    ]

Options

Specify the options below to configure the Origin selector:

Name Description Category
symbolSequenceReserved Words that do not count as evidence of a source Symbol
symbolPreambleEnclosures Brackets whose text before the message is skipped Symbol
symbolPreambleAssigners Characters that join a quoted key to its value Symbol
symbolPreambleBareAssigners Characters that also join an unquoted key to its value Symbol
symbolPreambleJoiners Separators that chain words into one name Symbol
symbolPreambleMessageKeys Keys whose value holds the message Symbol
symbolGroupLead Line of a multi-line event that names it Symbol
symbolMaxOrigins Most places in code compared for one phrase Symbol
symbolDebugOrigins Source files to log origin choices for Debug
symbolDebugSymbols Words to log origin choices for Debug
symbolDebugLoggerName Logger for debug output Debug

Symbol

symbolSequenceReserved

Words that do not count as evidence of a source.

Type Default Category
List [null, nil, true, false] Symbol

Words that do not count as evidence of where a line comes from, such as true, the, INFO or level. A match made only of these words cannot pick a source.

Before the message these words are skipped, and a key from this list is skipped with its value, such as level=info. Words of the statement itself stay in the name as written.

The default lists null, nil, true and false. The shipped configuration adds common short words, severity levels and keys such as timestamp and thread.

symbolPreambleEnclosures

Brackets whose text before the message is skipped.

Type Default Category
List [[], ()] Symbol

Bracket pairs, each written as its open and close character. Text inside them before the message stays out of the name, such as main in [main] Starting worker.

An open bracket with no close in the same field is ignored, such as the [ of a color code in \033[0;33mWARN.

symbolPreambleAssigners

Characters that join a quoted key to its value.

Type Default Category
List [=, :] Symbol

Characters that join a key to its value when the key or the value is quoted, as in "level":"info" or body: 'Charge request received.'.

A reserved key and its value stay out of the name, and the name starts inside a message key's value. Without quotes, : joins nothing, so Error: 13 INTERNAL and INFO:root:Starting keep their words.

symbolPreambleBareAssigners

Characters that also join an unquoted key to its value.

Type Default Category
List [=] Symbol

Characters that join a key to its value without quotes, as logfmt writes them: level=info, logger=cleanup. They also work with quotes.

symbolPreambleJoiners

Separators that chain words into one name.

Type Default Category
List [., ::] Symbol

Separators that chain words into one name without spaces, such as org.apache.kafka.log.LocalLog or cart.cartstore.ValkeyCartStore, usually a logger, class or file name.

Such a name stays in an event's name, since it often tells two statements apart. A line of a multi-line event that holds only such names does not name the event, see symbolGroupLead.

symbolPreambleMessageKeys

Keys whose value holds the message.

Type Default Category
List [msg, message, body] Symbol

Keys whose value holds the message. The name starts at the value's first word outside brackets, such as starting in msg=starting or Subchannel in "msg":"[core][Channel #1] Subchannel created".

In a record that opens with {, reading starts at the first of these keys that has a value. A message key with no value stays out of the name.

symbolGroupLead

Line of a multi-line event that names it.

Type Default Category
String firstWithMessage Symbol

Which line of a multi-line event names it. The name, origin and skeleton all come from that line.

  • firstWithMessage: the first line that holds message text. A line of only brackets, reserved keys and joined names is passed over. The .NET console logger writes info: cart.cartstore.ValkeyCartStore[0] and then the message, so the second line names the event. A stack trace is named by its exception line.
  • head: always the first line.

symbolMaxOrigins

Most places in code compared for one phrase.

Type Default Category
Number 64 Symbol

The most places in code to compare for one phrase. A phrase such as could not connect to can appear in many files. The first ones in library order, by file name and then by unit name, are compared, so the same library picks the same origin on every node.

Debug

symbolDebugOrigins

Source files to log origin choices for.

Type Default Category
List [] Debug

Source files in the symbol library, such as foo.java, for which to log how they take part in picking an event's origin. Set * for all files.

symbolDebugSymbols

Words to log origin choices for.

Type Default Category
List [] Debug

Words of a line, such as MyClass or Could not connect to, for which to log how their origin is picked. Set * for all of them.

symbolDebugLoggerName

Logger for debug output.

Type Default Category
String consoleOut Debug

The log4j logger that symbolDebugOrigins and symbolDebugSymbols write to.


This unit is defined in symbol/unit.yaml.