Skip to content

Run

Generate a symbol library from source code and binary inputs to enable structured event processing at runtime.

Setup Guide

Follow the steps below. Steps that require customization link to the relevant Configuration section where you can edit on github.dev or locally.

Step 1: Install

Install the Cloud binary flavor:

Step 2: Set Environment Variables

Set these environment variables before running. See path configuration for details.

Variable Description
TENX_CONFIG Path to your configuration directory
TENX_API_KEY Your Log10x API key (get one)
GH_TOKEN GitHub personal access token (create one) (optional)
DOCKER_USERNAME Docker registry username (Docker Hub) (optional)
DOCKER_TOKEN Docker registry token (Docker Hub) (optional)
ARTIFACTORY_TOKEN JFrog Artifactory token (create one) (optional) 
export TENX_CONFIG=/path/to/your/config
export TENX_API_KEY=your-api-key
export GH_TOKEN=your-github-token

See best practices for managing secrets in production.

Step 3: Set Up Input Sources

Specify source code or binary inputPaths to scan.

Clone a repository to the default input path:

git clone https://github.com/open-telemetry/opentelemetry-demo.git \
  $TENX_CONFIG/data/compile/sources

Configure githubPull to pull repos automatically:

githubPull:
  - token: $=TenXEnv.get("GH_TOKEN")
    repos:
      - open-telemetry/opentelemetry-demo

Configure helmChartNames to pull from Helm charts:

helm:
  chartNames:
    - open-telemetry/opentelemetry-demo

The compiler uses Docker CLI and Helm. The Docker image includes these; otherwise ensure helmCommand and dockerCommand are valid.

Step 4: Configure Output
  1. Set outputSymbolFolder for symbol unit files
  2. Set outputSymbolLibraryFile for the final library file
Step 5: Push to GitHub (optional)

Push outputs to GitHub for GitOps workflows, enabling symbol reuse and delivery to edge/cloud apps:

githubPush:
  - token: $=TenXEnv.get("GH_TOKEN")
    repo: myUser/myRepo
    folder: symbols
Step 6: Run

Best for: Local development and testing.

tenx @apps/compiler

Best for: Isolated environments with local configuration.

docker run --rm \
  -v $TENX_CONFIG:/etc/tenx/config/ \
  -e TENX_CONFIG=/etc/tenx/config/ \
  -e TENX_API_KEY=${TENX_API_KEY} \
  -e GH_TOKEN=${GH_TOKEN} \
  -e DOCKER_USERNAME=${DOCKER_USERNAME} \
  -e DOCKER_TOKEN=${DOCKER_TOKEN} \
  ghcr.io/log-10x/pipeline-10x:latest \
  @apps/compiler

Best for: CI/CD pipelines with version-controlled configuration.

docker run --rm \
  -e TENX_API_KEY=${TENX_API_KEY} \
  -e GH_TOKEN=${GH_TOKEN} \
  -e DOCKER_USERNAME=${DOCKER_USERNAME} \
  -e DOCKER_TOKEN=${DOCKER_TOKEN} \
  ghcr.io/log-10x/pipeline-10x:latest \
  '@github={"token": "${GH_TOKEN}", "repo": "my-user/my-repo"}' \
  @apps/compiler
Step 7: Verify

Verify no errors appear in the log file.

Check output files:

Verify symbol files were generated in your configured outputSymbolFolder and outputSymbolLibraryFile paths.

Configuration

To configure the Compiler app, Edit these settings:

Main Config

Main Config

The main config file loads the Compiler app's required modules.

Uncomment selected modules at: compiler/config.yaml.

Edit Online

Edit Compiler Config Locally 

#
# 🔟❎  Compiler app main config

# The compile app produces symbol library files which enable 10x apps to
# transform log/trace events into well-defined TenXObjects at runtime.

# To learn more see https://doc.log10x.com/apps/compiler

# =========================== Bootstrap Compiler ==============================

# To learn more see https://doc.log10x.com/compile/bootstrap

tenx: compile

runtimeName: $=TenXEnv.get("TENX_RUNTIME_NAME", "myCompiler")

# ============================ Load App Modules ===============================

# Uncomment and edit selected config.yaml files in the folders below (e.g., compile/scanners/config.yaml)

include:

# ------------------------------ App settings ---------------------------------

  # Load general app settings:
  - compiler

  # Load gitops settings for pulling additional assets from Github.
  # Edit the gitops/config.yaml file with specific config
  #
  # For additional info see - https://doc.log10x.com/config/github
  - gitops

  # --------------------------- Process Inputs --------------------------------

  # Process input source code and binary files on disk.

  - compile/scanners  # https://doc.log10x.com/compile/scan

  # ---------------------------- Pull Inputs ----------------------------------

  # This application can fetch source/binary files from remote inputs
  # Uncomment and edit config files (e.g. compile/pull/github/config.yaml) to remote pull
  #
  - compile/pull/github       # https://doc.log10x.com/compile/pull/github
  - compile/pull/docker       # https://doc.log10x.com/compile/pull/docker
  - compile/pull/helm         # https://doc.log10x.com/compile/pull/helm
  # - compile/pull/artifactory  # https://doc.log10x.com/compile/pull/artifactory

  # ---------------------------- Link Output ----------------------------------

  # Link output files into a single symbol library artifact for runtime use

  - compile/link              # https://doc.log10x.com/compile/link

  # ---------------------------- Push Output ----------------------------------

  # Push output files to GitHub for use by subsequent compile and run instances

  # - compile/push            # https://doc.log10x.com/compile/push
Pull

Pull

Activate a Repo Fetcher to pull input source code/binary files from remote repositories.

github

Configure GitHub repositories to pull folders from GitHub repositories to scan for symbol content.

Below is the default configuration from: github/config.yaml (* Required Fields).

Edit Online

Edit GitHub repository Config Locally 

# 🔟❎ 'compile' GitHub pull sources configuration

# Configures GitHub repositories folders to pull to disk to scan its contents.
# To learn more see https://doc.log10x.com/compile/pull/github

# For a reference list of industry-standard frameworks, see:
#   modules/pipelines/compile/modules/pull/github/library.yaml

# Set the 10x pipeline to 'compile'
tenx: compile

# ============================== GitHub Options ===============================

githubPull:

    # 'token' specifies the access token for this repo using a shell variable (recommended) or inline
    #  See https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens
  - token: $=TenXEnv.get("GH_TOKEN")  # (❗ EnvVar REQUIRED)

    # 'repo' specifies the repo names to pull from Github (e.g., apache/spark)
    repos: [
      #owner/project
    ]

    # 'branch' specifies the branch to pull within 'repos'. If not set, pulls each repo's default branch
    branch: null

    # 'repoFolders' defines folders within 'repos' to pull. If not specified, pull the entire repo
    folders: [
      # src/main
    ]

docker

Configure the Docker image to pull Docker image files to scan for symbol content.

Below is the default configuration from: docker/config.yaml (* Required Fields).

Edit Online

Edit Docker image Config Locally 

# 🔟❎ 'compile' Docker configuration

# Pull docker images files using the Docker CLI client from container registries to
# scan for symbol values. To learn more see https://doc.log10x.com/compile/pull/docker

# Set the 10x pipeline to 'compile'
tenx: compile

# ============================= Docker Options ================================

docker:

  # 'username' specifies the  user name for authenticating the docker client.
  #  If not set, the docker client is assumed to be pre-authenticated.
  #  To learn more see https://docs.docker.com/reference/cli/docker/login/#username
  username: $=TenXEnv.get("DOCKER_USERNAME") # (❗ EnvVar REQUIRED)

  # 'password' specifies the password argument for authenticating the docker client login command.
  #  This value is written into the docker client's std input via the --password-stdin argument.
  #  To learn more see https://docs.docker.com/reference/cli/docker/login/#password-stdin.
  password: $=TenXEnv.get("DOCKER_TOKEN")  # (❗ EnvVar REQUIRED)

  # 'command' specifies the location of docker cli https://docs.docker.com/reference/cli/docker/ program
  #command: $=TenXString.includes(TenXEnv.get("os.name"), "Windows") ? "C:/Program Files/Docker/Docker/resources/bin/docker.exe":"/usr/local/bin/docker"

  # 'images' specify images to pull and export to matching .tar files to scan
  images: [
  #   docker.io/grafana/grafana:11.1.0 
  ]

  # 'remove' specifies whether to remove exported images once their contents are extracted.
  #  To learn more see https://docs.docker.com/reference/cli/docker/image/rm
  remove: false

  # 'githubRepoToken' defines an access token for pulling Github repos referenced
  # by a target image's 'org.opencontainers.image.source' property.
  githubRepoToken: $=TenXEnv.get("GH_TOKEN")  # (❗ EnvVar REQUIRED)

helm

Configure the Helm chart to pull GitHub repositories and Docker images to scan for symbol content from Helm charts.

Below is the default configuration from: helm/config.yaml (* Required Fields).

Edit Online

Edit Helm chart Config Locally 

# 🔟❎ 'compile' k8s Helm scanner configuration

# Configures the k8s Helm chart symbol scanner which pulls referenced GitHub source repositories and
# Docker container images to scan for symbol values.

# To learn more see https://doc.log10x.com/compile/pull/helm

# For a reference list of common infrastructure charts, see:
#   modules/pipelines/compile/modules/pull/helm/library.yaml

# Set the 10x pipeline to 'compile'
tenx: compile

# =============================== Helm Options ===============================

helm:

  # Charts can be specified as Helm chart names OR as local chart files.

  # 'chartNames' specifies a list of remote chart names to scan for values and template files.
  chartNames: [
    # open-telemetry/opentelemetry-demo,
    # ingress-nginx/ingress-nginx,
  ]

  # 'chartFiles' specifies a list of local chart template and values.yaml files on disk.
  #  File paths may be relative to the 'includePaths' list. If canonical, enclosing folders must added to 'includePaths'.
  chartFiles: [
    # charts/opentelemetry-demo/values.yaml
  ]

  pull:

    # 'dockerImages' specifies whether to pull docker container images referenced by input charts
    #  To learn more see https://doc.log10x.com/compile/pull/docker
    dockerImages: true

    github:

      # 'repos' specifies whether to pull GitHub repos referenced by 'org.opencontainers.image.source' annotations
      #  To learn more see https://github.com/opencontainers/image-spec/blob/v1.0.1/annotations.md#pre-defined-annotation-keys
      repos: true

      # 'token' specifies the GitHub access token for pulling from referenced repos
      #  To learn more see: https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens
      token: $=TenXEnv.get("GH_TOKEN")  # (❗ EnvVar REQUIRED)

artifactory

Configure Artifactory repositories to pull files from Artifactory repositories to scan for symbol content.

Below is the default configuration from: artifactory/config.yaml (* Required Fields).

Edit Online

Edit Artifactory repository Config Locally 

# 🔟❎ 'compile' Artifactory configuration

# Pull artifacts (e.g., Java archives, .NET assemblies) from remote Artifactory instances to 
# scan for symbol values. To learn more see https://doc.log10x.com/compile/pull/artifactory

# Set the 10x pipeline to 'compile'
tenx: compile

# ============================= Artifactory Options ===========================

artifactory:

  # 'token' specifies an Artifactory API access token.
  #   To learn more see https://jfrog.com/help/r/jfrog-platform-administration-documentation/access-tokens
  - token: $=TenXEnv.get("ARTIFACTORY_TOKEN") # (❗ REQUIRED)

  # 'instance' specifies the of location of the remote Artifactory instance (e.g., https://demo.jfrog.io/artifactory)
    instance: "" # (❗ REQUIRED)

  # 'repo' specifies the target Artifactory repository (e.g., pypi-local)
    repo: "" # (❗ REQUIRED)

  # 'files' specifies files to pull from the target Artifactory repository  # (❗ REQUIRED)
    files: [
      # dist/jfrog-pypi-example-1.0.0.tar.gz
    ] 

    # 'folders' specifies folders paths within the target Artifactory instance repository to pull # (❗ REQUIRED)
    folders: [ 
    ]

    # 'recursive' specifies whether to pull folders recursively (i.e., pull sub-folders as well).
    recursive: true
Scan

Scan

Configure the Input Scanner to capture symbol values from source code/binary files and link to an output symbol library.

Below is the default configuration from: scanners/config.yaml.

Edit Online

Edit Input Scanner Config Locally 

# 🔟❎ 'compile' symbol scanner config

# Configure input and output options for the compile pipeline.
# To learn more see https://doc.log10x.com/compile/scan

tenx: compile

# ========================== Configure I/O Paths ==============================

# 'inputPaths' lists folders on disk containing source code and binary files to scan.
#  To learn more see https://doc.log10x.com/compile/scan/#inputpaths
inputPaths:
  - $=path("data/compile/sources", "")   # Default to none ("") if source folder not found
  # - <path-to-input-files>              # uncomment to specify additional input paths

# 'outputSymbolFolder' specifies the folder for output symbol unit files.
#  To learn more see https://doc.log10x.com/compile/scan/#outputsymbolfolder
outputSymbolFolder: $=TenXEnv.get("TENX_OUTPUT_SYMBOL_FOLDER", path("data/shared/symbols", "<tenx.io.tmpdir>")) # Default to 10x temp dir
Link

Configure the Symbol library linker to link symbol unit files to an output symbol library file.

Below is the default configuration from: link/config.yaml.

Edit Online

Edit Symbol library linker Config Locally 

# 🔟❎ 'compile' merge configuration

# Combine symbol units into a single output symbol library file.
# To learn more see https://doc.log10x.com/compile/link/

# Set the 10x pipeline to 'compile'
tenx: compile

# ============================ Symbol Library Output ==========================

# 'outputSymbolLibraryFile' specifies location of output symbol library file which
#  links the contents of 'outputSymbolFolder' to a single .json artifact and a
#  compressed .zip version for use at run time by 10x 'run' pipelines.
#  If a local output dir exists write to it, otherwise use the pipeline's temp folder.
#  To learn more see https://doc.log10x.com/compile/link

outputSymbolLibraryFile: $=TenXEnv.get("TENX_OUTPUT_SYMBOL_LIBRARY_FILE", path("data/shared/symbols", "<tenx.io.tmpdir>") + "/" + TenXEnv.get("runtimeName", "symbols") + ".10x.tar")


# ============================ Symbol Filter options ==========================

# The options below control which symbol files and values are merged into 'outputSymbolLibraryFile'

symbol:

  # 'fileNameFilters' — exclude test files from symbol library
  fileNameFilters:

    # ── Java ───────────────────────────────────────
    - ^.*(Test|Tests|TestCase|TestSuite)\.java$     # *Test.java, *TestCase.java, *TestSuite.java
    - ^Test.*\.java$                                # TestFoo.java (legacy)

    # ── JavaScript / TypeScript ────────────────────
    - ^.*\.(test|spec)\.(js|jsx|ts|tsx|mjs|cjs)$    # *.test.js, *.spec.tsx

    # ── Python ─────────────────────────────────────
    - ^test_.+\.py$                                 # test_utils.py
    - ^.+_test\.py$                                 # utils_test.py
    - ^test(s|case|cases|suite|suites)?\.py$        # test.py, tests.py, testcase.py, testcases.py, testsuite.py

    # ── Scala ──────────────────────────────────────
    - ^.*(Spec|Test|Suite)\.scala$                  # *Spec.scala, *Test.scala, *Suite.scala

    # ── C# ─────────────────────────────────────────
    - ^.*(Test|Tests|TestCase|TestSuite)\.cs$       # *Test.cs, *TestCase.cs, *TestSuite.cs

    # ── Go ─────────────────────────────────────────
    - ^.*_test\.go$                                 # *_test.go
    # Note: testdata.go is usually a data file, not test code → not filtered here

    # ── C++ ───────────────────────────────────────
    - ^.*(_test|Test|Tests)s?\.(cpp|cc|cxx|hpp|h|hxx)$  # *_test.cpp, *Test.hpp, *Tests.cpp

  # 'folderFilters' — patterns to exclude files by folder
  # Plain name (e.g., "test") → matches direct parent folder only
  # Regex (e.g., "[/\\]vendor[/\\]") → matches anywhere in full path
  folderFilters:
    - test           # direct parent is "test"
    - tests          # direct parent is "tests"
    - __tests__      # Jest convention
    - spec           # RSpec/Jasmine convention
    - specs          # RSpec/Jasmine convention
    - unit           # unit test folder
    - integration    # integration test folder
    - e2e            # e2e tests (Cypress, Playwright)
    - end-to-end     # alternate e2e naming
    - acceptance     # acceptance test folder
    - testdata       # Go test data folder
    - fixtures       # test fixtures folder
    - mocks          # mock implementations folder
    - "[/\\\\]vendor[/\\\\][a-z]+\\.[a-z]"   # Go vendored deps (vendor/domain.tld pattern)

  # 'types' controls the types of symbols written into 'outputSymbolLibraryFile'
  #  for reflective use at runtime.
  #  Narrowing the this list reduces the size of the output 10x symbols file. Possible values:
  types:
    - class   # source code class/interface name (e.g. 'AcmeLogic')
    - enum    # source code enum/literal name (e.g. 'severity: {error,warn,debug}')
    - log     # source code print format passed to a logging method/stream (e.g. 'err("could not connect to %s",..');
    - exec    # string value extracted from a binary executable input file (e.g. via 'strings' nix command)
    # - method # code method/function name (e.g. 'foo')
    # - const  # string constant assigned into a variable or method arg (e.g. 'greeting = "hello world"')
    # - text    # string value, usually read from a config file (e.g. JSON/XML/YAML field name)
    # - package # source code package/namespace/module name (e.g. 'com.acme.core')
Push

Push

Configure GitHub push committers to push output symbol unit files to GitHub.

Below is the default configuration from: push/config.yaml (* Required Fields).

Edit Online

Edit GitHub push committer Config Locally 

# 🔟❎ 'compile' GitHub configuration

# Push output symbol unit files for reuse by subsequent compile cycles.
# To learn more see https://doc.log10x.com/compile/push/

# Set the 10x pipeline to 'compile'
tenx: compile

# =============================== GitHub Options ==============================

githubPush:

    # 'token' specifies the access token for pushing files to the target repo, read from env or set inline.
    #  To learn more see: https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens
  - token: $=TenXEnv.get("GH_TOKEN") # (❗ EnvVar REQUIRED)

    # 'repo' specifies the repo name to pull in Github
    #  including the username of the owner (e.g., apache/spark)
    repo: null # (❗ REQUIRED)

    # 'branch' specifies the branch to use when pushing into this repo.
    #  If not specified, push to the default repo branch
    branch: null

    # 'folder' specifies a folder within the target repo to push files, if not specified push to root
    folder: null

    # 'filter' specifies a a regex pattern a file on disk must match to be pushed to the target repo.
    filter: null

Advanced Settings

To configure advanced options (optional) for the Compiler app, Edit these settings:

Bootstrap

Bootstrap

Configure the Pipeline Bootstrapper to authenticate the log10x account and launch a target pipeline.

Below is the default configuration from: bootstrap/config.yaml.

Edit Online

Edit Pipeline Bootstrapper Config Locally 

# 🔟❎ compile bootstrap configuration

# This config file specifies bootstrap options for the compile pipeline.
# To learn more see https://doc.log10x.com/compile/bootstrap

# To learn more see https://doc.log10x.com/config/

tenx: compile

# =============================== Launch Settings ==============================

# 'apiKey' specifies an api key used authenticating against the 10x service
apiKey: $=TenXEnv.get("TENX_API_KEY", "NO-API-KEY")

# 'includePaths' specifies folders on disk for resolving relative config file/folder references in addition to the working folder
includePaths: []

# 'quiet' disables printing version information to the console.
# quiet: true

# 'jarFiles' specifies .jar files to dynamically load for use by compile, input and output API extensions.
jarFiles: []

# 'metricEndpoint' specifies the Prometheus endpoint to report usage/health metrics (enterprise version only).
# metricEndpoint: https://prometheus.log10x.com/api/v1/write

# specifies a list of launch arguments that are disallowed from either command line or user config files.
disabledArgs: []
Github Sync

Github Sync

Below is the default configuration from: gitops/config.yaml (* Required Fields).

Edit Online

Edit config.yaml Locally 

# 🔟❎ GitOps configuration

# This config file pulls symbol, config and lookup files from a remote repo to
# enable centralized configuration management via GitHub.

# To learn more see https://doc.log10x.com/config/github

# The target 10x pipeline to run (e.g., run, compile) is specified in the config files pulled from the repo
tenx: any

include:

  - source: github

    options:
      # 'enabled' controls whether to enable this pull config
      enabled: false # (❗ REQUIRED)

      # 'token' specifies an access token for accessing the Github repo.
      #  To learn more see https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens
      token: $=TenXEnv.get("GH_TOKEN")

      # 'repo' specifies the GitHub user/repo from which to pull
      repo: null # (❗ REQUIRED)

      # 'branch' on 'repo' (if null, uses default repo default branch)
      branch: main

       # 'paths' specifies Glob patterns within 'repo' to pull and extract to a temp folder.
      paths:
        - test
        # - config/*.yaml      # https://doc.log10x.com/config/yaml
        # - symbols/*.10x.tar  # https://doc.log10x.com/run/symbol
        # - lookups/*.csv      # https://doc.log10x.com/api/js/#TenXLookup
        # - scripts/*.js       # https://doc.log10x.com/api/js

      # 'syncInterval' specifies an interval by which check to check the remote branch for updates (null to disable)
      #  This interval works in conjunction with: https://doc.log10x.com/run/reload
      syncInterval: 5min
Scan

Scan

Configure the Input Scanner to capture symbol values from source code/binary files and link to an output symbol library.

advanced

Configure the Input Scanner to capture symbol values from source code/binary files and link to an output symbol library.

Below is the default configuration from: advanced/config.yaml.

Edit Online

Edit Input Scanner Config Locally 

# 🔟❎ 'compile' symbol scanner advanced config

# Configure input and output options for the compile pipeline.
# To learn more see https://doc.log10x.com/compile/scan/

# Set the 10x pipeline to 'compile'
tenx: compile

inputPaths: []

# ============================ Advanced options ===============================

# ----------------------------- Tokenizer options -----------------------------

# Tokenization options provide control over how the symbol scanners parse source and config file input.
# To learn more https://doc.log10x.com/compile/scan/#tokenize

# 'tokenDelims' defines which characters break a string of event characters into tokens to 
#  classify as symbols or variables.
tokenDelims: "<>|. ][:-+/*=\\,{}_();'\"$\t\n@"

# 'fileExtStringFormats' controls which characters identify the start of a format specifier (https://en.wikipedia.org/wiki/Printf#Format_specifier). 
fileExtStringFormats: 
  - .java:%
  - .py:{
  - .js:${
  - .php:$
  - .swift:\(
  - .kt:$
  - .rb:#{
  - .ex:#{
  - .c:%
  - .cpp:%
  - .cs:{
  - .go:%
  - .rs:{
  - .ts:${
  - .pl:$
  - .sh:$
  - .r:%
  - .m:%
  - .scala:$


maxSymbol:

# 'unitsPerToken' controls the maximum number of symbol units to load from the 
#  pipeline's symbol library when searching for the origin of a specific TenXTemplate symbol. 
  unitsPerToken: 128

# ----------------------------- Subprocess options ----------------------------

# Subprocess options control whether scanners execute in the current 10x Engine instance
# or via a sub-process to provide better isolation over parsing of complex source code ASTs. 

# To learn more https://doc.log10x.com/compile/scan/#subprocess_1

# 'unitsToScanPerBatch' sets the maximum number of input files to scan per subprocess instance.
#  When this threshold is exceeded the current subprocess terminated and a new one is spawned until scan is complete.
unitsToScanPerBatch: 200

scan:

# ----------------------------- Threading  options ----------------------------

  # 'threadPoolSize'controls the maximum number of threads to concurrently
  #  process input source/binary files. If the value is between 0 and 1, the number calculates 
  #  a percentage of available cores (e.g., 0.5 = use up to 50% of available cores).
  #  If the value is >= 1, the value sets a fixed number of threads (e.g., 10 = 10 threads).
  #  If the value is 1, allocate a single thread to process input files.

  # To learn more see https://doc.log10x.com/compile/scan/#parallel

  threadPoolSize: "0.5"

  # 'scanUnitTimeout' sets the timeout interval when scanning a source/binary input
  #  file before dropping it. 
  unitTimeout: 30s

  # 'scanOperationTimeout' sets the timeout interval for the entire scanning operation
  #  before terminating it.
  operationTimeout: 10m

# -------------------------------- Debug Option -------------------------------

# The settings below provide control over what information is logged
# to console/file when scanning input source code/text/binary files for symbol values.

# These options are primarily used when adding support for a new programming language
# syntax via the 'ANTLR' scanner, text file formats via the Jackson 'text' scanner
# or an external command using the 'executable' scanner.

  debug:

  # 'origins' lists names of input source/binary files 
  #  (e.g. .class, .js, .yaml, .exe) for which debug information
  #  is printed to the console via 'consoleOut' log4j logger.

  #  Printed output is dependent on the scanner assigned to process the input file  
  #  and usually includes the AST of the input file from which tokens are selected.

  #  This information can be used when debugging which symbols are extracted
  #  from an input file in the context of an 'antlr', 'text' or 'executable' scanner. 
  #  Set to '*' to match all input files. 
    origins: [
  #   '*'
    ]

  # 'symbols' lists the values of nodes within the AST of a
  #  target input file (e.g. MyClass, MyEnum, myFunc, ..) for which 
  #  debug information is printed to the console via the 'consoleOut' logger.
  #  This includes source context (e.g. class, enum) and path within the AST 
  #  in which the node was detected. Set to '*' to match all AST nodes. 
    symbols: [
     #'*'
    ]

  # 'inProcess' controls whether to scan source/binary input files from within the current
  #  10x process or launch a sequential series subprocesses for better isolation
  # inProcess: true  

  # 'force' enables scanning of source/binary input files even if a matching
  #  symbol unit was found in 'inputPaths'. This flag is useful when debugging
  #  an Antlr/text/exec scanner using 'debugOrigins' and 'debugSymbols'
  #  to ensure a scan takes place even if a matching symbol unit is found.

  # force: false

pattern

Configure Regex pattern scanners to extract symbol values from a files using regular expressions.

Below is the default configuration from: pattern/config.yaml.

Edit Online

Edit Regex pattern scanner Config Locally 

# 🔟❎ 'compile' pattern quoted strings configuration

# This pattern symbol scanner extracts single and double quoted string values
# from target files. This pattern is used a default for capturing symbol values
# from source files for which no ANTLR grammar or built-in scanner is available.

# Set the 10x pipeline to 'compile'
tenx: compile

# ============================== Pattern Options ==============================

pattern: 

    # 'fileFilter' is set as a sink for prominent source code files. Available built-in and ANTLR
    #  scanners will take precedence over this pattern scanner
  - fileFilter: \b\w+\.(java|cs|php|cpp|cxx|c|h|H|hxx|cc|hh|py|go|js|rs|bash|scala|sc|ts|rb|kt|lua|groovy|swift)\b

    extractQuotes: true

    # 'lineFilter' defines a regex pattern applied to each line within a target file.
    #  Each matching region will be extracted as symbol value
    lineFilter: null

    # 'matchContext' sets the source context to assign any symbols collected using this this pattern selector. 
    #  For possible values, see https://doc.log10x.com/run/transform/symbol/#contexts
    matchContext: method_invoke

    # 'unescapeScheme' specifies an unescape algorithm to apply to matching sequences of patternLineFilter. 
    #  Supported values: [json,java,xml,html,javaScript].
    unescapeScheme: java

Cpp

Configure the ANTLR scanner to extract symbols from any programming language using the ANTLR framework.

Below is the default configuration from: antlr/cpp.yaml.

Edit Online

Edit ANTLR scanner Config Locally 

# 🔟❎ 'compile' C++ ANTLR symbol scanner configuration

# The 'cpp' configuration instructs the ANTLR scanner which symbol values (e.g. class/func names, code constants)
# to extract from c++ source files.  

# Set the 10x pipeline to 'compile'
tenx: compile

# ============================== ANTLR Options ================================

antlr:

  # The 'antlrLang' module is defined in: https://doc.log10x.com/compile/scanner/antlr/langs
  # These values define the ANTLR grammar .g4 file used to
  # to tokenize and construct an AST for the target cpp file

  lang: cpp

  fileExt:
    - .cpp
    - .cxx
    - .c++
    - .c
    - .h
    - .H
    - .hpp
    - .hxx
    - .h++
    - .cc
    - .hh

  grammarFile: grammar/CPP14Parser.g4

  lexerFile: grammar/CPP14Lexer.g4

  rootRule: translationUnit

  reserved:
    - '::'
    - '<<'
    - inline
    - namespace
    - <
    - class
    - '>'

  # 'lineFilters' specifies a list of regex patterns for skipping input lines 
  lineFilters:
    - ^\s*(//.*|/\*.*\*/)$
    - ^.*\bnew\(\);\b.*$

  # The 'rule' list defines the nodes within a cpp AST tree that are captured
  # and added to the output symbol unit.
  # To learn more see: https://doc.log10x.com/compile/scanner/antlr/rules/
  rule:
    - name: enumerator
      lang: cpp
      context: enum
      recursive: false
      capture: allSymbols

    - name: blockDeclaration
      lang: cpp
      context: method_invoke
      recursive: true
      capture: literalsOnly

    - name: enumbase
      lang: cpp
      context: enum
      recursive: true
      capture: literalsOnly

    - name: classHeadName
      lang: cpp
      context: class
      recursive: true
      capture: allSymbols

    - name: enumSpecifier
      lang: cpp
      context: class
      recursive: false
      capture: literalsOnly
      condition: ^enumHead$
      tag: EnumHead

    - name: enumHead
      lang: cpp
      context: class
      recursive: false
      capture: allSymbols
      ifTag: EnumHead

    - name: functionDefinition
      lang: cpp
      context: method_decl
      recursive: false
      capture: allSymbols
      condition: ^functionDefinition$
      tag: FunctionDefinition

    - name: declaratorid
      lang: cpp
      context: method_decl
      recursive: true
      capture: allSymbols
      ifTag: FunctionDefinition

    - name: jumpStatement
      lang: cpp
      context: var_assign
      recursive: true
      capture: literalsOnly

    - name: functionBody
      lang: cpp
      context: method_invoke
      recursive: false
      capture: literalsOnly
    # condition: ^functionBody$
    # tag: FunctionBody

    - name: statement
      lang: cpp
      context: method_invoke
      recursive: true
      capture: literalsOnly

    - name: classSpecifier
      lang: cpp
      context: class
      recursive: false
      capture: literalsOnly
      subRule: memberSpecification

    - name: staticAssertDeclaration
      lang: cpp
      context: var_assign
      recursive: true
      capture: literalsOnly

    - name: parameterDeclarationList
      lang: cpp
      context: method_invoke
      recursive: false
      capture: literalsOnly
    # condition: ^parameterDeclarationList$
    # tag: ParameterDeclarationList

    - name: enumeratorDefinition
      lang: cpp
      context: enum
      recursive: false
      capture: allSymbols

    - name: namespaceDefinition
      lang: cpp
      context: package
      recursive: false
      capture: allSymbols

    - name: constantExpression
      lang: cpp
      context: var_assign
      recursive: true
      capture: literalsOnly

    - name:  templateArgumentList
      lang: cpp
      context: method_invoke
      recursive: false
      capture: literalsOnly
    # condition: ^templateArgumentList$
    # tag: TemplateArgumentList

Csharp

Configure the ANTLR scanner to extract symbols from any programming language using the ANTLR framework.

Below is the default configuration from: antlr/csharp.yaml.

Edit Online

Edit ANTLR scanner Config Locally 

# 🔟❎ 'compile' C# ANTLR symbol scanner configuration

# The 'csharp' configuration instructs the ANTLR scanner which symbol values (e.g. class/func names, code constants)
# to extract from c# source files.  

# Set the 10x pipeline to 'compile'
tenx: compile

# ============================== ANTLR Options ================================

antlr:

  # The 'antlrLang' module is defined in: https://doc.log10x.com/compile/scanner/antlr/langs
  # These values define the ANTLR-generated parser class used to
  # to tokenize and construct an AST for the target c# file.

  lang: csharp

  fileExt:
    - .csx
    - .cs

  parserClass: com.log10x.antlr.generated.csharp.CSharpParser

  lexerClass: com.log10x.antlr.generated.csharp.CSharpLexer

  # 'charStreamClass' is set to remove language elements introduced later than V6.0 which is the latest release supported by this grammar.
  #  To learn more see https://github.com/antlr/grammars-v4/tree/master/csharp#readme
  charStreamClass: com.log10x.antlr.parsers.CSharpDowngrader

  rootRule: compilation_unit

  reserved:
    - namespace
    - interface
    - class
    - enum

  # 'lineFilters' specifies a list of regex patterns for skipping input lines 
  lineFilters:
    - ^\s*(//.*|/\*.*\*/)$

    # Remove unsupported C# v10 File scoped namespaces
    - ^namespace.*;$

    # Remove unsupported C# v9 Target-typed new operator
   # - "(?<!\\w)new\\s*\\(\\s*\\)\\s*;"

    - ".*new\\(.*"

  # The 'rule' list defines the nodes within a csharp AST tree that are captured
  # and added to the output symbol unit.
  # To learn more see: https://doc.log10x.com/compile/scanner/antlr/rules/
  rule:
    - name: interface_definition
      lang: csharp
      context: class
      recursive: false
      capture: allSymbols
      subRule: identifier

    - name: enum_member_declaration
      lang: csharp
      context: enum
      recursive: true
      capture: allSymbols

    - name: enum_definition
      lang: csharp
      context: class
      recursive: false
      capture: allSymbols
      subRule: identifier

    - name: namespace_body
      lang: csharp
      context: package
      recursive: true
      capture: literalsOnly

    - name: method_invocation
      lang: csharp
      context: method_invoke
      recursive: true
      capture: literalsOnly

    - name: object_creation_expression
      lang: csharp
      context: method_invoke
      recursive: true
      capture: literalsOnly

    - name: interpolated_regular_string_part
      lang: csharp
      context: method_invoke
      recursive: false
      capture: allSymbols
      subRule: interpolated_regular_string_part

    - name: namespace_declaration
      lang: csharp
      context: package
      recursive: true
      capture: allSymbols

    - name: statement_list
      lang: csharp
      context: method_invoke
      recursive: true
      capture: literalsOnly

    - name: constant_declaration
      lang: csharp
      context: var_assign
      recursive: true
      capture: literalsOnly

    - name: method_declaration
      lang: csharp
      context: method_decl
      recursive: true
      capture: allSymbols
      subRule: identifier

    - name: method_body
      lang: csharp
      context: method_decl
      recursive: false
      capture: literalsOnly

    #- name: method_member_name
    #  lang: csharp
    #  context: method_decl
    #  recursive: true
    #  capture: allSymbols

    - name: local_variable_initializer
      lang: csharp
      context: var_assign
      recursive: true
      capture: literalsOnly

    - name: returnStatement
      lang: csharp
      context: var_assign
      recursive: true
      capture: literalsOnly

    - name: class_definition
      lang: csharp
      context: class
      recursive: false
      capture: allSymbols
      subRule: identifier

    - name: attribute_argument
      lang: csharp
      context: annotation_invoke
      recursive: true
      capture: literalsOnly

pythonAST

Configure the CPython AST scanner to scan Python source code for symbol values.

Below is the default configuration from: pythonAST/default.yaml.

Edit Online

Edit CPython AST scanner Config Locally 

# 🔟❎ 'compile' python symbol scanner settings

# The Python AST symbol scanner leverages the Python native .py parsing engine.
# to launch the 'astpretty' script from https://github.com/asottile/astpretty 
# to parse the contents of an input .py file. 

# If a python runtime is unavailable when executing the 'compile' pipeline,
# the 'pythonASTPath' value can be set to null or empty string to default 
# to the ANTLR symbol scanner defined in 'python.yaml'

# 'pythonPath' specifies the command for launching a Python run-time. 
#  A test script verifies a Python run-time is available. If the test script is unsuccessful,
#  the scanner is disabled. 
pythonPath: python3

archive

Configure the Archive scanner to scan compressed archives for enclosed files containing symbol values.

Below is the default configuration from: archive/default.yaml.

Edit Online

Edit Archive scanner Config Locally 

# 🔟❎  'compile' archive symbol scanner config

# The archive'scanner iterates through the contents of file archives
# to seek source/binary files whose contents to scan into
# the output symbol files.

# Set the 10x pipeline to 'compile'
tenx: compile

archiveExtensions:
  - .zip
  - .jar
  - .war
  - .ear

  - .7s
  - .gz
  - .br
  - .lz4
  - .lzma

  - .tar
  - .tar.br
  - .tar.gz
  - .tar.bz2
  - .tar.lz4
  - .tar.lzma

logMethods

Configure the Log method/stream definitions to define logger method and output stream names.

Below is the default configuration from: logMethods/default.yaml.

Edit Online

Edit Log method/stream definitions Config Locally 

# 🔟❎ 'compile' log methods configuration

# Set the 10x pipeline to 'compile'
tenx: compile

# ============================ Log Options =============================

log:

  # 'methods' lists known method names collected from common logging
  #  frameworks used to indicate source code logging events to stream at runtime.

  #  Identifying those points in source code that perform logging operations
  #  and marking them within an output symbol unit file (*.json)
  #  allows the 'run' pipeline to correlate each TenXObject structured from the event
  #  data back to its origin in the codebase.

  #  To learn more about symbol contexts and their use, see https://doc.log10x.com/run/transform/symbol/#contexts
  methods:

    # Assertions are not enabled by default as they are less likely to appear in prod logs
    # - Assert
    # - assert
    # - assertEquals
    # - assertEqual

    - exception

    - Log
    - LOG
    - log
    - BOOST_LOG_TRIVIAL

    - trace
    - Trace
    - log_trace
    - logTrace
    - TRACE
    - LOG_TRACE

    - debug
    - Debug
    - logDebug
    - log_debug
    - DEBUG
    - LOG_DEBUG
    - Debugf

    - info
    - Info
    - log_info
    - logInfo
    - INFO
    - LOG_INFO
    - information
    - LogInformation
    - InfoF

    - warn
    - Warn
    - log_warn
    - logWarn
    - WARN
    - LOG_WARN
    - WarnF

    - warning
    - logWarning
    - LOG_WARNING

    - error
    - Error
    - logError
    - log_error
    - ERROR
    - LOG_ERROR
    - ErrorF

    - err

    - fatal
    - Fatal
    - log_fatal
    - logFatal
    - FATAL
    - LOG_FATAL
    - FatalF

    - critical
    - Panic

  # 'methodRegex' specifies an optional regex that will be applied
  #  to a scanned source code method/function name. If matched, the method
  #  will be considered to be a 'logging method' (see above).
  #  For example: ^(log|Log)|(Log|log)$
  methodRegex: null

  # 'streams' defines which source code variables within
  #  this ANTLR syntax are treated as output streams (e.g. stdout, stderr, cerr).
  #  Literal constants (e.g. "error connecting to {}") passed to a variable
  #  listed below (e.g. 'cout') will be assigned a 'logger_method_invoke'
  #  symbol source context.
  streams:

  # - assert
    - cout
    - cerr
    - Error
    - Debug
    - err
    - out
    - Console

Go

Configure the ANTLR scanner to extract symbols from any programming language using the ANTLR framework.

Below is the default configuration from: antlr/go.yaml.

Edit Online

Edit ANTLR scanner Config Locally 

# 🔟❎ 'compile' Go ANTLR symbol scanner configuration

# The 'go' configuration instructs the ANTLR scanner which symbol values (e.g. class/func names, code constants)
# to extract from GO source files.  

# Set the 10x pipeline to 'compile'
tenx: compile

# ============================== ANTLR Options ================================

antlr:

  # The 'antlrLang' module is defined in: https://doc.log10x.com/compile/scanner/antlr/langs
  # These values define the ANTLR-generated parser class used to
  # to tokenize and construct an AST for the target go file.
  lang: go

  fileExt:
    - .go

  parserClass: com.log10x.antlr.generated.golang.GoParser

  lexerClass: com.log10x.antlr.generated.golang.GoLexer

  rootRule: sourceFile

  reserved:
    - package
    - func

  # 'lineFilters' specifies a list of regex patterns for skipping input lines 
  lineFilters:
    - ^\s*(//.*|/\*.*\*/)$

  # The 'rule' list defines the nodes within a go AST tree that are captured
  # and added to the output symbol unit.
  # To learn more see: https://doc.log10x.com/compile/scanner/antlr/rules/
  rule:
    - name: constSpec
      lang: go
      context: enum
      recursive: false
      capture: allSymbols
      subRule: identifierList

    - name: returnStmt
      lang: go
      context: var_assign
      recursive: true
      capture: literalsOnly

    - name: assignment
      lang: go
      context: var_assign
      recursive: true
      capture: literalsOnly

    - name: expressionStmt
      lang: go
      context: method_invoke
      recursive: true
      capture: literalsOnly

    - name: packageClause
      lang: go
      context: package
      recursive: false
      capture: allSymbols

    - name: arguments
      lang: go
      context: method_invoke
      recursive: true
      capture: literalsOnly

    - name: typeSpec
      lang: go
      context: class
      recursive: false
      capture: allSymbols

    - name: functionDecl
      lang: go
      context: method_decl
      recursive: false
      capture: allSymbols

    - name: methodDecl
      lang: go
      context: method_decl
      recursive: false
      capture: allSymbols

Jackson

Configure Text file scanners to scan text/binary files for symbol values using the Jackson library.

Below is the default configuration from: text/jackson.yaml.

Edit Online

Edit Text file scanner Config Locally 

# 🔟❎ 'compile' text symbol scanner configuration

# The 'text' scanner parses text files for symbol values. It utilizes
# the Jackson parser library to parse any text/binary format it supports.
# This includes formats such json, yaml, xml, ini, protobuf and more.

# The configuration below is added by default to the 10x 'compile' pipeline.
# Even so, if another text scanner is defined via the 'textScanners' options group
# whose 'fileNameFilter' matches that of the current target input file,
# it will take precedence over the text scanners defined below.
# To learn more about text scanner options below, see:
# https://doc.log10x.com/compile/scanner/text

# Set the 10x pipeline to 'compile'
tenx: compile

# =============================== Text Options ================================

text:

  - parserName: text
    fileNameFilter: '^.*\.(csv|txt|properties|csv|tsv|ini|conf|sh|log|out)$'
    maxLines: 500
    lineOffset: 0
    allowDigits: false
    minLength: 2
    maxLength: 50

  - parserName: json
    fileNameFilter: '^.*\.(json)$'
    parserFactoryClass: com.fasterxml.jackson.core.JsonFactory
    scanFieldValues: true

  - parserName: yaml
    fileNameFilter: '^.*\.(yml|yaml)$'
    parserFactoryClass: com.log10x.eng.scanner.text.TextYamlFactory
    scanFieldValues: true

  - parserName: xml
    fileNameFilter: '^.*\.(xml|xsd)$'
    parserFactoryClass: com.fasterxml.jackson.dataformat.xml.XmlFactory
    scanFieldValues: true

Java

Configure the ANTLR scanner to extract symbols from any programming language using the ANTLR framework.

Below is the default configuration from: antlr/java.yaml.

Edit Online

Edit ANTLR scanner Config Locally 

# 🔟❎ 'compile' Java ANTLR symbol scanner configuration

# The 'java' configuration instructs the ANTLR scanner which symbol values (e.g. class/func names, code constants)
# to extract from Java source files.  

# IMPORTANT: while set to enabled by default, the dedicated java scanner utilizes 
# takes precedence over this scanner unless disabled.
# To learn more see https://doc.log10x.com/compile/scanner/javaParser

# Set the 10x pipeline to 'compile'
tenx: compile

# ============================== ANTLR Options ================================

antlr:

  # The 'antlrLang' module is defined in: https://doc.log10x.com/compile/scanner/antlr/langs
  # These values define the ANTLR-generated parser class used to
  # to tokenize and construct an AST for the target Java file.

  lang: java

  fileExt:
    - .java

  parserClass: com.log10x.antlr.generated.java.Java9Parser

  lexerClass: com.log10x.antlr.generated.java.Java9Lexer

  rootRule: compilationUnit

  reserved:
    - package
    - interface
    - class
    - enum

  # 'lineFilters' specifies a list of regex patterns for skipping input lines 
  lineFilters:
    - ^\s*(//.*|/\*.*\*/)$

  # The 'rule' list defines the nodes within a Java AST tree that are captured
  # and added to the output symbol unit.
  # To learn more see: https://doc.log10x.com/compile/scanner/antlr/rules/
  rule:
    - name: packageDeclaration
      lang: java
      context: package
      recursive: false
      capture: allSymbols

    - name: methodInvocation
      lang: java
      context: method_invoke
      recursive: true
      capture: literalsOnly

    - name: enumDeclaration
      lang: java
      context: class
      recursive: false
      capture: allSymbols

    - name: annotation
      lang: java
      context: annotation_invoke
      recursive: true
      capture: literalsOnly

    - name: normalClassDeclaration
      lang: java
      context: class
      recursive: false
      capture: allSymbols

    - name: variableInitializer
      lang: java
      context: var_assign
      recursive: true
      capture: literalsOnly

    - name: enumConstant
      lang: java
      context: enum
      recursive: false
      capture: allSymbols

    - name: returnStatement
      lang: java
      context: var_assign
      recursive: true
      capture: literalsOnly

    - name: methodDeclaration
      lang: java
      context: method_decl
      recursive: false
      capture: literalsOnly

    - name: argumentList
      lang: java
      context: method_invoke
      recursive: true
      capture: literalsOnly

    - name: methodInvocation_lfno_primary
      lang: java
      context: method_invoke
      recursive: true
      capture: literalsOnly

    - name: methodDeclarator
      lang: java
      context: method_decl
      recursive: false
      capture: allSymbols

    - name: methodInvocation_lf_primary
      lang: java
      context: method_invoke
      recursive: true
      capture: literalsOnly

    - name: normalInterfaceDeclaration
      lang: java
      context: class
      recursive: false
      capture: allSymbols

Javascript

Configure the ANTLR scanner to extract symbols from any programming language using the ANTLR framework.

Below is the default configuration from: antlr/javascript.yaml.

Edit Online

Edit ANTLR scanner Config Locally 

# 🔟❎ 'compile' JavaScript ANTLR symbol scanner configuration

# The 'javascript' configuration instructs the ANTLR scanner which symbol values (e.g. class/func names, code constants)
# to extract from JavaScript source files.  

# Set the 10x pipeline to 'compile'
tenx: compile

# ============================== ANTLR Options ================================

antlr:

  # The 'antlrLang' module is defined in: https://doc.log10x.com/compile/scanner/antlr/langs
  # These values define the ANTLR-generated parser class used to
  # to tokenize and construct an AST for the target JavaScript file.

  lang: javascript

  fileExt:
  - .js

  parserClass: com.log10x.antlr.parsers.JavaScriptDecoratedParser

  lexerClass: com.log10x.antlr.generated.javascript.JavaScriptLexer

  rootRule: program2

  reserved:
    - function
    - constructor
    - class

  maxLineLength: 4096

  # 'lineFilters' specifies a list of regex patterns for skipping input lines 
  lineFilters:
    - ^\s*(//.*|/\*.*\*/)$

  # The 'rule' list defines the nodes within a js AST tree that are captured
  # and added to the output symbol unit.
  # To learn more see: https://doc.log10x.com/compile/scanner/antlr/rules/
  rule:
    - name: functionDeclaration
      lang: javascript
      context: method_decl
      recursive: false
      capture: allSymbols
      subRule: identifier

    - name: returnStatement
      lang: javascript
      context: var_assign
      recursive: true
      capture: literalsOnly

    - name: classDeclaration
      lang: javascript
      context: class
      recursive: false
      capture: allSymbols
      subRule: identifier

    - name: methodDefinition
      lang: javascript
      context: method_decl
      recursive: true
      capture: allSymbols
      subRule: identifier

    - name: formalParameterList
      lang: javascript
      context: method_decl
      recursive: false
      capture: literalsOnly

    - name: functionBody
      lang: javascript
      context: method_invoke
      recursive: false
      capture: literalsOnly

    - name: functionDecl
      lang: javascript
      context: method_decl
      recursive: false
      capture: allSymbols

    - name: anoymousFunctionDecl
      lang: javascript
      context: method_decl
      recursive: false
      capture: allSymbols

    - name: argumentsExpression
      lang: javascript
      context: method_invoke
      recursive: true
      capture: literalsOnly

    - name: variableDeclaration
      lang: javascript
      context: var_assign
      recursive: true
      capture: literalsOnly

    - name: assignmentExpression
      lang: javascript
      context: var_assign
      recursive: true
      capture: literalsOnly

Python

Configure the ANTLR scanner to extract symbols from any programming language using the ANTLR framework.

Below is the default configuration from: antlr/python.yaml.

Edit Online

Edit ANTLR scanner Config Locally 

# 🔟❎ 'compile' Python symbol scanner configuration

# The 'python' configuration instructs the ANTLR scanner which symbol values (e.g. class/func names, code constants)
# to extract from Python source files.  

# NOTE: while set to enabled by default, the 'pythonAST' scanner which utilizes
# the python run-time's built-in AST parser takes precedence over this scanner
# unless a python run-time is not installed or the 'pythonAST' scanner is disabled.
# To learn see https://doc.log10x.com/compile/scanner/pythonAST

# Set the 10x pipeline to 'compile'
tenx: compile

# ============================== ANTLR Options ================================

antlr:

  # The 'antlrLang' module is defined in: https://doc.log10x.com/compile/scanner/antlr/langs
  # These values define the ANTLR-generated parser class used to
  # to tokenize and construct an AST for the target Python file.

  lang: python

  fileExt:
    - .py

  parserClass: com.log10x.antlr.generated.python.PythonParser

  lexerClass: com.log10x.antlr.generated.python.PythonLexer

  rootRule: root

  reserved:
    - def
    - __str__
    - __init__
    - class

  # 'lineFilters' specifies a list of regex patterns for skipping input lines 
  lineFilters:
    - ^\s*(//.*|/\*.*\*/)$

  # The 'rule' list defines the nodes within a Python AST tree that are captured
  # and added to the output symbol unit.
  # To learn more see: https://doc.log10x.com/compile/scanner/antlr/rules/
  rule:
    - name: classdef
      lang: python
      context: class
      recursive: false
      capture: allSymbols
      subRule: name
      condition: (^Enum$|^enum$)
      tag: enumLiteral

    - name: testlist_star_expr
      lang: python
      context: enum
      recursive: true
      capture: allSymbolsIfMatchCond
      ifTag: enumLiteral

    - name: trailer
      lang: python
      context: method_invoke
      recursive: true
      capture: literalsOnly
      subRule: name

    - name: return_stmt
      lang: python
      context: var_assign
      recursive: true
      capture: literalsOnly

    - name: decorator
      lang: python
      context: annotation_invoke
      recursive: true
      capture: literalsOnly
      subRule: arglist

    - name: arglist
      lang: python
      context: method_invoke
      recursive: true
      capture: literalsOnly

    - name: funcdef
      lang: python
      context: method_decl
      recursive: false
      capture: allSymbols
      subRule: name

    - name: assign_part
      lang: python
      context: var_assign
      recursive: true
      capture: literalsOnly

    - name: dictorsetmaker
      lang: python
      context: var_assign
      recursive: true
      capture: literalsOnly

Scala

Configure the ANTLR scanner to extract symbols from any programming language using the ANTLR framework.

Below is the default configuration from: antlr/scala.yaml.

Edit Online

Edit ANTLR scanner Config Locally 

# 🔟❎ 'compile' Scala symbol scanner configuration

# The 'scala' configuration instructs the ANTLR scanner which symbol values (e.g. class/func names, code constants)
# to extract from Scala source files.

# IMPORTANT: while set to enabled by default, the dedicated scala scanner utilizes 
# takes precedence over this scanner unless disabled.
# To learn more see https://doc.log10x.com/compile/scanner/scalameta

# Set the 10x pipeline to 'compile'
tenx: compile

# ============================== ANTLR Options ================================

antlr:

  # The 'antlrLang' module is defined in: https://doc.log10x.com/compile/scanner/antlr/langs
  # These values define the ANTLR grammar .g4 file used to
  # to tokenize and construct an AST for the target .Scala file.

  lang: scala

  fileExt:
    - .scala
    - .sc

  grammarFile: grammar/scala.g4

  rootRule: compilationUnit

  reserved:
    - interface
    - class
    - enum
    - this

  # 'lineFilters' specifies a list of regex patterns for skipping input lines 
  lineFilters:
    - ^\s*(//.*|/\*.*\*/)$

  # The 'rule' list defines the nodes within a scala AST tree that are captured
  # and added to the output symbol unit.
  # To learn more see: https://doc.log10x.com/compile/scanner/antlr/rules/
  rule:
  - name: classParamClauses
    lang: scala
    context: class
    recursive: false
    capture: literalsOnly

  - name: patVarDef
    lang: scala
    context: var_assign
    recursive: true
    capture: literalsOnly

  - name: argumentExprs
    lang: scala
    context: method_invoke
    recursive: true
    capture: literalsOnly

  - name: annotation
    lang: scala
    context: annotation_invoke
    recursive: true
    capture: literalsOnly

  - name: objectDef
    lang: scala
    context: class
    recursive: true
    capture: allSymbols

  - name: paramClauses
    lang: scala
    context: class
    recursive: false
    capture: literalsOnly

  - name: classTemplateOpt
    lang: scala
    context: class
    recursive: false
    capture: literalsOnly

  - name: funDcl
    lang: scala
    context: method_decl
    recursive: false
    capture: allSymbols

  - name: classDef
    lang: scala
    context: class
    recursive: true
    capture: allSymbols

  - name: qualId
    lang: scala
    context: package
    recursive: true
    capture: allSymbols

  - name: traitDef
    lang: scala
    context: class
    recursive: true
    capture: allSymbols

  - name: funDef
    lang: scala
    context: method_decl
    recursive: true
    capture: allSymbols
    subRule: funSig

  - name: type_
    lang: scala
    context: method_decl
    recursive: false
    capture: literalsOnly

  - name: blockStat
    lang: scala
    context: var_assign
    recursive: true
    capture: literalsOnly

  - name: templateStat
    lang: scala
    context: method_invoke
    recursive: true
    capture: literalsOnly

Strings-nix

Configure Executable scanners to launch a target process from which to read symbol values via its stdout.

Below is the default configuration from: executable/strings-nix.yaml.

Edit Online

Edit Executable scanner Config Locally 

# # 🔟❎ 'compile' executable 'strings' configuration

# This executable symbol scanner configuration uses OS-specific utilities to extract 
# constant string values from pre-compiled binaries (e.g. programs, shared libraries).
# Any matching text values are added to the output symbol files.

# The following configuration is added by default to the 10x 'compile' pipeline.
# If another 'executable' scanner is defined whose 'osFilter' matches that of the 
# current target input file, it will take precedence over the scanners defined below.
# To learn more about the 'executable' scanner, 
# see: https://doc.log10x.com/compile/scanner/executable

# Set the 10x pipeline to 'compile'
tenx: compile

# =============================== Exec Options ================================

exec: 

    # This scanner uses the Linux built-in 'strings' (https://linux.die.net/man/1/file)
    # command to extract symbol values from a target binary executable file
  - name: stringsNIX

    # This configuration targets NIX OSs
    osFilter: (nix|nux|lux|mac)  

    # try for known extensions before invoking a 'selectorProcess'
    extensions:   
      - .so
      - .dynlib   

    # 'selector' processes provide a method for determining whether a file is an executable binary.
    #  If a selector process is not provided the file's ELF/PE/OSX binary header is tested. 
    #  Since on NIX, executables do not have a known extension,
    #  the 'selector' process uses the built-in 'file' command whose 
    #  output is used to verify the target file is a NIX executable file.   
    selector:
      process: file
      args:
        - file # this macro value is replaced at run-time by the actual input file name
      timeout: 5s      
      outputFilter: ".*?\\bexecutable\\b.*?" # select 'executable' files

    process: strings    
    args:
      - -a   # scan the whole file, not just initialized and loaded sections 
      - file # This macro value is replaced at run-time by the actual text/binary input file   
    timeout: 100s

    symbol: 
      # 'minLength' specifies the min number of chars a value extracted via strings must be to 
      #  in order to qualify as a symbol. 
      minLength: 4  

      # 'maxLength' specifies the max number of chars a value extracted via strings must be to 
      #  in order to qualify as a symbol. 
      maxLength: 1024

      # 'filters' defines regex patterns applied to each line read from the scanner process' stdout 
      #  that must NOT match for the line value to be considered a symbol.
      filters: 
        - AWAV.  # https://stackoverflow.com/questions/39322552/meaning-of-a-common-string-in-executables
        - "^(?=^.{0,7}$)(?=.*[^a-zA-Z]).*$" # Matches any string that is shorter than 8 characters (0 to 7 chars) and contains at least one non-alphabetic character.

      # 'setSelector' specifies regex patterns that determines whether to treat an input value as set of tokens    
      setSelectors:
        - '^[0-9].*$'                  # Starts with digit
        - '^\*.*$'                     # Starts with *
        - '^(?:[a-zA-Z0-9](?:[a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?\.)+[a-zA-Z]{2,}(?:/[^\s]*)?$' # Domain + optional path (e.g., github.com/user/repo)
        - '^type:\\..*$'               # Go land metadata starts with "type:." (any suffix)

Strings-win

Configure Executable scanners to launch a target process from which to read symbol values via its stdout.

Below is the default configuration from: executable/strings-win.yaml.

Edit Online

Edit Executable scanner Config Locally 

# # 🔟❎ 'compile' executable 'strings' configuration

# This executable symbol scanner configuration uses OS-specific utilities to extract 
# constant string values from pre-compiled binaries (e.g. programs, shared libraries).
# Any matching text values are added to the output symbol files.

# The following configuration is added by default to the 10x 'compile' pipeline.
# If another 'executable' scanner is defined whose 'osFilter' matches that of the 
# current target input file, it will take precedence over the scanners defined below.
# To learn more about the 'executable' scanner, 
# see: https://doc.log10x.com/compile/scanner/executable/

# Set the 10x pipeline to 'compile'
tenx: compile

# =============================== Exec Options ================================

exec: 

    # This configuration provides a similar capability for scanning Windows executable files.

    # *IMPORTANT*: On Windows, the 'strings' command is not installed by default and
    # should be downloaded from: https://docs.microsoft.com/en-us/sysinternals/downloads/strings   
    # and added to 'PATH'    
  - name: stringsWin

      # target Win
    osFilter: win   

    extensions:
      - .so
      - .dll         

    process: strings
    args:
      - file # This macro value is replaced at run-time by the actual input file name
    timeout: 10s


This app is defined in compiler/app.yaml.

Next: Deploy