YAML

YAML is the main configuration format for specifying launch arguments for pipeline units (i.e., core runtime components) and modules (i.e., reusable run/compile libraries).

To load YAML config file(s) into the 10x Engine, use one of the following methods:

• Specify its path as a CLI argument preceded with a @ macro character:

  $ tenx @~/acme/config.yaml
  

• Place it in a folder loaded by the engine.

  $ tenx @~/acme
  

• Pull the file or its enclosing folder from a GitHub repo.

• Reference it or its containing folder from another YAML file via an +include directive as demonstrated below.

JavaScript Expressions

Config files can specify values as JavaScript API expressions to allow for dynamic evaluation of values.

To specify a JavaScript expression, prefix it with: $=. For example:

backpressure:
  intervalLimit: $=parseBytes("2MB")      # convert to 2,097,152 
  intervalDuration: $=parseDuration("1s") # convert to 1000 

writeTemplates: $=TenXEnv.get("myWriteTemplates") # read from a preceding launch argument, OS var or JVM -D property

# include config based on launch argument or an OS shell var:
include: '$=TenXEnv.get("myConfig") ? a.yaml : b.yaml' 

To delay evaluation of an expression until all arguments have been loaded into the runtime, prefix the target JavaScript expression with: $=yield. For example:

  # read from an launch argument after all arguments have been loaded in case 'myWriteTemplates'
  # is loaded via an include operator later in the loading sequence
  writeTemplates: $=yield TenXEnv.get("myWriteTemplates")

Include Directives

Config files can specify launch arguments explicitly or source them from other config files (e.g., .yaml, .js), folders, and GitHub repos via +include directives.

This example is derived from the Elastic input module's default elastic/config.yaml:

# 'tenx' specifies the pipeline this file targets (e.g., run, compile).
tenx: run

# =============================== Dependencies ================================

# 'include' directives load configuration files from folder, disk and GitHub:
include:

#   Load the Elastic input module folder: https://doc.log10x.com/run/input/analyzer/elasticsearch 
  - modules/input/analyzer/elasticsearch

#   Load a 10x JavaScript file https://doc.log10x.com/api#js
  - ~/tenx/my.js

# =============================== Elastic Options ==============================

# Once a module is loaded you can specify its options.

# Modules can specify a common prefix for their members (e.g., 'elasticSearch') which
# the config parser will prepend to all specified values for convenience. 
elasticSearch:

  # If the module spec is defined with an 'allowMultiple' value of true,
  # multiple entries can be specified under the 'elasticSearch' entry.

  # Specify a logical name for this Elastic input instance
  - name: acmeElastic

    host: acme.us-central1.gcp.cloud.es.io

    # Specify the '$=' prefix to use a 10x JavaScript expression to calculate values
    token: $=TenXEnv.get("acmeElasticToken")

    # Pass two instances of the 'elasticSearchTargets' list argument 
    targets:
      - acmeClient
      - acmeServer

  # For modules who support multiple instances, add a new entry:
  - name: acmeElastic2
    host: acme2.us-central1.gcp.cloud.es.io
    ...  

# Arguments can also be set via their fully qualified name. For example:
elasticSearchName: acmeElastic3
elasticSearchHost: acme3.us-central1.gcp.cloud.es.io
...