YAML

YAML is the main configuration format for specifying launch arguments for pipeline units (i.e., core runtime components) and modules (i.e., reusable run/compile libraries).

To load YAML config file(s) into the 10x Engine, use one of the following methods:

• Specify its path as a CLI argument preceded with a @ macro character:

  $ tenx @~/acme/config.yaml
  

• Place it in a folder loaded by the engine.

  $ tenx @~/acme
  

• Pull the file or its enclosing folder from a GitHub repo or Kubernetes ConfigMap.

• Reference it or its containing folder from another YAML file via an include directive as demonstrated below.

JavaScript Expressions

Config files can specify values as JavaScript API expressions to allow for dynamic evaluation of values.

To specify a JavaScript expression, prefix it with: $=. For example:

backpressure:
  intervalLimit: $=parseBytes("2MB")      # convert to 2,097,152 
  intervalDuration: $=parseDuration("1s") # convert to 1000 

writeTemplates: $=TenXEnv.get("myWriteTemplates") # read from a preceding launch argument, OS var or JVM -D property

# include config based on launch argument or an OS shell var:
include: '$=TenXEnv.get("myConfig") ? a.yaml : b.yaml' 

To delay evaluation of an expression until all arguments have been loaded into the runtime, prefix the target JavaScript expression with: $=yield. For example:

  # read from an launch argument after all arguments have been loaded in case 'myWriteTemplates'
  # is loaded via an include operator later in the loading sequence
  writeTemplates: $=yield TenXEnv.get("myWriteTemplates")

Include Directives

Config files can specify launch arguments explicitly or source them from other config files (e.g., .yaml, .js), folders, GitHub repos, and Kubernetes ConfigMaps via include directives.

This example is derived from the Elastic input module's default elastic/config.yaml:

# 'tenx' specifies the pipeline this file targets (e.g., run, compile).
tenx: run

# =============================== Dependencies ================================

# 'include' directives load configuration files from folder, disk, GitHub and Kubernetes ConfigMaps:
include:

#   Load the Elastic input module folder: https://doc.log10x.com/run/input/analyzer/elastic 
  - run/modules/input/analyzer/elasticsearch

#   Load a 10x JavaScript file https://doc.log10x.com/api#js
  - ~/tenx/my.js

#   Load a Kubernetes ConfigMap: https://doc.log10x.com/config/k8s/
  - configmap/tenx-config

# =============================== Elastic Options ==============================

# Once a module is loaded its options can be specified.

# Modules can specify a common prefix for their members (e.g., 'elasticSearch') which
# the config parser will prepend to all specified values for convenience. 
elasticSearch:

  # If the module spec is defined with an 'allowMultiple' value of true,
  # multiple entries can be specified under the 'elasticSearch' entry.

  # Specify a logical name for this Elastic input instance
  - name: acmeElastic

    host: acme.us-central1.gcp.cloud.es.io

    # Specify the '$=' prefix to use a 10x JavaScript expression to calculate values
    token: $=TenXEnv.get("acmeElasticToken")

    # Pass two instances of the 'elasticSearchTargets' list argument 
    targets:
      - acmeClient
      - acmeServer

  # For modules who support multiple instances, add a new entry:
  - name: acmeElastic2
    host: acme2.us-central1.gcp.cloud.es.io
    ...  

# Arguments can also be set via their fully qualified name. For example:
elasticSearchName: acmeElastic3
elasticSearchHost: acme3.us-central1.gcp.cloud.es.io
...