Field Parser

Extracts JSON objects and KV structures from TenXTemplates as named fields.

This process enables TenXObjects to access nested JSON/KV values at runtime as named members.

For an hybrid JSON/plain event whose text value is:

17/06/24 17:38:00 INFO ExecutorRunnable: Prepared Local resources Map(__spark__.jar -> resource 
{ scheme: "hdfs" host: "10.10.34.11" port: 9000 
file: "/user/curi/.sparkStaging/application_1485248649253_0144/spark-assembly-1.6.0-hadoop2.2.0.jar" } 
size: 109525492 timestamp: 1497001131801 type: FILE visibility: PRIVATE)

To increase a counter identified by a JSON field (scheme) value with that of a KV entry (size):

TenXCounter.inc(this.scheme, this.size);

To drop the instance if either a KV entry (e.g., visibility) or JSON field (e.g., port) match a specified value:

this.drop((this.visibility == "PRIVATE") || (this.port == 9000));

Arrays

If a JSON/KV field appears more than once within an TenXObject's text, it can be accessed as an array field to access individual entries (e.g., this.myField[i]).

Config Files

To configure the Field parser unit, Edit these files.

Below is the default configuration from: fields/config.yaml.

Edit Online

# 🔟❎ 'run' TenXTemplate field extract configuration

# Configure how to extract JSON objects and KV structures from TenXTemplates.
# To learn more see https://doc.log10x.com/run/transform/fields/

# Set the 10x pipeline to 'run'
tenx: run

# =============================== Extract Options =============================

field:

  # 'extract' controls whether to scan TenXTemplate for  JSON
  #  objects or key-value lists (e.g., 'X=Y'). 
  extract: true

  # 'nameBreaks' controls which characters found to the left of a token
  #  whose is a candidate for being a 'key' in a KV field formation should serve 
  #  as a terminator for the search. For example, for an object whose 'text' field
  #  contains the following entry ',tx_result=OK' the desired key name should be 'tx_result'
  #  and as such, the name terminator character should be ',' vs. '_' (in which case
  #  the key would have been named 'result').
  nameBreaks: ', /\{}.()[]'

  # 'valueBreaks' controls which characters found to the right of a token
  #  whose is a candidate for being a 'value' in a KV field formation should serve 
  #  as a terminator for the search. For example, for an object whose 'text' field
  #  contains the following entry: 'status=RESULT_SUCCESS' the desired KV value 
  #  should be 'RESULT_SUCCESS', and as such, the value terminator character 
  #  should be ',' vs. '_' (in which case the value would be 'RESULT').
  valueBreaks: ', /\{}()[]'

Options

Specify the options below to configure the Field parser:

Name	Description
fieldExtract	Scan TenXTemplate for nested JSON objects or key-value lists
fieldNameBreaks	KV field name terminator chars
fieldValueBreaks	KV field value terminator chars

`fieldExtract`

Scan TenXTemplate for nested JSON objects or key-value lists.

Type	Default
Boolean	true

Controls whether to scan TenXTemplates for JSON objects or key-value lists (e.g., {"price": 10} or price=10).

TenXObjects instances of this template will provide access to these fields as named members (e.g., this.price). If a field appears more than once in text it is accessible as an array (e.g., this.price[1]).

`fieldNameBreaks`

KV field name terminator chars.

Type	Default
String	, /\{}.()[]

Controls which 'text' characters left of a candidate token for being a 'key' in a KV field formation should serve as a terminator for the search.

For example, for an TenXObject whose text contains the following entry ',tx_result=OK', the desired key should be 'tx_result', and so the name terminator character should be ',' vs. '_' (in which case the key would have been named 'result').

`fieldValueBreaks`

KV field value terminator chars.

Type	Default
String	/\{}()[]

Controls which characters found to the right of a token whose is a candidate for being a 'value' in a KV field formation should serve as a terminator for the search.

For example, for an object whose text field contains the following entry: 'status=RESULT_SUCCESS' the desired KV value should be 'RESULT_SUCCESS', and as such the value terminator character should be ',' vs. '_' (in which case the value would be 'RESULT').

This unit is defined in fields/unit.yaml.