File Lookup
Enrich TenXObjects using TSV/CSV lookup files.
Example: map HTTP status codes to human-readable messages.
Lookup files reload on disk changes and can sync from GitHub for dynamic updates.
The policy regulator generates centralized lookup tables on GitHub, enabling edge instances to filter noisy telemetry based on environment-wide conditions.
Configuration
To configure the File Lookup module, Edit these settings.
Below is the default configuration from: lookup/config.yaml.
ewogICJ0eXBlIiA6ICJvYmplY3QiLAogICJwcm9wZXJ0aWVzIiA6IHsKICAgICJpbmNsdWRlIiA6IHsKICAgICAgInR5cGUiIDogInN0cmluZyIKICAgIH0sCiAgICAidGVueCIgOiB7CiAgICAgICJ0eXBlIiA6ICJzdHJpbmciCiAgICB9LAogICAgImxvb2t1cCIgOiB7CiAgICAgICJ0eXBlIiA6ICJvYmplY3QiLAogICAgICAiYWRkaXRpb25hbFByb3BlcnRpZXMiIDogZmFsc2UsCiAgICAgICJwcm9wZXJ0aWVzIiA6IHsKICAgICAgICAiZmlsZSIgOiB7CiAgICAgICAgICAidHlwZSIgOiBbCiAgICAgICAgICAgICJzdHJpbmciLAogICAgICAgICAgICAibnVsbCIKICAgICAgICAgIF0sCiAgICAgICAgICAibWFya2Rvd25EZXNjcmlwdGlvbiIgOiAiVFNWL0NTViBmaWxlIHRvIGxvYWRcblxuVGhlIHRhcmdldCAudHN2Ly5jc3YgbG9va3VwIGZpbGUgdG8gbG9hZCIKICAgICAgICB9LAogICAgICAgICJrZXlGaWVsZCIgOiB7CiAgICAgICAgICAidHlwZSIgOiBbCiAgICAgICAgICAgICJzdHJpbmciLAogICAgICAgICAgICAibnVsbCIKICAgICAgICAgIF0sCiAgICAgICAgICAibWFya2Rvd25EZXNjcmlwdGlvbiIgOiAiVGVuWE9iamVjdCBmaWVsZCBuYW1lIHRvIHVzZSBhcyBsb29rdXAga2V5XG5cblNwZWNpZmllcyBhIGZpZWxkIG5hbWUgd2hvc2UgdmFsdWUgdG8gZXh0cmFjdCBmcm9tIGEgdGFyZ2V0IFRlblhPYmplY3QgdG8gdXNlIGFzIHRoZSBsb29rdXAga2V5LCB0aGF0IGlzIHRoZSB2YWx1ZSB1c2VkIHRvIHNlYXJjaCBmb3IgYSBtYXRjaGluZyB2YWx1ZSB3aXRoaW4gdGhlIGxvb2t1cCBmaWxlLiIKICAgICAgICB9LAogICAgICAgICJ2YWx1ZUZpZWxkIiA6IHsKICAgICAgICAgICJ0eXBlIiA6IFsKICAgICAgICAgICAgInN0cmluZyIsCiAgICAgICAgICAgICJudWxsIgogICAgICAgICAgXSwKICAgICAgICAgICJtYXJrZG93bkRlc2NyaXB0aW9uIiA6ICJUZW5YT2JqZWN0IGZpZWxkIG5hbWUgdG8gdXNlIGFzIHRoZSBsb29rdXAgdmFsdWVcblxuU3BlY2lmaWVzIHRoZSBmaWVsZCBuYW1lIHdpdGggdGhlIHRhcmdldCBUZW5YT2JqZWN0IHRvIHNldCB3aXRoIHRoZSByZXN1bHQgbG9va3VwIHZhbHVlIGFjcXVpcmVkIGJ5IHNlYXJjaGluZyB0aGUgdGFibGUgZm9yIHRoZSB2YWx1ZSBvZiBbbG9va3VwS2V5RmllbGRdKGh0dHBzOi8vZG9jLmxvZzEweC5jb20vcnVuL2luaXRpYWxpemUvbG9va3VwLyNsb29rdXBrZXlmaWVsZCkuIgogICAgICAgIH0sCiAgICAgICAgInZhbHVlQ29sdW1uIiA6IHsKICAgICAgICAgICJ0eXBlIiA6IFsKICAgICAgICAgICAgInN0cmluZyIsCiAgICAgICAgICAgICJudWxsIgogICAgICAgICAgXSwKICAgICAgICAgICJtYXJrZG93bkRlc2NyaXB0aW9uIiA6ICJOYW1lIG9mIGxvb2t1cCB2YWx1ZSBjb2x1bW5cblxuU3BlY2lmaWVzIGEgY29sdW1uIGluIHRoZSBbbG9va3VwRmlsZV0oaHR0cHM6Ly9kb2MubG9nMTB4LmNvbS9ydW4vaW5pdGlhbGl6ZS9sb29rdXAvI2xvb2t1cGZpbGUpIHdob3NlIHZhbHVlIHRvIGFzc2lnbiBmcm9tIHRoZSBmaXJzdCBjb2x1bW4gaW4gdGhlIHRhYmxlIChpLmUuLCB0aGUga2V5IGNvbHVtbikgaW50byBbbG9va3VwVmFsdWVGaWVsZF0oaHR0cHM6Ly9kb2MubG9nMTB4LmNvbS9ydW4vaW5pdGlhbGl6ZS9sb29rdXAvI2xvb2t1cHZhbHVlZmllbGQpIGJhc2VkIG9uIHRoZSB2YWx1ZSBvZiAgdGhlIFtsb29rdXBLZXlGaWVsZF0oaHR0cHM6Ly9kb2MubG9nMTB4LmNvbS9ydW4vaW5pdGlhbGl6ZS9sb29rdXAvI2xvb2t1cGtleWZpZWxkKSBmaWVsZCB2YWx1ZS4gKERlZmF1bHQ6IHNlY29uZCBjb2x1bW4gaW4gdGFibGUgKGZpcnN0IGNvbHVtbiBpcyBhc3N1bWVkIGFzIHRoZSBrZXkpKSIsCiAgICAgICAgICAiZGVmYXVsdCIgOiAic2Vjb25kIGNvbHVtbiBpbiB0YWJsZSAoZmlyc3QgY29sdW1uIGlzIGFzc3VtZWQgYXMgdGhlIGtleSkiCiAgICAgICAgfQogICAgICB9LAogICAgICAicmVxdWlyZWQiIDogWwogICAgICAgICJmaWxlIiwKICAgICAgICAia2V5RmllbGQiLAogICAgICAgICJ2YWx1ZUZpZWxkIgogICAgICBdCiAgICB9CiAgfSwKICAiYWRkaXRpb25hbFByb3BlcnRpZXMiIDogZmFsc2UKfQ==
# 🔟❎ 'run' lookup file configuration
# Configuration text lookup to enrich TenXObjects
# To learn more see https://doc.log10x.com/run/initialize/lookup/
# Set the 10x pipeline to 'run'
tenx: run
# =============================== Dependencies ================================
include: run/modules/initialize/lookup
# ============================== Lookup Options ===============================
lookup:
# 'file' specifies the .csv/.tsv lookup table file
file: $=path("pipelines/run/initialize/lookup/http.csv")
# 'keyField' specifies the field whose value is query lookup key.
# The calculated code field below captures the penultimate HTTP code value from a web log event.
# For example, 200 for: Mar 10 12:34:56 webserver nginx: 192.168.1.10 - - [10/Mar/2025:12:34:56 +0000] "GET /index.html HTTP/1.1" 200 1024
keyField: $=yield TenXEnv.get("httpCodeField")
# 'valueField' specifies the name of the result field to assign to the lookup table value
valueField: http_message
Options
Specify the options below to configure the File Lookup:
| Name | Description |
|---|---|
| lookupFile | TSV/CSV file to load |
| lookupKeyField | TenXObject field name to use as lookup key |
| lookupValueField | TenXObject field name to use as the lookup value |
| lookupValueColumn | Name of lookup value column |
lookupFile
TSV/CSV file to load.
| Type | Required |
|---|---|
| File | ✔ |
The target .tsv/.csv lookup file to load.
lookupKeyField
TenXObject field name to use as lookup key.
| Type | Required |
|---|---|
| String | ✔ |
Specifies a field name whose value to extract from a target TenXObject to use as the lookup key, that is the value used to search for a matching value within the lookup file.
lookupValueField
TenXObject field name to use as the lookup value.
| Type | Required |
|---|---|
| String | ✔ |
Specifies the field name with the target TenXObject to set with the result lookup value acquired by searching the table for the value of lookupKeyField.
lookupValueColumn
Name of lookup value column.
| Type | Default |
|---|---|
| String | second column in table (first column is assumed as the key) |
Specifies a column in the lookupFile whose value to assign from the first column in the table (i.e., the key column) into lookupValueField based on the value of the lookupKeyField field value.
This module is defined in lookup/module.yaml.