Template
The 10x Engine employs an optimization model inspired by Chrome V8’s intelligent design principle. Rather than treating each log event as a "bag of properties," the 10x Engine assumes only a finite number of object shapes will emerge and that these shapes will follow stereotypical usage templates.
The 10x Engine utilizes symbol library files to assign a shared TenXTemplate (i.e., hidden class) to input log/trace events with the same structure, creating a cached optimized schema for each event type.
Operating on typed TenXObjects enables direct access to symbol and variable values at runtime without repeatedly parsing JSON structures or evaluating complex, brittle regular expressions for each event.
Template Files
Template files describes the structure of TenXTemplates generated during previous execution cycles, similar to a .proto file which describes the structure of serialized records.
Each template instance is read as a single line JSON object which contains a templateHash and template fields into an in-memory map to enable expanding of previously compact instances read from inputs.
For example, a template .json file or data read from an input stream may contain the following template objects:
{"templateHash":"7C9 /a$3Lv","template":"{\"log\":\".:$\\n\",\"stream\":\"stdout\",\"docker\":{\"container_id\":\"$\"},\"kubernetes\":{\"container_name\":\"coredns\",\"namespace_name\":\"kube-system\",\"pod_name\":\"coredns-$-$\",\"container_image\":\"registry.k8s.io//coredns//coredns:v1.$.$\",\"container_image_id\":\"docker-$:////registry.k8s.io//coredns//coredns@sha256:$\",\"pod_id\":\"$-$-$-$$\",\"pod_ip\":\"$.$.$.$\",\"host\":\"minikube\",\"labels\":{\"k8s-app\":\"kube-dns\",\"pod-template-hash\":\"$\"}},\"$_tag\":\"kubernetes.var.log.containers.coredns-$-$_kube-system_coredns-$.log\"}"}
{"templateHash":"-EqO[AdeT!#","template":"{\"log\":\"[INFO] plugin//reload: Running configuration SHA512 = $\\n\",\"stream\":\"stdout\",\"docker\":{\"container_id\":\"$\"},\"kubernetes\":{\"container_name\":\"coredns\",\"namespace_name\":\"kube-system\",\"pod_name\":\"coredns-$-$\",\"container_image\":\"registry.k8s.io//coredns//coredns:v1.$.$\",\"container_image_id\":\"docker-$:////registry.k8s.io//coredns//coredns@sha256:$\",\"pod_id\":\"$-$-$-$$\",\"pod_ip\":\"$.$.$.$\",\"host\":\"minikube\",\"labels\":{\"k8s-app\":\"kube-dns\",\"pod-template-hash\":\"$\"}},\"$_tag\":\"kubernetes.var.log.containers.coredns-$-$_kube-system_coredns-$.log\"}"}
{"templateHash":"-*pAW5IsbiJ","template":"{\"log\":\"[INFO] $.$.$.$:$ - $ \\\"AAAA IN github.com.default.svc.cluster.local. udp $ false $\\\" NXDOMAIN qr,aa,rd $ $7.$\\n\",\"stream\":\"stdout\",\"docker\":{\"container_id\":\"$\"},\"kubernetes\":{\"container_name\":\"coredns\",\"namespace_name\":\"kube-system\",\"pod_name\":\"coredns-$-$\",\"container_image\":\"registry.k8s.io//coredns//coredns:v1.$.$\",\"container_image_id\":\"docker-pullable:////registry.k8s.io//coredns//coredns@sha256:$\",\"pod_id\":\"$-$-$-$$\",\"pod_ip\":\"$.$.$.$\",\"host\":\"minikube\",\"labels\":{\"k8s-app\":\"kube-dns\",\"pod-template-hash\":\"$\"}},\"$_tag\":\"kubernetes.var.log.containers.coredns-$-$_kube-system_coredns-$.log\"}"}
The last template in this example whose hash is -*pAW5IsbiJ can be used to expand the contents of the following compact event from an input stream:
~-*pAW5IsbiJ,10,244,0,102,45981,64767,77,1232,147,000221854s,4c195cfdbf7e41f640631629970b9af2d8a1f40f63dcffd15edca84e2e2e497e,7db6d8ff4d,pddxj,11,1,1eeb4c7316bacb1d4c8ead65571cd92dd21e27359f0d4917f1a5822a73b75db1,38b91d65,ba47,4d0f,a689,-711056955842,10,244,0,99,7db6d8ff4d,10x,7db6d8ff4d,pddxj,4c195cfdbf7e41f640631629970b9af2d8a1f40f63dcffd15edca84e2e2e497e
Given both template and compact event information, the 10x Engine will automatically expand the event to its original full-text form:
{"log":"[INFO] 10.244.0.102:45981 - 64767 \"AAAA IN github.com.default.svc.cluster.local. udp 77 false 1232\" NXDOMAIN qr,aa,rd 147 0.000221854s\n","stream":"stdout","docker":{"container_id":"4c195cfdbf7e41f640631629970b9af2d8a1f40f63dcffd15edca84e2e2e497e"},"kubernetes":{"container_name":"coredns","namespace_name":"kube-system","pod_name":"coredns-7db6d8ff4d-pddxj","container_image":"registry.k8s.io/coredns/coredns:v1.11.1","container_image_id":"docker-pullable://registry.k8s.io/coredns/coredns@sha256:1eeb4c7316bacb1d4c8ead65571cd92dd21e27359f0d4917f1a5822a73b75db1","pod_id":"38b91d65-ba47-4d0f-a689-711056955842","pod_ip":"10.244.0.99","host":"minikube","labels":{"k8s-app":"kube-dns","pod-template-hash":"7db6d8ff4d"}},"tenx_tag":"kubernetes.var.log.containers.coredns-7db6d8ff4d-pddxj_kube-system_coredns-4c195cfdbf7e41f640631629970b9af2d8a1f40f63dcffd15edca84e2e2e497e.log"}
At runtime, matching TenXTemplate JSON objects from pipeline inputs are automatically added to the available templates for expanding. This allows compact files and streams to include template definitions followed by their compact instances.
Configuration
To configure the TenXTemplate Loader unit, Edit these settings.
Below is the default configuration from: template/config.yaml.
ewogICJ0eXBlIiA6ICJvYmplY3QiLAogICJwcm9wZXJ0aWVzIiA6IHsKICAgICJ0ZW54IiA6IHsKICAgICAgInR5cGUiIDogInN0cmluZyIKICAgIH0sCiAgICAidGVtcGxhdGUiIDogewogICAgICAidHlwZSIgOiAib2JqZWN0IiwKICAgICAgImFkZGl0aW9uYWxQcm9wZXJ0aWVzIiA6IGZhbHNlLAogICAgICAicHJvcGVydGllcyIgOiB7CiAgICAgICAgImZpbGVzIiA6IHsKICAgICAgICAgICJ0eXBlIiA6IFsKICAgICAgICAgICAgImFycmF5IiwKICAgICAgICAgICAgIm51bGwiCiAgICAgICAgICBdLAogICAgICAgICAgIm1hcmtkb3duRGVzY3JpcHRpb24iIDogIklucHV0IGZpbGVzIG9yIGdsb2IgcGF0dGVybnMgY29udGFpbmluZyBKU09OLWVuY29kZWQgVGVuWFRlbXBsYXRlc1xuXG5JbnB1dCBmaWxlcyBvciBnbG9iIHBhdHRlcm5zIGZyb20gd2hpY2ggdG8gbG9hZCBKU09OLWVuY29kZWQgVGVuWFRlbXBsYXRlcy4gKERlZmF1bHQ6IFtcInBhdGgoXFxcImRhdGEvc2FtcGxlL3RlbXBsYXRlcy8qLmpzb25cXFwiKVwiXSkiLAogICAgICAgICAgIml0ZW1zIiA6IHsKICAgICAgICAgICAgInR5cGUiIDogInN0cmluZyIKICAgICAgICAgIH0sCiAgICAgICAgICAiZGVmYXVsdCIgOiBbCiAgICAgICAgICAgICJwYXRoKFwiZGF0YS9zYW1wbGUvdGVtcGxhdGVzLyouanNvblwiKSIKICAgICAgICAgIF0KICAgICAgICB9LAogICAgICAgICJjYWNoZVNpemUiIDogewogICAgICAgICAgInR5cGUiIDogWwogICAgICAgICAgICAibnVtYmVyIiwKICAgICAgICAgICAgInN0cmluZyIKICAgICAgICAgIF0sCiAgICAgICAgICAibWFya2Rvd25EZXNjcmlwdGlvbiIgOiAiTWF4aW11bSB0b3RhbCBieXRlIHNpemUgb2YgaW4tbWVtb3J5IGNhY2hlZCB0ZW1wbGF0ZXNcblxuQ29udHJvbHMgdGhlIG1heGltdW0gdG90YWwgYnl0ZSBzaXplIG9mIHRlbXBsYXRlcyBoZWxkIGluIHRoZSBpbi1tZW1vcnkgY2FjaGUuIFdoZW4gdGhlIGNhY2hlIGV4Y2VlZHMgdGhpcyBsaW1pdCwgbGVhc3QgcmVjZW50bHkgdXNlZCB0ZW1wbGF0ZXMgYXJlIHBydW5lZCB0byBhIHRlbXBvcmFyeSBmaWxlIG9uIGRpc2suIFBydW5lZCB0ZW1wbGF0ZXMgYXJlIGF1dG9tYXRpY2FsbHkgcmVsb2FkZWQgZnJvbSBkaXNrIHdoZW4gYWNjZXNzZWQgYWdhaW4uICBTZXQgdG8gMCB0byBkaXNhYmxlIHBydW5pbmcgYW5kIGtlZXAgYWxsIHRlbXBsYXRlcyBpbiBtZW1vcnkuIChBY2NlcHRzIG51bWJlciBvciBzdHJpbmcgd2l0aCAkPSBwcmVmaXggZm9yIHJ1bnRpbWUgZXZhbHVhdGlvbikgKERlZmF1bHQ6IDIwOTcxNTIwKSIsCiAgICAgICAgICAiZGVmYXVsdCIgOiAyMDk3MTUyMAogICAgICAgIH0KICAgICAgfQogICAgfSwKICAgICJ2YXIiIDogewogICAgICAidHlwZSIgOiAib2JqZWN0IiwKICAgICAgImFkZGl0aW9uYWxQcm9wZXJ0aWVzIiA6IGZhbHNlLAogICAgICAicHJvcGVydGllcyIgOiB7CiAgICAgICAgInBsYWNlaG9sZGVyIiA6IHsKICAgICAgICAgICJ0eXBlIiA6IFsKICAgICAgICAgICAgInN0cmluZyIsCiAgICAgICAgICAgICJudWxsIgogICAgICAgICAgXSwKICAgICAgICAgICJtYXJrZG93bkRlc2NyaXB0aW9uIiA6ICJQbGFjZWhvbGRlciBjaGFyYWN0ZXIgZm9yIG5vdGF0aW5nIFRlblhPYmplY3QgaW5zdGFuY2UgdmFyaWFibGVzXG5cblNwZWNpZmllcyBhIGNoYXJhY3RlciB0byB1c2Ugd2hlbiBlbmNvZGluZyBhbiBUZW5YVGVtcGxhdGUgdG8gc2lnbmlmeSB0aGUgbG9jYXRpb24gb2YgYSBydW50aW1lIHZhcmlhYmxlIHZhbHVlLiAgRm9yIGV4YW1wbGUsIGZvciBhbiBUZW5YVGVtcGxhdGUgY29udGFpbmluZyB0aGUgZm9sbG93aW5nIHNlZ21lbnQ6ICAnRVJST1I6IGNvdWxkIG5vdCBjb25uZWN0IHRvIGhvc3QgSVA6ICQuJC4kLiQnICBUaGUgJyQnIGVudHJpZXMgZGVzaWduYXRlIHRoYXQgYW55IFRlblhPYmplY3QgYXNzb2NpYXRlZCB3aXRoIHRoaXMgdGVtcGxhdGUgIHdpbGwgc3Vic3RpdHV0ZSB0aGUgJyQnIHNsb3RzIHdpdGggbWF0Y2hpbmcgdmFsdWVzIGZyb20gaXRzIGluZGl2aWR1YWwgW3ZhcnNdKGh0dHBzOi8vZG9jLmxvZzEweC5jb20vYXBpL2pzLyNUZW5YQmFzZU9iamVjdCt2YXJzKSBhcnJheSAoZS5nLiAwLDAsMCwxMjcpLiAoRGVmYXVsdDogJCkiLAogICAgICAgICAgImRlZmF1bHQiIDogIiQiCiAgICAgICAgfSwKICAgICAgICAibWF4UmVjdXJJbmRleGVzIiA6IHsKICAgICAgICAgICJ0eXBlIiA6IFsKICAgICAgICAgICAgIm51bWJlciIsCiAgICAgICAgICAgICJzdHJpbmciCiAgICAgICAgICBdLAogICAgICAgICAgIm1hcmtkb3duRGVzY3JpcHRpb24iIDogIk1heGltdW0gbnVtYmVyIG9mIHRlbXBsYXRlIHJldXNhYmxlIHZhcmlhYmxlIHZhbHVlXG5cbkNvbnRyb2xzIHRoZSBtYXhpbXVtIG51bWJlciBvZiB2YXJpYWJsZSB2YWx1ZXMgdG8gcmV1c2Ugd2hlbiBlbmNvZGluZyBhbiBUZW5YT2JqZWN0J3MgW3ZhcnNdKGh0dHBzOi8vZG9jLmxvZzEweC5jb20vYXBpL2pzLyNUZW5YQmFzZU9iamVjdCt2YXJzKSBhcnJheSB0byBzdHJlYW0uIFdoZW4gZ2VuZXJhdGluZyBhIG5ldyAgVGVuWFRlbXBsYXRlIGZvciBhbiBUZW5YT2JqZWN0IGluc3RhbmNlIHdob3NlIHZhcmlhYmxlcyBjb250YWluIHJlcGV0aXRpdmUgdmFsdWVzLCAgdGhlIHRlbXBsYXRlIG1heSBpbmRpY2F0ZSB0aGF0IGEgdmFsdWUgaXMgYSByZXBldGl0aW9uIG9mIGEgcHJldmlvdXMgdmFyaWFibGUgdmFsdWUgYW5kICBkb2VzIG5vdCBuZWVkIHRvIGJlIGVuY29kZWQgYWdhaW4uICBGb3IgZXhhbXBsZSwgZm9yIGFuIFRlblhPYmplY3Qgd2hvc2UgW3RleHRdKGh0dHBzOi8vZG9jLmxvZzEweC5jb20vYXBpL2pzLyNUZW5YQmFzZU9iamVjdCt0ZXh0KSBmaWVsZCBjb250YWlucyB0aGUgZm9sbG93aW5nIHNlZ21lbnQ6IGB1c2VySUQ9NDVkZmdkZnM0LHBhcmVudElEPTQ1ZGZnZGZzNGAsIHRoZSByZXN1bHRpbmcgVGVuWFRlbXBsYXRlIGNhbiBiZSBlbmNvZGVkIGFzOiBgdXNlcklEPSQscGFyZW50SUQ9JDFgIHNwZWNpZnlpbmcgdGhhdCB0aGUgdmFsdWUgb2YgdGhlIGBwYXJlbnRJRGAgdmFyaWFibGUgY2FuIGJlIGZvdW5kIG9uZSBzbG90IGJlaGluZCBpbiB0aGUgVGVuWE9iamVjdCdzIFt2YXJzIGFycmF5XShodHRwczovL2RvYy5sb2cxMHguY29tL2FwaS9qcy8jVGVuWEJhc2VPYmplY3QrdmFycyksIHNhdmluZyB0aGUgbmVlZCB0byBlbmNvZGUgdGhpcyB2YWx1ZSBtdWx0aXBsZSB0aW1lcy4gKEFjY2VwdHMgbnVtYmVyIG9yIHN0cmluZyB3aXRoICQ9IHByZWZpeCBmb3IgcnVudGltZSBldmFsdWF0aW9uKSAoRGVmYXVsdDogMTApIiwKICAgICAgICAgICJkZWZhdWx0IiA6IDEwCiAgICAgICAgfQogICAgICB9CiAgICB9LAogICAgInRpbWVzdGFtcCIgOiB7CiAgICAgICJ0eXBlIiA6ICJvYmplY3QiLAogICAgICAiYWRkaXRpb25hbFByb3BlcnRpZXMiIDogZmFsc2UsCiAgICAgICJwcm9wZXJ0aWVzIiA6IHsKICAgICAgICAicHJlZml4IiA6IHsKICAgICAgICAgICJ0eXBlIiA6IFsKICAgICAgICAgICAgInN0cmluZyIsCiAgICAgICAgICAgICJudWxsIgogICAgICAgICAgXSwKICAgICAgICAgICJtYXJrZG93bkRlc2NyaXB0aW9uIiA6ICJUZW1wbGF0ZSB0aW1lc3RhbXAgcHJlZml4XG5cblNwZWNpZmllcyBhIHByZWZpeCBmb3IgYW4gVGVuWFRlbXBsYXRlJ3Mgc2V0IG9mIHZhcmlhYmxlIGFuZCBzeW1ib2wgdG9rZW5zIGRlc2NyaWJpbmcgYSB0aW1lc3RhbXAgKGUuZy4sICR7REQvTU0vWVkgSEgtbW0tc3N9KS4gKERlZmF1bHQ6ICgpIiwKICAgICAgICAgICJkZWZhdWx0IiA6ICIoIgogICAgICAgIH0sCiAgICAgICAgInBvc3RmaXgiIDogewogICAgICAgICAgInR5cGUiIDogWwogICAgICAgICAgICAic3RyaW5nIiwKICAgICAgICAgICAgIm51bGwiCiAgICAgICAgICBdLAogICAgICAgICAgIm1hcmtkb3duRGVzY3JpcHRpb24iIDogIlRlbXBsYXRlIHRpbWVzdGFtcCBwb3N0Zml4XG5cblNwZWNpZmllcyBhIHZhbHVlIGFwcGVuZGVkIHRvIGFuIFRlblhUZW1wbGF0ZSdzIGFuZCBzZXQgb2YgdmFyaWFibGUgYW5kIHN5bWJvbCB0b2tlbnMgZGVzY3JpYmluZyBhIHRpbWVzdGFtcCAoZS5nLiwgJHtERC9NTS9ZWSBISC1tbS1zc30pLiAoRGVmYXVsdDogKSkiLAogICAgICAgICAgImRlZmF1bHQiIDogIikiCiAgICAgICAgfQogICAgICB9CiAgICB9CiAgfSwKICAiYWRkaXRpb25hbFByb3BlcnRpZXMiIDogZmFsc2UKfQ==
# 🔟❎ 'run' template file configuration
# Load TenXTemplates .json files that define the structure/schema of TenXObjects.
# To learn more see https://doc.log10x.com/run/template/
# Set the 10x pipeline to 'run'
tenx: run
# =============================== Template ===================================
template:
# 'files' specifies GLOB pattern for finding JSON-encoded TenXTemplates.
files:
- $=path("data/templates/*.json")
- $=path("data/sample/output/*.json")
# 'cacheSize' controls the maximum total byte size of templates held
# in the in-memory cache vs. on disk. Set to 0 to disable pruning.
cacheSize: $=parseBytes("10MB")
# =============================== Variable ===================================
var:
# 'placeholder' specifies a character to use when encoding a TenXTemplate
# to signify the location of a runtime variable value.
placeholder: "$"
# 'maxRecurIndexes' controls the maximum number of variable values to reuse.
maxRecurIndexes: 10
# =============================== Timestamp ==================================
timestamp:
# 'prefix' specifies a prefix for a TenXTemplate's timestamp tokens.
prefix: (
# 'postfix' specifies a postfix for a TenXTemplate's timestamp tokens.
postfix: )
Options
Specify the options below to configure the TenXTemplate Loader:
| Name | Description | Category |
|---|---|---|
| templateFiles | Input files or glob patterns containing JSON-encoded TenXTemplates | Load |
| varPlaceholder | Placeholder character for notating TenXObject instance variables | Parse |
| timestampPrefix | Template timestamp prefix | Parse |
| timestampPostfix | Template timestamp postfix | Parse |
| varMaxRecurIndexes | Maximum number of template reusable variable value | Parse |
| templateCacheSize | Maximum total byte size of in-memory cached templates | Advanced |
Load
templateFiles
Input files or glob patterns containing JSON-encoded TenXTemplates.
| Type | Default | Category |
|---|---|---|
| List | [path("data/sample/templates/*.json")] | Load |
Input files or glob patterns from which to load JSON-encoded TenXTemplates.
Parse
varPlaceholder
Placeholder character for notating TenXObject instance variables.
| Type | Default | Category |
|---|---|---|
| String | $ | Parse |
Specifies a character to use when encoding an TenXTemplate to signify the location of a runtime variable value.
For example, for an TenXTemplate containing the following segment: 'ERROR: could not connect to host IP: $.$.$.$'
The '$' entries designate that any TenXObject associated with this template will substitute the '$' slots with matching values from its individual vars array (e.g. 0,0,0,127).
timestampPrefix
Template timestamp prefix.
| Type | Default | Category |
|---|---|---|
| String | ( | Parse |
Specifies a prefix for an TenXTemplate's set of variable and symbol tokens describing a timestamp (e.g., ${DD/MM/YY HH-mm-ss}).
timestampPostfix
Template timestamp postfix.
| Type | Default | Category |
|---|---|---|
| String | ) | Parse |
Specifies a value appended to an TenXTemplate's and set of variable and symbol tokens describing a timestamp (e.g., ${DD/MM/YY HH-mm-ss}).
varMaxRecurIndexes
Maximum number of template reusable variable value.
| Type | Default | Category |
|---|---|---|
| Number | 10 | Parse |
Controls the maximum number of variable values to reuse when encoding an TenXObject's vars array to stream. When generating a new TenXTemplate for an TenXObject instance whose variables contain repetitive values, the template may indicate that a value is a repetition of a previous variable value and does not need to be encoded again.
For example, for an TenXObject whose text field contains the following segment:
userID=45dfgdfs4,parentID=45dfgdfs4, the resulting TenXTemplate can be encoded
as: userID=$,parentID=$1 specifying that the value of the parentID variable
can be found one slot behind in the TenXObject's vars array, saving
the need to encode this value multiple times.
Advanced
templateCacheSize
Maximum total byte size of in-memory cached templates.
| Type | Default | Category |
|---|---|---|
| Number | 20971520 | Advanced |
Controls the maximum total byte size of templates held in the in-memory cache. When the cache exceeds this limit, least recently used templates are pruned to a temporary file on disk. Pruned templates are automatically reloaded from disk when accessed again.
Set to 0 to disable pruning and keep all templates in memory.
This unit is defined in template/unit.yaml.