Elastic

Publish TenXSummary instance values as metric counters to an ElasticSearch Micrometer registry.

Availability

This implementation is only available by default in the 10x Engine Cloud flavor to reduce the footprint of the edge/JIT-edge flavors.

Querying Metrics

The Micrometer ElasticMeterRegistry disables _source storage for metrics documents to optimize storage. This means individual metric documents cannot be retrieved directly. Use Elasticsearch aggregation queries to analyze the metrics:

GET tenx-metrics-*/_search
{
  "size": 0,
  "aggs": {
    "by_severity": {
      "terms": { "field": "severity_level.keyword", "size": 10 }
    }
  }
}

Configuration

To configure the ElasticSearch metrics output module, Edit these settings.

Below is the default configuration from: elastic/config.yaml (* Required Fields).

Edit Online

# 🔟❎ 'run' ElasticSearch metric output configuration

# Configure ElasticSearch metric counter outputs.
# To learn more see https://doc.log10x.com/run/output/metric/elastic/

# Set the 10x pipeline to 'run'
tenx: run

# =============================== Dependencies ================================

include: run/modules/output/metric/elastic

# =============================== Elastic Options =============================

# Define multiple Elastic metric outputs below

elastic:

    # --------------------------- Connection Options --------------------------

    # 'host' sets the host to send metrics to (e.g., https://<deployment-name>.us-central1.gcp.cloud.es.io).
  - host: "" # (❗ REQUIRED)

    # 'apiKeyCredentials' sets base64-encoded credentials string
    #  To learn more see: https://www.elastic.co/guide/en/elasticsearch/reference/current/security-api-create-api-key.html
    apiKeyCredentials: $=TenXEnv.get("ELASTIC_API_KEY")  # (❗ EnvVar REQUIRED)

    # 'userName' sets the the Basic Authentication username if 'apiKeyCredentials' is not set.
    userName: $=TenXEnv.get("ELASTIC_USERNAME")  # (❗ EnvVar REQUIRED)

    # 'password' sets a Basic Authentication password.
    password: $=TenXEnv.get("ELASTIC_PASSWORD")  # (❗ EnvVar REQUIRED)

    # ---------------------------- Counter Options ----------------------------

    # 'nameField' sets the TenXSummary field whose value identifies the micrometer counter to increment. 
    #  The 'inputName' field returns the name of the aggregator that produced the target TenXSummary instance.
    #  To learn more see https://doc.log10x.com/run/aggregate
    nameField: inputName

    # 'counterFields' specifies the TenXSummary fields whose values are used to increment the counter.
    #  To learn more see https://doc.log10x.com/run/aggregate/#summary-fields
    counterFields:
      - summaryVolume
      - summaryBytes
      - summaryTotals

    # 'tagFields' sets the TenXSummary fields used as the counter tag dimension values.
    #  Defaults to https://doc.log10x.com/run/initialize/#enrichmentFields
    tagFields:
      - $=yield TenXEnv.get("enrichmentFields")

    # 'tagFieldNames' allows setting custom names to aggregated fields for metrics reporting.
    #  Defaults to https://doc.log10x.com/run/initialize/#metricFieldNames
    tagFieldNames:
      - $=yield TenXEnv.get("metricFieldNames")

    # 'filter' sets a JavaScript expression that TenXSummaries must evaluate as truthy to write to the output.
    #  To learn more see https://doc.log10x.com/run/output/regulate/#filter-expressions
    filter: isSummary

    # ---------------------------- Publish Options ----------------------------

    # 'step' sets the interval for sending metrics to Elastic (e.g., '1m').
    #  To learn more see: https://micrometer.io/docs/concepts#rate-aggregation
    step: 5s

    # ----------------------------- Index Options -----------------------------

    # 'elasticIndex' sets the index name to which to write metrics.
    index: tenx-metrics

    # 'autoCreateIndex' sets whether to create the index automatically if it doesn't exist.
    autoCreateIndex: true

    # 'indexDateFormat' sets the index date format used for rolling indices.
    #  This is appended to the index name, separated by 'elasticIndexDateSeparator'.
    indexDateFormat: 'yyyy-MM'

    # 'indexDateFormat' sets the separator between the index name and the date part
    indexDateSeparator: '-'

    # 'pipeline' sets the ingest pipeline name
    pipeline: null

    # 'timestampFieldName' sets the name of the elastic timestamp field.
    timestampFieldName: '@timestamp'

Options

Specify the options below to configure multiple ElasticSearch metrics output:

Name	Description	Category
elasticEnabled	Enable this output	General
elasticNameField	Name of the TenXSummary field identifying the metric	Counter
elasticCounterFields	Names of the TenXSummary fields used to increment the metric counter value	Counter
elasticTagFields	List of TenXSummary fields to use as metric tag values	Counter
elasticTagFieldNames	List of custom names to use for TenXSummary fields.	Counter
elasticFilter	A JavaScript expression an TenXSummary must evaluate as 'truthy' against to be written to this output	Counter
elasticMaxCardinality	Maximum number of unique metric combinations allowed	Counter
elasticStep	The interval at which to publish metrics to output	Counter
elasticHost	Host to send metrics to	Authentication
elasticApiKeyCredentials	Base64-encoded credentials string	Authentication
elasticUserName	Basic Authentication username	Authentication
elasticPassword	Basic Authentication password	Authentication
elasticIndex	The index name to write metrics to	Index
elasticAutoCreateIndex	Whether to create the index automatically if it doesn't exist	Index
elasticPipeline	The ingest pipeline name	Index
elasticIndexDateFormat	Index date format used for rolling indices	Index
elasticTimestampFieldName	Name of the elastic timestamp field	Index
elasticIndexDateSeparator	Separator between the index name and the date part	Index

General

`elasticEnabled`

Enable this output.

Type	Default	Category
Boolean	true	General

Enable or disable this output.

Counter

`elasticNameField`

Name of the TenXSummary field identifying the metric.

Type	Default	Category
String	summaryValues	Counter

Defines the name of the TenXSummary field whose value identifies a metric counter. Defaults to the summaryValues field which holds the value(s) by which TenXObjects are grouped into the current TenXSummary instance. To learn more see micrometer counters.

`elasticCounterFields`

Names of the TenXSummary fields used to increment the metric counter value.

Type	Default	Category
List	summaryVolume	Counter

Defines the name of the TenXSummary field by whose value to increment the counter.

`elasticTagFields`

List of TenXSummary fields to use as metric tag values.

Type	Default	Category
List	[]	Counter

Defines the TenXSummary field names to use as tag values. Tags are dimensions that enable slicing a particular named metric to drill down to reason about its value. To learn more see micrometer tags.

`elasticTagFieldNames`

List of custom names to use for TenXSummary fields.

Type	Default	Category
List	[]	Counter

Defines the a list of custom names to give fields when reporting them. Format as a list of name1, custom1, name2, custom2.

For example, to set a custom name of 'My-Custom-Field' to an existing 'intrinsic-field':

elastic:
  tagFieldNames:
    - intrinsic-field
    - My-Custom-Field.

`elasticFilter`

A JavaScript expression an TenXSummary must evaluate as 'truthy' against to be written to this output.

Type	Default	Category
String	""	Counter

Specifies a JavaScript expression that an TenXSummary must evaluate as truthy to write its instance/template field values to the target counter.

For example, to only emit TenXSummaries produced by the 'myMetric' aggregator, specify:

elastic:
  filter: this.inputName == "myMetric"

To learn more see inputName.

`elasticMaxCardinality`

Maximum number of unique metric combinations allowed.

Type	Default	Category
Number	500	Counter

Sets the maximum number of unique metric combinations (cardinality) allowed for this output. When exceeded, metrics will be dropped to prevent excessive cardinality.

`elasticStep`

The interval at which to publish metrics to output.

Type	Default	Category
String	1 minute.	Counter

Sets the interval at which to publish metrics to the time-series destination (e.g., '1m'). To learn more see rate-aggregation.

Authentication

`elasticHost`

Host to send metrics to.

Type	Default	Category
String	http://localhost:9200	Authentication

Sets the host to send metrics to.

`elasticApiKeyCredentials`

Base64-encoded credentials string.

Type	Default	Category
String	""	Authentication

Sets a base64-encoded credentials string To learn more see: Elastic credentials.

`elasticUserName`

Basic Authentication username.

Type	Default	Category
String	""	Authentication

Sets the the Basic Authentication username. If 'elasticApiKeyCredentials' not is set, it will be used instead.

`elasticPassword`

Basic Authentication password.

Type	Default	Category
String	""	Authentication

Sets a Basic Authentication password. If 'elasticApiKeyCredentials' is not set it will be used instead.

Index

`elasticIndex`

The index name to write metrics to.

Type	Default	Category
String	micrometer-metrics	Index

Sets the index name to write metrics to.

`elasticAutoCreateIndex`

Whether to create the index automatically if it doesn't exist.

Type	Default	Category
Boolean	true	Index

Sets whether to create the index automatically if it doesn't exist.

`elasticPipeline`

The ingest pipeline name.

Type	Default	Category
String	""	Index

The ingest pipeline name.

`elasticIndexDateFormat`

Index date format used for rolling indices.

Type	Default	Category
String	yyyy-MM	Index

Sets the index date format used for rolling indices. This is appended to the index name, separated by 'elasticIndexDateSeparator'.

`elasticTimestampFieldName`

Name of the elastic timestamp field.

Type	Default	Category
String	@timestamp	Index

Sets the name of the elastic timestamp field.

`elasticIndexDateSeparator`

Separator between the index name and the date part.

Type	Default	Category
String	-	Index

Sets the separator between the index name and the date part.

This module is defined in elastic/module.yaml.