ElasticSearch

Reads events from Elasticsearch clusters and transforms them into TenXObjects.

Instances of this module define a connection to a hosted/on-premises ElasticSearch cluster from which events to retrieve, as well as the querying logic used such as chronological direction, start values, time ranges, and page size of each API request sent.

ElasticSearch inputs commonly run within scheduled jobs (e.g., k8s CronJob) to retrieve a recent sample amount of events (e.g., 200MB in the last 10min) to transform into TenXObjects as part of the Cloud Reporter app.

Configuration

To configure the ElasticSearch input module, Edit these settings:

Elastic

Activate ElasticSearch inputs to read events from an ElasticSearch hosted/on-premises cluster.

Below is the default configuration from: elastic/config.yaml (* Required Fields).

Edit Online

# 🔟❎ 'run' Elastic input configuration

# Configure an Elastic event input
# To learn more see https://doc.log10x.com/run/input/analyzer/elasticsearch/

# Set the 10x pipeline to 'run'
tenx: run

# =============================== Dependencies ================================

include: run/modules/input/analyzer/elasticsearch

# =============================== Elastic Options ==============================

# Multiple Elastic inputs can be defined below:
elasticSearch:

    # 'name' sets a unique logical name across all pipeline inputs
  - name: ElasticSearch

    # --------------------------- Connection Options --------------------------

    # 'host' and 'port' set the ElasticSearch API endpoint to connect to
    #  For hosted Elastic (e.g. my-project-ec5189.es.us-central1.gcp.elastic.cloud).
    #  See https://cloud.elastic.co/deployments and https://i.stack.imgur.com/jXjLJ.png
    host: null # (❗ REQUIRED)
    port: "" # (not mandatory if the host already encapsulates it)

    # 'protocol' sets the connection protocol ('https' or 'http' for local testing)
    protocol: https

    # 'token' used to authenticate with. To learn more see:
    # https://www.elastic.co/guide/en/cloud-enterprise/current/ece-restful-api-authentication.html
    token: $=TenXEnv.get("ELASTIC_TOKEN") # (❗ REQUIRED)

    # ----------------------------- Query Options -----------------------------

    # 'targets' lists data streams, indices, and aliases to search.
    #  To search all data streams and indices, use '*' or '_all'.
    targets:
      - _all

    # 'startTime' and 'endTime' set the time frame of the query.
    startTime: now-1h
    endTime: now

    # --------------------------- Backpressure Options -----------------------

  # 'queryInterval' sets the interval between queries to the remote API
    queryInterval: $=parseDuration("2s")

    # 'totalDuration' sets the max duration to try reading from the the remote input
    totalDuration: $=parseDuration("5min")

    # 'totalBytesLimit' sets the max total bytes to read from the remote input
    totalBytesLimit: $=parseBytes("50MB")

    # 'totalEventsLimit' sets the max number of events to read the remote input
    totalEventsLimit: 10000

    # --------------------------- Ancillary Options ---------------------------

    # 'printProgress' controls whether to print a progress gage to the console
    #  This option helps debug and test the input
    printProgress: $=!TenXEnv.get("quiet")

    # --------------------------- Advanced Options ---------------------------

    # 'querySize' sets the number of documents retrieved with each results page.
    querySize: 500

    # 'query' sets the elastic query to request paginated documents from the  '/_search' REST endpoint.
    #  The default query (defined in 'input/analyzer/elasticsearch/query.txt')
    #  retrieves paginated values sorted in desc order by @timestamp
    query: null

Logz.io

Below is the default configuration from: logzio/config.yaml (* Required Fields).

Edit Online

# 🔟❎ 'run' Logz.io ElasticSearch input configuration

# Configures a Logz.io ElasticSearch event input
# To learn more see https://doc.log10x.com/run/input/analyzer/elastic/

# Set the 10x pipeline to 'run'
tenx: run

# =============================== Dependencies ================================

include: run/modules/input/analyzer/elasticsearch

# =============================== Logzio Options ==============================

# Multiple Logz.io inputs can be defined below:
elasticSearch:

    # 'name' sets a unique logical name across all pipeline inputs
  - name: logzioElasticSearch

    # --------------------------- Connection Options --------------------------

    # 'token' must be set to the Logz.io API token.
    #  See: https://docs.logz.io/user-guide/tokens/api-tokens.html
    token: null # (❗ REQUIRED)

    # 'host' and 'port' set the Logz.io service address to connect to
    host: api.logz.io
    port: 443

    # ----------------------------- Query Options -----------------------------

    # 'targets' lists data streams, indices, and aliases to search.
    #  To search all data streams and indices, use '*' or '_all'.
    targets:
      - v1

    # 'startTime' and 'endTime' set the time frame of the query.
    startTime: now-5m
    endTime: now

    # --------------------------- Backpressure Options -----------------------

    # 'queryInterval' sets the interval between queries to the remote API
    queryInterval: $=parseDuration("2s")

    # 'totalDuration' sets the max duration to try reading from the the remote input 
    totalDuration: $=parseDuration("5min")

    # 'totalBytesLimit' sets the max total bytes to read from the remote input
    totalBytesLimit: $=parseBytes("50MB")

    # 'totalEventsLimit' sets the max number of events to read the remote input
    totalEventsLimit: 10000

    # --------------------------- Ancillary Options ---------------------------

    # 'printProgress' controls whether to print a progress gage to the console
    #  This option helps debug and test the input
    printProgress: $=!TenXEnv.get("quiet")

    # --------------------------- Advanced Options ---------------------------

    # 'querySize' sets the number of documents retrieved with each results page.
    querySize: 500

    # 'query' sets the elastic query to request paginated documents from the  '/_search' REST endpoint.
    #  The default query (defined in 'input/analyzer/elasticsearch/query.txt')
    #  retrieves paginated values sorted in desc order by @timestamp
    query: null

    # 'endpoint' sets the REST API endpoint to query
    endpoint: search

    # 'messageField' defines the name of the field to extract from the retrieved
    # documents containing the log message.
    messageField: message

Coralogix

Below is the default configuration from: coralogix/config.yaml (* Required Fields).

Edit Online

# 🔟❎ 'run' Coralogix ElasticSearch input configuration

# This configuration defines a Coralogix ElasticSearch input from
# which events are read and transformed into TenXObjects for aggregation and reporting.

# Set the 10x pipeline to 'run'
tenx: run

# =============================== Dependencies ================================

include: run/modules/input/analyzer/elasticsearch

# ============================== Coralogix Options ============================

# Multiple Coralogix inputs can be defined below
elasticSearch:

    # 'name' sets a unique logical name across all pipeline inputs
  - name: coralogixElasticSearch

    # ------------------------- Connection Options ----------------------------

    # 'token' must be set to the Coralogix send data API key
    #  To learn more see: https://coralogix.com/docs/user-guides/account-management/api-keys/send-your-data-api-key/
    token: null # (❗ REQUIRED)

    # 'authorizationKeyName' must be set to match Coralogix auth type
    #  To learn more see: https://coralogix.com/docs/user-guides/visualizations/hosted-opensearch-view/opensearch-api/
    authorizationKeyName: Bearer

    # 'host' and 'port' set the Coralogix service address to connect to
    host: api.coralogix.us/data/os-api

    # --------------------------- Query Options -------------------------------

    # 'targets' lists data streams, indices, and aliases to search.
    #  To search all data streams and indices, use '*' or '_all.'
    targets:
      - '*'

    # 'startTime' and 'endTime' set the time frame of the query.
    startTime: now-5m
    endTime: now

    # --------------------------- Backpressure Options -----------------------

    # 'queryInterval' sets the interval between queries to the remote API
    queryInterval: $=parseDuration("2s")

    # 'totalDuration' sets the max duration to try reading from the the remote input 
    totalDuration: $=parseDuration("5min")

    # 'totalBytesLimit' sets the max total bytes to read from the remote input
    totalBytesLimit: $=parseBytes("50MB")

    # 'totalEventsLimit' sets the max number of events to read the remote input
    totalEventsLimit: 10000

    # --------------------------- Ancillary Options ---------------------------

    # 'printProgress' controls whether to print a progress gage to the console
    #  This option helps debug and test the input
    printProgress: $=!TenXEnv.get("quiet")

    # --------------------------- Advanced Options ----------------------------

    # 'querySize' sets the number of documents retrieved with each results page.
    querySize: 500

    # 'query' sets the elastic query to request paginated documents from the  '/_search' REST endpoint.
    #  The default query (defined in 'input/analyzer/elasticsearch/query.txt')
    #  retrieves paginated values sorted in desc order by @timestamp
    query: null

    # 'timestampField' specifies the name of the field to get the event timestamp from.
    timestampField: coralogix.timestamp

    # 'messageField' defines the name of the field to extract from the retrieved
    #  documents containing the log message.
    messageField: log

Options

Specify the options below to configure multiple ElasticSearch input:

Name	Description	Category
elasticSearchName	Logical name for this elastic input	General
elasticSearchEnabled	Sets whether this input is enabled	General
elasticSearchPrintProgress	Sets whether this input prints throughput stats to the console	General
elasticSearchTotalDuration	Sets maximum time to query the backend	General
elasticSearchExtractors	List of extractors to apply on the events returning from the query	General
elasticSearchHost	Elastic host address	Authentication
elasticSearchPort	Elastic server port	Authentication
elasticSearchProtocol	Defines the protocol to connect to ElasticSearch	Authentication
elasticSearchToken	Elastic authentication token	Authentication
elasticSearchAuthorizationKeyName	Prefix for the elastic authentication token	Authentication
elasticSearchQuery	Query to pass to the elastic server	Query
elasticSearchMessageField	Name of the field used to extract the log message from retrieved documents	Query
elasticSearchTimestampField	Name of the field used to sort retrieved documents by	Query
elasticSearchStartTime or the number of milliseconds after Jan 1, 1970 00:00:00 UTC.")	The start of the time range, expressed as a relative string (i.e., "now-7d") or the number of milliseconds after Jan 1, 1970 00:00:00 UTC.	Query
elasticSearchEndTime or the number of milliseconds after Jan 1, 1970 00:00:00 UTC")	The end of the time range, expressed as a relative string (i.e., "now-7d") or the number of milliseconds after Jan 1, 1970 00:00:00 UTC	Query
elasticSearchTargets	List of data streams, indices, and aliases to search	Query
elasticSearchEndpoint	Name of search REST endpoint	Query
elasticSearchQuerySize	Number of documents to read per batch from the elastic server	Query
elasticSearchRetryTimeout	A timeout period (in milliseconds) to wait for querying of new data	Query
elasticSearchTotalBytesLimit	Maximum total bytes to read from input before closing	Backpressure
elasticSearchTotalEventsLimit	Maximum total events to read from input before closing	Backpressure
elasticSearchQueryInterval	Query interval (in milliseconds) for checking new data from remote source	Backpressure

General

`elasticSearchName`

Logical name for this elastic input.

Type	Default	Category
String	""	General

Sets a logical name (e.g., 'myElastic') for this input. The inputName field returns this value at run time to allow for identifying and operating on instances originating from this input.

`elasticSearchEnabled`

Sets whether this input is enabled.

Type	Default	Category
Boolean	true	General

Sets whether to open the input stream. To enable this input only when a 'elasticSearchHost' startup argument value is truthy, use:

elasticSearchEnabled: $=TenXEnv.get("elasticSearchHost")

To learn more see TenXEnv.get.

`elasticSearchPrintProgress`

Sets whether this input prints throughput stats to the console.

Type	Default	Category
Boolean	false	General

Sets whether this input prints throughput stats to the console for testing an integration to a remote endpoint.

`elasticSearchTotalDuration`

Sets maximum time to query the backend.

Type	Default	Category
String	5min	General

Sets the maximum time the pipeline will continue to query the backend before exiting. For example, setting this to '1min' means that queries to the backend will stop after 1 minute has passed.

`elasticSearchExtractors`

List of extractors to apply on the events returning from the query.

Type	Default	Category
List	[]	General

Sets list of extractor names which will work on the events returning from the query. For more info on extractors, see: https://doc.log10x.com/run/input/extract.

Authentication

`elasticSearchHost`

Elastic host address.

Type	Required	Category
String	✔	Authentication

Set the elastic host address to connect to. Examples: coralogix-esapi.coralogix.com api.logz.io .us-central1.gcp.cloud.es.io.

`elasticSearchPort`

Elastic server port.

Type	Default	Category
Number	0	Authentication

Sets the elastic server port to connect to (e.g., '9443') The port value is not needed if elasticSearchHost value encapsulates the port.

`elasticSearchProtocol`

Defines the protocol to connect to ElasticSearch.

Type	Default	Category
String	https	Authentication

Sets the protocol to connect to elastic with (e.g., https).

`elasticSearchToken`

Elastic authentication token.

Type	Required	Category
String	✔	Authentication

Sets the API token used to authenticate against the target elastic cluster. To learn more see: api tokens.

`elasticSearchAuthorizationKeyName`

Prefix for the elastic authentication token.

Type	Default	Category
String	ApiKey	Authentication

Sets the prefix for the provided elasticSearchToken when using it in an 'Authorization' header. commonly used values by different elastic vendors are 'ApiKey' and 'Bearer'.

Query

`elasticSearchQuery`

Query to pass to the elastic server.

Type	Default	Category
String	""	Query

Sets the elastic query to request paginated documents from the '/_search' REST endpoint. The default query (defined in 'elasticsearch/query.txt') retrieves paginated values sorted in desc order by @timestamp, starting from the value of 'now()'.

`elasticSearchMessageField`

Name of the field used to extract the log message from retrieved documents.

Type	Default	Category
String	log	Query

Sets the name of the field used to extract the log message from retrieved documents.

`elasticSearchTimestampField`

Name of the field used to sort retrieved documents by.

Type	Default	Category
String	@timestamp	Query

Sets the name of the field used to sort retrieved documents by.

`elasticSearchStartTime`

The start of the time range, expressed as a relative string (i.e., "now-7d") or the number of milliseconds after Jan 1, 1970 00:00:00 UTC.

Type	Default	Category
String	""	Query

Sets the start of the time range, expressed as a relative string (i.e., "now-7d") or the number of milliseconds after Jan 1, 1970 00:00:00 UTC. The query includes events with a timestamp equal or later to this value.

`elasticSearchEndTime`

The end of the time range, expressed as a relative string (i.e., "now-7d") or the number of milliseconds after Jan 1, 1970 00:00:00 UTC.

Type	Default	Category
String	""	Query

Sets the end of the time range, expressed as a relative string (i.e., "now-7d") or the number of milliseconds after Jan 1, 1970 00:00:00 UTC. The query excludes events with a timestamp later than this value.

`elasticSearchTargets`

List of data streams, indices, and aliases to search.

Type	Default	Category
List	[]	Query

Sets list of data streams, indices, and aliases to search. To search all data streams and indices, omit this parameter or use '*' or '_all'.

`elasticSearchEndpoint`

Name of search REST endpoint.

Type	Default	Category
String	_search	Query

Sets the name of the search REST endpoint.

`elasticSearchQuerySize`

Number of documents to read per batch from the elastic server.

Type	Default	Category
Number	500	Query

Sets the number of documents to retrieve with each page of results.

`elasticSearchRetryTimeout`

A timeout period (in milliseconds) to wait for querying of new data.

Type	Default	Category
Number	10000	Query

Determines how long to wait to invoke the search endpoint API again to query for new data. See elastic-input.js.

Backpressure

`elasticSearchTotalBytesLimit`

Maximum total bytes to read from input before closing.

Type	Default	Category
Number	50000000	Backpressure

sets the maximum number of bytes a target pipeline input will read into the pipeline. This value limits the volume of events to read from a local/remote source (e.g., log analyzer).

For example:

elasticSearchTotalBytesLimit: $=parseBytes("1GB")

`elasticSearchTotalEventsLimit`

Maximum total events to read from input before closing.

Type	Default	Category
Number	10000	Backpressure

Sets the maximum number of events a target pipeline input will read into the pipeline. This value limits the volume of events to read from a local/remote source (e.g., log analyzer).

`elasticSearchQueryInterval`

Query interval (in milliseconds) for checking new data from remote source.

Type	Default	Category
Number	2000	Backpressure

sets the interval between queries to the remote Elasticsearch API. This controls how frequently the input polls for new log data.

For example:

elasticSearchQueryInterval: $=parseDuration("5s")

This module is defined in elasticsearch/module.yaml.