Datadog Logs

Configures a Datadog Logs input from which to read events to transform into typed TenXObjects.

Instances of this module define a connection to a Datadog Logs service from which to retrieve log messages, as well as the querying logic used such as chronological direction, start values, time ranges, and page size of each API request sent.

Datadog Logs inputs commonly run within scheduled jobs (e.g., k8s CronJob) to retrieve a recent sample amount of events (e.g., 200MB in the last 10min) to transform into TenXObjects as part of the Cloud Reporter app.

Configuration

To configure the Datadog Logs input module, Edit these settings.

Below is the default configuration from: datadogLogs/config.yaml (* Required Fields).

Edit Online

# 🔟❎ 'run' Datadog Logs input configuration

# Configure a Datadog Logs event input
# To learn more see https://doc.log10x.com/run/input/analyzer/datadogLogs/

# Set the 10x pipeline to 'run'
tenx: run

# =============================== Dependencies ================================

include: run/modules/input/analyzer/datadogLogs

# ============================== Datadog Inputs ===============================

# Multiple Datadog inputs can be defined below
datadogLogs:

    # 'name' sets a unique logical name across all pipeline inputs
  - name: DatadogLogs

    # -------------------------- Connection Options ---------------------------

    # 'apiKey' and 'appKey' must be set to authenticate with the logs endpoint
    #  To learn more see: https://docs.datadoghq.com/account_management/api-app-keys/
    apiKey: $=TenXEnv.get("DD_API_KEY") # (❗ REQUIRED)

    appKey: $=TenXEnv.get("DD_APP_KEY") # (❗ REQUIRED)

    # 'host' points to the target Datadog site
    #  To learn more see: https://docs.datadoghq.com/getting_started/site/
    host: us5.datadoghq.com

    # ----------------------------- Query Options -----------------------------

    # 'query' sets the Datadog query used to request paginated documents.
    #  The default query (defined in 'run/modules/input/analyzer/datadog/query.txt') retrieves
    #  values sorted in desc order by @timestamp, starting from the value of 'searchFrom'
    query: null

    # 'searchFrom sets the initial value used to request the first page of results.
    searchFrom: now-1000h

    # 'querySize' sets the number of documents to retrieve with each paginated request
    querySize: 500

    # 'indexes' sets the lists of data indices to search in
    # If not provided, will use a default of '*'
    indexes: null

    # --------------------------- Backpressure Options -----------------------

    # 'queryInterval' sets the interval between queries to the remote API
    queryInterval: $=parseDuration("5s")

    # 'totalDuration' sets the max duration to try reading from the the remote input 
    totalDuration: $=parseDuration("5min")

    # 'totalBytesLimit' sets the max total bytes to read from the remote input
    totalBytesLimit: $=parseBytes("50MB")

    # 'totalEventsLimit' sets the max number of events to read the remote input
    totalEventsLimit: 10000

    # --------------------------- Ancillary Options ---------------------------

    # 'printProgress' controls whether to print a progress gage to the console
    #  This option helps test the input
    printProgress: $=!TenXEnv.get("quiet")

Options

Specify the options below to configure multiple Datadog Logs input:

Name	Description	Category
datadogLogsName	Logical name for this DatadogLogs input	General
datadogLogsEnabled	Sets whether this input is enabled	General
datadogLogsPrintProgress	Sets whether this input prints throughput stats to the console	General
datadogLogsTotalDuration	Sets maximum time to query the backend	General
datadogLogsExtractors	List of extractors to apply on the events returning from the query	Advanced
datadogLogsHost	DatadogLogs host address	Authentication
datadogLogsApiKey	Datadog API key token	Authentication
datadogLogsAppKey	Datadog app key	Authentication
datadogLogsQuery	Query to pass to the Datadog 'api/v2/logs/events/search' REST endpoint	Query
datadogLogsIndexes	List of indices to search in	Query
datadogLogsQuerySize	Number of documents to read per batch from the Datadog server	Query
datadogLogsSearchFrom	Timestamp from which to begin the search	Query
datadogLogsRetryTimeout	A timeout period (in milliseconds) to wait for querying of new data	Query
datadogLogsTotalBytesLimit	Maximum total bytes to read from input before closing	Backpressure
datadogLogsTotalEventsLimit	Maximum total events to read from input before closing	Backpressure
datadogLogsQueryInterval	Query interval (in milliseconds) for checking new data from remote source	Backpressure

General

`datadogLogsName`

Logical name for this DatadogLogs input.

Type	Required	Category
String	✔	General

Sets a logical name (e.g., 'myDatadogLogs') for this input. The inputName field returns this value at run time to allow for identifying and operating on instances originating from this input.

`datadogLogsEnabled`

Sets whether this input is enabled.

Type	Default	Category
Boolean	true	General

Sets whether to open the input stream. To enable this input only when a 'datadogLogsHost' startup argument value is truthy, use:

datadogLogsEnabled: $=TenXEnv.get("datadogLogsHost")

To learn more see TenXEnv.get.

`datadogLogsPrintProgress`

Sets whether this input prints throughput stats to the console.

Type	Default	Category
Boolean	false	General

Sets whether this input prints throughput stats to the console for testing an integration to a remote endpoint.

`datadogLogsTotalDuration`

Sets maximum time to query the backend.

Type	Default	Category
String	5min	General

Sets the maximum time the pipeline will continue to query the datadog api backend before exiting. For example, setting this to '1min' means that queries to the backend will stop after 1 minute has passed.

Advanced

`datadogLogsExtractors`

List of extractors to apply on the events returning from the query.

Type	Default	Category
List	[]	Advanced

Sets list of extractor names which will work on the events returning from the query. For more info on extractors, see: https://doc.log10x.com/run/input/extract.

Authentication

`datadogLogsHost`

DatadogLogs host address.

Type	Required	Category
String	✔	Authentication

Sets the Datadog site address to connect to (e.g., 'us5.datadoghq.com'). To learn more see Datadog sites.

`datadogLogsApiKey`

Datadog API key token.

Type	Required	Category
String	✔	Authentication

Sets the 'DD-API-KEY' API key authentication request header. To learn more see: API keys.

`datadogLogsAppKey`

Datadog app key.

Type	Required	Category
String	✔	Authentication

Sets the 'DD-APPLICATION-KEY' app key authentication request header. To learn more see: API keys.

Query

`datadogLogsQuery`

Query to pass to the Datadog 'api/v2/logs/events/search' REST endpoint.

Type	Default	Category
String	""	Query

Sets the Datadog query used to request paginated documents from which to extract events to transform into TenXObjects. The default query retrieves paginated values sorted in desc order by @timestamp, starting from the value of 'now-15m'.

`datadogLogsIndexes`

List of indices to search in.

Type	Default	Category
List	[]	Query

Sets the lists of data indices to search in. To search all data streams and indices, omit this parameter or use * or _all.

`datadogLogsQuerySize`

Number of documents to read per batch from the Datadog server.

Type	Default	Category
Number	500	Query

Sets the number of documents to retrieve with each page of results.

`datadogLogsSearchFrom`

Timestamp from which to begin the search.

Type	Default	Category
String	now() in nano.	Query

This value sets the initial value used to request the first page of results.

`datadogLogsRetryTimeout`

A timeout period (in milliseconds) to wait for querying of new data.

Type	Default	Category
Number	10000	Query

Used by datadog-input.js to determine how long to wait until the next time the search endpoint API is invoked to query for new data.

Backpressure

`datadogLogsTotalBytesLimit`

Maximum total bytes to read from input before closing.

Type	Default	Category
Number	50000000	Backpressure

sets the maximum number of bytes a target pipeline input will read into the pipeline. This value limits the volume of events to read from a local/remote source (e.g., log analyzer).

For example:

datadogLogsTotalBytesLimit: $=parseBytes("1GB")

`datadogLogsTotalEventsLimit`

Maximum total events to read from input before closing.

Type	Default	Category
Number	10000	Backpressure

Sets the maximum number of events a target pipeline input will read into the pipeline. This value limits the volume of events to read from a local/remote source (e.g., log analyzer).

`datadogLogsQueryInterval`

Query interval (in milliseconds) for checking new data from remote source.

Type	Default	Category
Number	2000	Backpressure

sets the interval between queries to the remote Datadog Logs API. This controls how frequently the input polls for new log data.

For example:

datadogLogsQueryInterval: $=parseDuration("5s")

This module is defined in datadogLogs/module.yaml.