Datadog

Publishes TenXSummary values to Datadog Micrometer.

Availability

This implementation is only available by default in the 10x Engine Compiler flavor to reduce the footprint of the Runtime flavor.

What Datadog counts

Datadog bills a custom metric for each distinct metric name and tag set. The shipped output tags each series with the pattern and where it ran (message_pattern, tenx_hash, tenx_user_service, severity_level, routeState, k8s_container, k8s_namespace, and tenx_team when k8sTeamLabel names a pod label) and with the runtime attributes, and with nothing that names one node or one process, so the same series comes from every node. It publishes the 200 tag sets with the most bytes (topSeries), 800 custom metrics with the two shipped aggregators, inside a Pro plan's allotment from ten hosts up. A pattern outside the top is published once it costs more than the smallest one inside.

Config Files

To configure the Datadog metrics output module, Edit these files.

Below is the default configuration from: datadog/config.yaml (** Required Fields*).

Edit Online Edit Locally

Edit config.yaml Locally

Download and save to:

• Linux / Docker / macOS
• Custom directory
  $TENX_CONFIG/run/output/metric/datadog/config.yaml
• Within cloned repo
  ./pipelines/run/output/metric/datadog/config.yaml

Download

# 🔟❎ 'run' Datadog metric output configuration

# Configure Datadog metric counter outputs.
# To learn more see https://doc.log10x.com/run/output/metric/datadog/

# Set the 10x pipeline to 'run'
tenx: run

# =============================== Dependencies ================================

include: run/modules/output/metric/datadog

# ============================== Datadog Options ==============================

# Define multiple Datadog metric outputs below
datadog:

    # --------------------------- Connection Options --------------------------

    # 'apiKey' and 'applicationKey' set the Datadog API and app keys (required).
    #  To learn more see: https://docs.datadoghq.com/account_management/api-app-keys/
  - apiKey: $=TenXEnv.get("DD_API_KEY")         # (❗ EnvVar REQUIRED)

    applicationKey: $=TenXEnv.get("DD_APP_KEY") # (❗ EnvVar REQUIRED)

    # 'uri' sets the Datadog site address to which to connect.
    #  'DD_SITE' is the bare site domain (e.g. 'us5.datadoghq.com', 'datadoghq.eu'),
    #  matches the Datadog Agent convention so existing cluster secrets work as-is.
    #  To learn more see: https://docs.datadoghq.com/getting_started/site/
    uri: $="https://" + TenXEnv.get("DD_SITE", "us5.datadoghq.com")

    # ---------------------------- Publish Options ----------------------------

    # 'descriptions' sets whether to send meter descriptions to Datadog
    #  Turn this off to minimize the amount of data sent on each write
    descriptions: false

    # 'step' sets the interval to send metrics to CloudWatch (e.g., '1m').
    #  To learn more see: https://micrometer.io/docs/concepts#rate-aggregation
    step: 5s

    # ---------------------------- Counter Options ----------------------------

    # 'nameField' sets the TenXSummary field whose value identifies the micrometer counter to increment. 
    #  The 'inputName' field returns the name of the aggregator that produced the target TenXSummary instance.
    #  To learn more see https://doc.log10x.com/run/aggregate
    nameField: inputName

    # 'counterFields' specifies the TenXSummary fields whose values are used to increment the counter.
    #  To learn more see https://doc.log10x.com/run/aggregate/#summary-fields
    counterFields:
      - summaryVolume
      - summaryBytes
      - summaryTotals

    # 'tagFields' sets the TenXSummary fields used as the counter tag dimension values.
    #  Datadog counts one custom metric per distinct tag set, so this list carries the
    #  pattern, where it ran and the team that owns it (when the k8s team label is set),
    #  and nothing that names one node or one process. The same set on every node, so the
    #  count does not grow with the fleet. A listed field that no summary carries is sent
    #  as a tag with an empty value, so the team field is listed only when the label is set.
    tagFields:
      - message_pattern
      - tenx_hash
      - tenx_user_service
      - severity_level
      - routeState=l1e_routeState
      - k8s_container
      - k8s_namespace
      - $=yield TenXEnv.get("k8sTeamLabel") ? TenXEnv.get("k8sTeamField", "tenx_team"):""

    # 'topSeries' keeps the output to the tag sets with the most bytes, ranked on the
    #  summaryBytes counter as the series accumulate. A set outside the top keeps
    #  accumulating and is published once it enters it. 0 publishes every set. With
    #  two aggregators and two counters, 200 sets are 800 custom metrics, inside a
    #  Pro plan's allotment from ten hosts up.
    topSeries: 200

    # 'tagFieldNames' allows setting custom names to aggregated fields for metrics reporting.
    #  Defaults to https://doc.log10x.com/run/initialize/#metricFieldNames
    tagFieldNames:
      - $=yield TenXEnv.get("metricFieldNames")

    # 'filter' sets a JavaScript expression that TenXSummaries must evaluate as truthy to write to the output.
    #  To learn more see https://doc.log10x.com/run/output/receive/#filter-expressions
    filter: isSummary

Options

Specify the options below to configure multiple Datadog metrics output:

Name	Description	Category
datadogEnabled	Enable this output	General
datadogNameField	Name of the TenXSummary field identifying the metric	Counter
datadogCounterFields	Names of the TenXSummary fields used to increment the metric counter value	Counter
datadogTagFields	List of TenXSummary fields to use as metric tag values	Counter
datadogTagFieldNames	List of custom names to use for TenXSummary fields.	Counter
datadogFilter	A JavaScript expression an TenXSummary must evaluate as 'truthy' against to be written to this output	Counter
datadogMaxCardinality	Maximum number of unique metric combinations allowed	Counter
datadogTopSeries	Number of tag sets with the most bytes to publish	Counter
datadogDescriptions	Sets whether to send meter descriptions to Datadog scrape	Counter
datadogStep	The interval at which to publish metrics to output	Counter
datadogApiKey	Datadog apiKey	Authentication
datadogApplicationKey	Datadog application Key.	Authentication
datadogHostTag	Value to map to "host" when shipping metrics to Datadog	Authentication
datadogUri	The URI of an internal proxy en route to Datadog.	Authentication

General

datadogEnabled

Enable this output.

Type	Default	Category
Boolean	true	General

Enable or disable this output.

Counter

datadogNameField

Name of the TenXSummary field identifying the metric.

Type	Default	Category
String	summaryValues	Counter

Defines the name of the TenXSummary field whose value identifies a metric counter. Defaults to the summaryValues field which holds the value(s) by which TenXObjects are grouped into the current TenXSummary instance. To learn more see micrometer counters.

datadogCounterFields

Names of the TenXSummary fields used to increment the metric counter value.

Type	Default	Category
List	summaryVolume	Counter

Defines the name of the TenXSummary field by whose value to increment the counter.

datadogTagFields

List of TenXSummary fields to use as metric tag values.

Type	Default	Category
List	[]	Counter

Defines the TenXSummary field names to use as tag values. Tags are dimensions that enable slicing a particular named metric to drill down to reason about its value. To learn more see micrometer tags

Datadog counts one custom metric per distinct tag set, so the shipped list names the pattern and where it ran, and nothing that names one node or one process: message_pattern, tenx_hash, tenx_user_service, severity_level, routeState, k8s_container and k8s_namespace, plus tenx_team when k8sTeamLabel names a pod label. The same set comes from every node, so the count does not grow with the fleet or with restarts. To tag the host as well, add it as a runtime attribute, host_name:<host>.

datadogTagFieldNames

List of custom names to use for TenXSummary fields.

Type	Default	Category
List	[]	Counter

Defines the a list of custom names to give fields when reporting them. Format as a list of name1, custom1, name2, custom2.

For example, to set a custom name of 'My-Custom-Field' to an existing 'intrinsic-field':

datadog:
  tagFieldNames:
    - intrinsic-field
    - My-Custom-Field.

datadogFilter

A JavaScript expression an TenXSummary must evaluate as 'truthy' against to be written to this output.

Type	Default	Category
String	""	Counter

Specifies a JavaScript expression that an TenXSummary must evaluate as truthy to write its instance/template field values to the target counter.

For example, to only emit TenXSummaries produced by the 'myMetric' aggregator, specify:

datadog:
  filter: this.inputName == "myMetric"

To learn more see inputName.

datadogMaxCardinality

Maximum number of unique metric combinations allowed.

Type	Default	Category
Number	500	Counter

Sets the maximum number of unique metric combinations (cardinality) allowed for this output. When exceeded, metrics will be dropped to prevent excessive cardinality.

datadogTopSeries

Number of tag sets with the most bytes to publish.

Type	Default	Category
Number	200	Counter

Keeps the output to the tag sets with the most bytes, ranked on the summaryBytes counter as the series accumulate. A tag set outside the top keeps accumulating and is published once it has cost more than the smallest one inside, which stops. 0 publishes every tag set.

With the two shipped aggregators and two counters each, 200 tag sets are 800 custom metrics, inside a Pro plan's allotment of 100 per host from ten hosts up; the top 200 patterns carried 98.8% of the bytes on the OpenTelemetry demo capture.

datadogDescriptions

Sets whether to send meter descriptions to Datadog scrape.

Type	Default	Category
Boolean	false	Counter

Sets whether to send meter descriptions to Datadog Turn this off to minimize the amount of data sent on each write.

datadogStep

The interval at which to publish metrics to output.

Type	Default	Category
String	1 minute.	Counter

Sets the interval at which to publish metrics to the time-series destination (e.g., '1m'). To learn more see rate-aggregation.

Authentication

datadogApiKey

Datadog apiKey.

Type	Default	Category
String	""	Authentication

Sets the Datadog api key.

datadogApplicationKey

Datadog application Key.

Type	Default	Category
String	""	Authentication

Sets the Datadog app Key.

datadogHostTag

Value to map to "host" when shipping metrics to Datadog.

Type	Default	Category
String	""	Authentication

Value to map to "host" when shipping metrics to Datadog.

datadogUri

The URI of an internal proxy en route to Datadog.

Type	Default	Category
String	https://api.datadoghq.com	Authentication

Sets the Datadog site address to connect to (e.g. 'us5.datadoghq.com'). To learn more see Datadog sites.

This module is defined in datadog/module.yaml.