CloudWatch Logs

Configures an AWS CloudWatch Logs input from which to read events to transform into typed TenXObjects.

Instances of this module define a connection to an AWS CloudWatch Logs service from which to retrieve log messages, as well as the querying logic used such as chronological direction, start values, time ranges, and page size of each API request sent.

Cloudwatch Logs inputs commonly run within scheduled jobs (e.g., k8s CronJob) to retrieve a recent sample amount of events (e.g., 200MB in the last 10min) to transform into TenXObjects as part of the Cloud Reporter app.

Configuration

To configure the AWS Cloudwatch Logs input module, Edit these settings.

Below is the default configuration from: cloudwatchLogs/config.yaml (* Required Fields).

Edit Online

# 🔟❎ 'run' AWS CloudWatch Logs input configuration

# Configure an AWS CloudWatch Logs event input
# To learn more see https://doc.log10x.com/run/input/analyzer/cloudwatchLogs/

# Set the 10x pipeline to 'run'
tenx: run

# =============================== Dependencies ================================

include: run/modules/input/analyzer/cloudwatchLogs

# ============================ CloudWatch Options =============================

# Multiple CloudWatch inputs can be defined below
cloudwatchLogs:

    # 'name' sets a unique logical name across all pipeline inputs
  - name: CloudwatchLogs

    # --------------------------- Connection Options --------------------------

    # 'awsAccessKeyID' sets the AWS access key (e.g. 'AKIAIOSFODNN7EXAMPLE')
    #  To learn more see: https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html
    awsAccessKeyID: $=TenXEnv.get("AWS_ACCESS_KEY_ID") # (❗ EnvVar REQUIRED)

    # 'awsSecretKey' sets the AWS access secret key  (e.g. 'wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY')
    #  To learn more see: https://docs.aws.amazon.com/general/latest/gr/aws-sec-cred-types.html
    awsSecretKey: $=TenXEnv.get("AWS_SECRET_ACCESS_KEY") # (❗ EnvVar REQUIRED)

    # 'groupName' sets the log group name (e.g., 'my-log-group').
    #  To learn more see: https://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_GetLogEvents.html#CWL-GetLogEvents-request-logGroupName
    groupName: "" # (❗ REQUIRED)

    # 'streamName' sets the log stream name (e.g., 'my-log-stream').
    #  To learn more see: https://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_GetLogEvents.html#CWL-GetLogEvents-request-logStreamName
    streamName: "" # (❗ REQUIRED)

    # 'awsRegion' sets the region hosting the CloudWatch Logs stream
    #  To learn more see: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html
    awsRegion: $=TenXEnv.get("AWS_DEFAULT_REGION", "us-east-1")

    # ----------------------------- Query Options -----------------------------

    # 'limit' sets the number of documents retrieved with each results page.
    #  To learn more see: https://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_GetLogEvents.html#CWL-GetLogEvents-request-limit
    limit: 500

    # 'startFromHead' controls the direction of the search (old <-> new events).
    #  To learn more see: https://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_GetLogEvents.html#CWL-GetLogEvents-request-startFromHead
    startFromHead: true

    # 'startTime' sets the start of the time range, expressed as the number of milliseconds after Jan 1, 1970 00:00:00 UTC.
    #  The search includes events with a timestamp equal to or later than this value.
    #  The search excludes events with a timestamp earlier than this value. 
    #  To learn more see: https://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_GetLogEvents.html#CWL-GetLogEvents-request-startTime
    startTime: $=now("-240h")

    # 'endTime' sets the end of the time range, expressed as the number of milliseconds after Jan 1, 1970 00:00:00 UTC.
    #  The search excludes events with a timestamp equal to or later than this value.
    #  To learn more see: https://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_GetLogEvents.html#CWL-GetLogEvents-request-endTime
    endTime: $=now()

    # --------------------------- Backpressure Options -----------------------

    # 'queryInterval' sets the interval between queries to the remote API
    queryInterval: $=parseDuration("2s")

    # 'totalDuration' sets the max duration to try reading from the the remote input 
    totalDuration: $=parseDuration("5min")

    # 'totalBytesLimit' sets the max total bytes to read from the remote input
    totalBytesLimit: $=parseBytes("50MB")

    # 'totalEventsLimit' sets the max number of events to read the remote input
    totalEventsLimit: 10000

    # --------------------------- Ancillary Options ---------------------------

    # 'printProgress' controls whether to print a gage to the console
    #  This option helps debug and test the input
    printProgress: $=!TenXEnv.get("quiet")

    # 'messageField' sets the JSON field to extract from CloudWatch log messages
    #  For K8s/Docker logs with 10x encoding, use 'log' to extract the encoded log field
    messageField: log

Options

Specify the options below to configure multiple AWS Cloudwatch Logs input:

Name	Description	Category
cloudwatchLogsName	Logical name for this AWS CloudWatch Logs input	General
cloudwatchLogsEnabled	Sets whether this input is enabled	General
cloudwatchLogsPrintProgress	Sets whether this input prints throughput stats to the console	General
cloudwatchLogsAwsRegion	AWS region	Authentication
cloudwatchLogsAwsAccessKeyID	AWS authentication access key ID	Authentication
cloudwatchLogsAwsSecretKey	AWS authentication secret key	Authentication
cloudwatchLogsGroupName	AWS CloudWatch Logs Group name	Log group
cloudwatchLogsStreamName	AWS CloudWatch Logs Stream name	Log group
cloudwatchLogsLimit	Number of events to read per batch from the AWS CloudWatch Logs service	Query
cloudwatchLogsNextToken	The token for the next set of items to return (received this token from a previous call)	Query
cloudwatchLogsStartFromHead	If true, return the earliest log events first. If false, return the most recent log events first.	Query
cloudwatchLogsStartTime	The start of the time range, expressed as the number of milliseconds after Jan 1, 1970 00:00:00 UTC.	Query
cloudwatchLogsEndTime	The end of the time range, expressed as the number of milliseconds after Jan 1, 1970 00:00:00 UTC	Query
cloudwatchLogsRetryTimeout	A timeout period (in milliseconds) to wait for querying of new data	Query
cloudwatchLogsTotalBytesLimit	Maximum total bytes to read from input before closing	Backpressure
cloudwatchLogsTotalEventsLimit	Maximum total events to read from input before closing	Backpressure
cloudwatchLogsTotalDuration	Maximum duration to keep input open before closing	Backpressure
cloudwatchLogsQueryInterval	Query interval (in milliseconds) for checking new data from remote source	Backpressure
cloudwatchLogsMessageField	JSON field name to extract from CloudWatch log messages	Extraction

General

`cloudwatchLogsName`

Logical name for this AWS CloudWatch Logs input.

Type	Required	Category
String	✔	General

Sets a logical name (e.g., 'myCloudWatchLogs') for this input. The inputName field returns this value at run time to allow for identifying and operating on instances originating from this input.

`cloudwatchLogsEnabled`

Sets whether this input is enabled.

Type	Default	Category
Boolean	false	General

Sets whether to open the input stream. To enable this input only when a 'cloudwatchLogsName' startup argument value is truthy, use:

cloudwatchLogsEnabled: $=TenXEnv.get("cloudwatchLogsName")

To learn more see TenXEnv.get.

`cloudwatchLogsPrintProgress`

Sets whether this input prints throughput stats to the console.

Type	Default	Category
Boolean	false	General

Sets whether this input prints throughput stats to the console for testing an integration to a remote endpoint.

Authentication

`cloudwatchLogsAwsRegion`

AWS region.

Type	Default	Category
String	""	Authentication

Sets the AWS region (e.g., 'us-east-1') hosting the target CloudWatch logs stream.

To learn more see AWS Regions.

`cloudwatchLogsAwsAccessKeyID`

AWS authentication access key ID.

Type	Default	Category
String	""	Authentication

Sets the AWS access key (e.g., 'AKIAIOSFODNN7EXAMPLE') used as part of the request signing process.

If not specified, the system automatically falls back to the AWS default credential provider chain, which checks (in order):

Environment variables (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY)
Java system properties
Web identity token (for EKS)
Shared credentials file (~/.aws/credentials)
ECS container credentials
EC2 instance profile credentials

To learn more see AWS access keys.

`cloudwatchLogsAwsSecretKey`

AWS authentication secret key.

Type	Default	Category
String	""	Authentication

Sets the AWS access secret key (e.g. 'wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY') used as part of the request signing process.

If not specified, the system automatically falls back to the AWS default credential provider chain. This allows using credentials from ~/.aws/credentials, IAM roles, or environment variables without explicit configuration.

To learn more see AWS security credentials.

Log group

`cloudwatchLogsGroupName`

AWS CloudWatch Logs Group name.

Type	Required	Category
String	✔	Log group

Used within the body of the 'GetLogEvents' AWS request to specify the log group name (e.g., 'my-log-group').

To learn more see logGroupName.

`cloudwatchLogsStreamName`

AWS CloudWatch Logs Stream name.

Type	Required	Category
String	✔	Log group

Used within the body of the AWS 'AGetLogEvents' request to specify the log stream name (e.g., 'my-log-stream').

To learn more see logStreamName.

Query

`cloudwatchLogsLimit`

Number of events to read per batch from the AWS CloudWatch Logs service.

Type	Default	Category
Number	500	Query

Sets the number of documents to retrieve with each page of results. To learn more see: request limit.

`cloudwatchLogsNextToken`

The token for the next set of items to return (received this token from a previous call).

Type	Default	Category
String	""	Query

Sets the token from which the requested result page is to start. This value is returned from a previous call to the AWS 'GetLogEvents' command. To learn more see: nextToken.

`cloudwatchLogsStartFromHead`

If true, return the earliest log events first. If false, return the most recent log events first.

Type	Default	Category
Boolean	false	Query

Controls the direction of the search (old \<-> new events). By default, the search starts from the most recent events. To learn more see: startFromHead.

`cloudwatchLogsStartTime`

The start of the time range, expressed as the number of milliseconds after Jan 1, 1970 00:00:00 UTC.

Type	Default	Category
Number	0	Query

Sets the start of the time range, expressed as milliseconds after Jan 1, 1970 00:00:00 UTC. Events with a timestamp equal to this time or later than this time are included. Events with a timestamp earlier than this time are excluded: To learn more see: startTime.

`cloudwatchLogsEndTime`

The end of the time range, expressed as the number of milliseconds after Jan 1, 1970 00:00:00 UTC.

Type	Default	Category
Number	0	Query

Sets the end of the time range, expressed as milliseconds after Jan 1, 1970 00:00:00 UTC. Events with a timestamp equal to or later than this time are not included: To learn more see: endTime.

`cloudwatchLogsRetryTimeout`

A timeout period (in milliseconds) to wait for querying of new data.

Type	Default	Category
Number	20000	Query

Used by cloudwatchLogs-input.js to determine how long to wait to invoke the 'GetLogEvents' API again to query for new data.

Backpressure

`cloudwatchLogsTotalBytesLimit`

Maximum total bytes to read from input before closing.

Type	Default	Category
Number	50000000	Backpressure

sets the maximum number of bytes a target pipeline input will read into the pipeline. This value limits the volume of events to read from a local/remote source (e.g., log analyzer).

For example:

totalBytesLimit: $=parseBytes("1GB")

`cloudwatchLogsTotalEventsLimit`

Maximum total events to read from input before closing.

Type	Default	Category
Number	10000	Backpressure

Sets the maximum number of events a target pipeline input will read into the pipeline. This value limits the volume of events to read from a local/remote source (e.g., log analyzer).

`cloudwatchLogsTotalDuration`

Maximum duration to keep input open before closing.

Type	Default	Category
String	parseDuration("5min")	Backpressure

sets the maximum duration a target pipeline input will remain open. When reached, the input will close and no more data will be read.

For example:

cloudwatchLogsTotalDuration: $=parseDuration("10min")

`cloudwatchLogsQueryInterval`

Query interval (in milliseconds) for checking new data from remote source.

Type	Default	Category
Number	2000	Backpressure

sets the interval between queries to the remote CloudWatch Logs API. This controls how frequently the input polls for new log data.

For example:

cloudwatchLogsQueryInterval: $=parseDuration("5s")

Extraction

`cloudwatchLogsMessageField`

JSON field name to extract from CloudWatch log messages.

Type	Default	Category
String		Extraction

sets the JSON field name to extract from each CloudWatch log message. When set, the extractor will look for this field in each message's JSON payload. When empty, the message extraction step is skipped.

For example, to extract the 'log' field from Kubernetes/Docker logs:

messageField: log

This module is defined in cloudwatchLogs/module.yaml.