S3 Logs

The S3 Logs module enables reading log and trace events from a single AWS S3 object for analysis and transformation into TenXObjects.

How It Works

The S3 Logs input fetches a single S3 object by key using the GetObject API. The object's line-delimited content is then split into individual events for processing.

graph LR
    A["Timer<br/>Trigger"] --> B["GetObject<br/>API"]
    B --> C["Stream<br/>Content"]
    C --> D["Split by<br/>Newlines"]
    D --> E["Transform to<br/>TenXObjects"]
    E --> F["Close<br/>Route"]

    classDef trigger fill:#2563eb88,stroke:#1d4ed8,color:#ffffff
    classDef api fill:#ea580c88,stroke:#c2410c,color:#ffffff
    classDef process fill:#05966988,stroke:#047857,color:#ffffff

    class A trigger
    class B api
    class C,D,E process
    class F trigger

Authentication

The module supports the AWS default credential provider chain:

Environment variables (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY)
Java system properties
Web identity token (for EKS/IRSA)
Shared credentials file (~/.aws/credentials)
ECS container credentials
EC2 instance profile credentials

Explicit credentials can also be provided via configuration:

s3Logs:
  - name: MyS3LogsInput
    bucket: my-bucket
    key: logs/app.log
    awsAccessKeyID: AKIAIOSFODNN7EXAMPLE
    awsSecretKey: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

Backpressure Controls

Control resource usage with backpressure options:

Option	Description	Default
`totalBytesLimit`	Max bytes to read	50MB
`totalEventsLimit`	Max events to read	10000
`totalDuration`	Max time to keep input open	5min

Example Configuration

tenx: run

include: run/modules/input/analyzer/s3Logs

s3Logs:
  - name: ProductionLogs
    bucket: my-company-logs
    key: logs/2024/01/15/application.log
    awsRegion: us-west-2
    totalBytesLimit: $=parseBytes("100MB")
    totalEventsLimit: 50000
    printProgress: true

CloudWatch Logs Input - Read from CloudWatch Logs
Object Storage Query - Query-based object storage access

Configuration

To configure the AWS S3 Logs input module, Edit these settings.

Below is the default configuration from: s3Logs/config.yaml (* Required Fields).

Edit Online

# 'run' AWS S3 Logs input configuration

# Configure an AWS S3 Logs event input to read a single S3 object
# To learn more see https://doc.log10x.com/run/input/analyzer/s3Logs/

# Set the 10x pipeline to 'run'
tenx: run

# =============================== Dependencies ================================

include: run/modules/input/analyzer/s3Logs

# ============================== S3 Logs Options ==============================

# Multiple S3 Logs inputs can be defined below
s3Logs:

    # 'name' sets a unique logical name across all pipeline inputs
  - name: S3LogsInput

    # --------------------------- Connection Options --------------------------

    # 'awsAccessKeyID' sets the AWS access key (e.g. 'AKIAIOSFODNN7EXAMPLE')
    #  To learn more see: https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html
    awsAccessKeyID: $=TenXEnv.get("AWS_ACCESS_KEY_ID") # (EnvVar or default credential chain)

    # 'awsSecretKey' sets the AWS access secret key  (e.g. 'wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY')
    #  To learn more see: https://docs.aws.amazon.com/general/latest/gr/aws-sec-cred-types.html
    awsSecretKey: $=TenXEnv.get("AWS_SECRET_ACCESS_KEY") # (EnvVar or default credential chain)

    # 'awsRegion' sets the region hosting the S3 bucket
    #  To learn more see: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html
    awsRegion: $=TenXEnv.get("AWS_DEFAULT_REGION", "us-east-1")

    # ---------------------------- S3 Object Options ---------------------------

    # 'bucket' sets the S3 bucket name (e.g., 'my-log-bucket').
    #  To learn more see: https://docs.aws.amazon.com/AmazonS3/latest/userguide/bucketnamingrules.html
    bucket: "" # (❗ REQUIRED)

    # 'key' sets the S3 object key to fetch (e.g., 'logs/app.log').
    #  To learn more see: https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObject.html
    key: "" # (❗ REQUIRED)

    # --------------------------- Backpressure Options -----------------------

    # 'totalDuration' sets the max duration to try reading from the remote input
    totalDuration: $=parseDuration("5min")

    # 'totalBytesLimit' sets the max total bytes to read from the remote input
    totalBytesLimit: $=parseBytes("50MB")

    # 'totalEventsLimit' sets the max number of events to read the remote input
    totalEventsLimit: 10000

    # --------------------------- Ancillary Options ---------------------------

    # 'printProgress' controls whether to print a gauge to the console
    #  This option helps debug and test the input
    printProgress: $=!TenXEnv.get("quiet")

Options

Specify the options below to configure multiple AWS S3 Logs input:

Name	Description	Category
s3LogsName	Logical name for this AWS S3 input	General
s3LogsEnabled	Sets whether this input is enabled	General
s3LogsPrintProgress	Sets whether this input prints throughput stats to the console	General
s3LogsAwsRegion	AWS region	Authentication
s3LogsAwsAccessKeyID	AWS authentication access key ID	Authentication
s3LogsAwsSecretKey	AWS authentication secret key	Authentication
s3LogsBucket	AWS S3 bucket name	Object
s3LogsKey	S3 object key to fetch	Object
s3LogsTotalBytesLimit	Maximum total bytes to read from input before closing	Backpressure
s3LogsTotalEventsLimit	Maximum total events to read from input before closing	Backpressure
s3LogsTotalDuration	Maximum duration to keep input open before closing	Backpressure

General

`s3LogsName`

Logical name for this AWS S3 input.

Type	Required	Category
String	✔	General

Sets a logical name (e.g., 'myS3Input') for this input. The inputName field returns this value at run time to allow for identifying and operating on instances originating from this input.

`s3LogsEnabled`

Sets whether this input is enabled.

Type	Default	Category
Boolean	false	General

Sets whether to open the input stream. To enable this input only when an 's3LogsBucket' startup argument value is truthy, use:

s3LogsEnabled: $=TenXEnv.get("s3LogsBucket")

To learn more see TenXEnv.get.

`s3LogsPrintProgress`

Sets whether this input prints throughput stats to the console.

Type	Default	Category
Boolean	false	General

Sets whether this input prints throughput stats to the console for testing an integration to a remote endpoint.

Authentication

`s3LogsAwsRegion`

AWS region.

Type	Default	Category
String	""	Authentication

Sets the AWS region (e.g., 'us-east-1') hosting the target S3 bucket.

To learn more see AWS Regions.

`s3LogsAwsAccessKeyID`

AWS authentication access key ID.

Type	Default	Category
String	""	Authentication

Sets the AWS access key (e.g., 'AKIAIOSFODNN7EXAMPLE') used as part of the request signing process.

If not specified, the system automatically falls back to the AWS default credential provider chain, which checks (in order):

Environment variables (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY)
Java system properties
Web identity token (for EKS)
Shared credentials file (~/.aws/credentials)
ECS container credentials
EC2 instance profile credentials

To learn more see AWS access keys.

`s3LogsAwsSecretKey`

AWS authentication secret key.

Type	Default	Category
String	""	Authentication

Sets the AWS access secret key (e.g. 'wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY') used as part of the request signing process.

If not specified, the system automatically falls back to the AWS default credential provider chain. This allows using credentials from ~/.aws/credentials, IAM roles, or environment variables without explicit configuration.

To learn more see AWS security credentials.

Object

`s3LogsBucket`

AWS S3 bucket name.

Type	Required	Category
String	✔	Object

Sets the S3 bucket name (e.g., 'my-log-bucket') containing the object to read.

To learn more see S3 Bucket Naming.

`s3LogsKey`

S3 object key to fetch.

Type	Required	Category
String	✔	Object

sets the S3 object key (e.g., 'logs/2024/01/15/app.log') to fetch.

This input reads a single S3 object and transforms its line-delimited content into TenXObjects for analysis.

For example:

s3Logs:
  - name: MyLogs
    bucket: my-bucket
    key: logs/application.log

Backpressure

`s3LogsTotalBytesLimit`

Maximum total bytes to read from input before closing.

Type	Default	Category
String	50000000	Backpressure

sets the maximum number of bytes a target pipeline input will read into the pipeline. This value limits the volume of events to read from the S3 object.

For example:

s3LogsTotalBytesLimit: $=parseBytes("1GB")

`s3LogsTotalEventsLimit`

Maximum total events to read from input before closing.

Type	Default	Category
String	10000	Backpressure

Sets the maximum number of events a target pipeline input will read into the pipeline. This value limits the volume of events to read from the S3 object.

`s3LogsTotalDuration`

Maximum duration to keep input open before closing.

Type	Default	Category
String	parseDuration("5min")	Backpressure

sets the maximum duration a target pipeline input will remain open. When reached, the input will close and no more data will be read.

For example:

s3LogsTotalDuration: $=parseDuration("10min")

This module is defined in s3Logs/module.yaml.

S3 Logs

How It Works

Authentication

Backpressure Controls

Example Configuration

Related Modules

Configuration

Options

General

s3LogsName

s3LogsEnabled

s3LogsPrintProgress

Authentication

s3LogsAwsRegion

s3LogsAwsAccessKeyID

s3LogsAwsSecretKey

Object

s3LogsBucket

s3LogsKey

Backpressure

s3LogsTotalBytesLimit

s3LogsTotalEventsLimit

s3LogsTotalDuration

`s3LogsName`

`s3LogsEnabled`

`s3LogsPrintProgress`

`s3LogsAwsRegion`

`s3LogsAwsAccessKeyID`

`s3LogsAwsSecretKey`

`s3LogsBucket`

`s3LogsKey`

`s3LogsTotalBytesLimit`

`s3LogsTotalEventsLimit`

`s3LogsTotalDuration`