Deploy

Deploy the Storage Streamer app to AWS EKS via Terraform using the terraform-aws-tenx-streamer module.

Step 1: Prerequisites

Requirement	Description
Log10x License	Your license key (get one)
EKS Cluster	Existing Kubernetes cluster on AWS with OIDC provider configured
AWS CLI	Configured with appropriate credentials
Terraform	Terraform >= 1.0
kubectl	Configured to access your cluster
OIDC Provider Info	Your EKS cluster's OIDC provider ARN and URL (see AWS docs)
AWS IAM Permissions	Permissions to create S3 buckets, SQS queues, and IAM roles
GitHub Token	Optional, for GitOps: Personal access token for config/symbols repo (create one) - see Step 5

Step 2: Deployment Method

The terraform-aws-tenx-streamer module provides automated deployment with complete AWS integration.

A single terraform apply automatically creates and deploys everything end-to-end:

• S3 Buckets: Source bucket for logs and index bucket for results (or use existing buckets)
• S3 Event Notification: Configures S3 to send ObjectCreated events directly to the Index SQS queue
• SQS Queues: Index, query, subquery, and stream queues with appropriate policies
• IAM Role: IRSA role with least-privilege permissions for S3 and SQS access
• Service Account: Kubernetes service account with IRSA annotation
• Helm Release: Deploys the streamer-10x Helm chart with your custom values file

The module uses a two-layer configuration approach:

1. Helm values file: Application configuration (clusters, replicas, resources, scaling)
2. Terraform overrides: Infrastructure details (S3 buckets, SQS queues, service account)

Step 3: Configure Infrastructure

The module requires your EKS cluster's OIDC provider information for IRSA authentication.

Get your EKS OIDC provider info:

# Get OIDC provider ARN
aws eks describe-cluster --name YOUR_CLUSTER_NAME \
  --query "cluster.identity.oidc.issuer" --output text

# Example output: https://oidc.eks.us-east-1.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE
# OIDC provider = oidc.eks.us-east-1.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE
# OIDC provider ARN = arn:aws:iam::ACCOUNT:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE

Create Terraform configuration:

main.tf

# Provider configuration - required for the module to access your cluster
provider "aws" {
  region = "us-east-1"  # Your AWS region
}

data "aws_eks_cluster" "cluster" {
  name = "YOUR_CLUSTER_NAME"
}

provider "kubernetes" {
  config_path    = "~/.kube/config"
  config_context = "YOUR_KUBECTL_CONTEXT"  # Run: kubectl config current-context
}

provider "helm" {
  # Note: 'kubernetes' is an attribute (uses '='), not a block
  kubernetes = {
    config_path    = "~/.kube/config"
    config_context = "YOUR_KUBECTL_CONTEXT"
  }
}

# Get current AWS account ID
data "aws_caller_identity" "current" {}

# Extract OIDC provider info from EKS cluster
locals {
  oidc_issuer = replace(data.aws_eks_cluster.cluster.identity[0].oidc[0].issuer, "https://", "")
}

module "tenx_streamer" {
  source  = "log-10x/tenx-streamer/aws"

  # Required: API key and OIDC provider for IRSA
  tenx_api_key      = var.tenx_api_key
  oidc_provider_arn = "arn:aws:iam::${data.aws_caller_identity.current.account_id}:oidc-provider/${local.oidc_issuer}"
  oidc_provider     = local.oidc_issuer

  # Kubernetes configuration
  namespace        = "log10x-streamer"
  create_namespace = true

  # Resource naming prefix (used for S3, SQS, IAM resources)
  resource_prefix = "my-app-streamer"

  # S3 bucket configuration (creates new buckets by default)
  # Note: S3 bucket names must be globally unique - uses account ID for uniqueness
  # Set create_s3_buckets = false to use existing buckets
  tenx_streamer_index_source_bucket_name  = "my-app-logs-${data.aws_caller_identity.current.account_id}"
  tenx_streamer_index_results_bucket_name = "my-app-index-${data.aws_caller_identity.current.account_id}"

  # S3 trigger configuration (which files trigger indexing)
  tenx_streamer_index_trigger_prefix = "app/"   # Only index files in app/ prefix
  tenx_streamer_index_trigger_suffix = ".log"   # Only index .log files

  # Application configuration via Helm values file
  helm_values_file = "${path.module}/values/streamer.yaml"

  # Tagging
  tags = {
    Environment = "production"
    ManagedBy   = "terraform"
  }
}

variable "tenx_api_key" {
  description = "Log10x API key"
  type        = string
  sensitive   = true
}

Finding your kubectl context: Run kubectl config current-context to get your context name. eksctl creates contexts like user@cluster.region.eksctl.io, while aws eks update-kubeconfig uses arn:aws:eks:REGION:ACCOUNT:cluster/NAME.

Using an existing EKS module: If you're using the terraform-aws-modules/eks/aws module, replace the OIDC configuration with:

oidc_provider_arn = module.eks.oidc_provider_arn
oidc_provider     = module.eks.oidc_provider

Using existing S3 buckets: To use existing S3 buckets instead of creating new ones:

# Disable bucket creation
create_s3_buckets = false

# Reference your existing bucket names
tenx_streamer_index_source_bucket_name  = "my-existing-logs-bucket"
tenx_streamer_index_results_bucket_name = "my-existing-logs-bucket"  # Can be same bucket

The module will still create the S3→SQS event notification on your existing bucket.

Step 4: Configure Application

Application configuration is provided via a Helm values file. The Terraform module automatically injects infrastructure details (S3 buckets, SQS queues, service account).

Create a values file for your deployment. Choose a cluster topology:

All-in-oneSeparate clusters per role

A single cluster handles all roles. Simpler to manage, suitable for development and small workloads.

values/streamer.yaml

clusters:
  - name: all-in-one
    roles: ["index", "query", "stream"]
    replicaCount: 2
    maxParallelRequests: 10
    maxQueuedRequests: 1000
    readinessThresholdPercent: 90

    resources:
      requests:
        cpu: 1000m
        memory: 2Gi
      limits:
        memory: 4Gi

    autoscaling:
      enabled: true
      minReplicas: 2
      maxReplicas: 10
      targetCPUUtilizationPercentage: 70

fluentBit:
  output:
    type: s3  # Options: stdout, s3, cloudwatch, elasticsearch, splunk, datadog
    config:
      s3:
        bucket: my-output-logs-bucket
        region: us-west-2

All pods show 2/2 READY (streamer + fluent-bit sidecar) since every pod includes the "stream" role.

Dedicated clusters for each role, allowing independent scaling and resource allocation. Each cluster consumes from its respective SQS queue:

• Index — processes S3 event notifications, CPU/memory intensive during file parsing
• Query — handles incoming search requests, parallelizes sub-queries across index partitions
• Stream — delivers matched events to the fluent-bit sidecar output

values/streamer.yaml

clusters:
  - name: indexer
    roles: ["index"]
    replicaCount: 2
    maxParallelRequests: 5
    maxQueuedRequests: 500
    readinessThresholdPercent: 90

    resources:
      requests:
        cpu: 2000m
        memory: 4Gi
      limits:
        memory: 6Gi

    autoscaling:
      enabled: true
      minReplicas: 2
      maxReplicas: 10
      targetCPUUtilizationPercentage: 70

  - name: query-handler
    roles: ["query"]
    replicaCount: 3
    maxParallelRequests: 20
    maxQueuedRequests: 1000
    readinessThresholdPercent: 90

    resources:
      requests:
        cpu: 1000m
        memory: 2Gi
      limits:
        memory: 4Gi

    autoscaling:
      enabled: true
      minReplicas: 3
      maxReplicas: 15
      targetCPUUtilizationPercentage: 70

  - name: stream-worker
    roles: ["stream"]
    replicaCount: 5
    maxParallelRequests: 15
    maxQueuedRequests: 1000
    readinessThresholdPercent: 90

    resources:
      requests:
        cpu: 1000m
        memory: 2Gi
      limits:
        memory: 4Gi

    autoscaling:
      enabled: true
      minReplicas: 5
      maxReplicas: 20
      targetCPUUtilizationPercentage: 70

fluentBit:
  output:
    type: cloudwatch
    config:
      cloudwatch:
        region: us-west-2
        logGroupName: /aws/eks/streamer-logs
        logStreamPrefix: stream-

Only stream-worker pods show 2/2 READY (streamer + fluent-bit sidecar). Index and query pods show 1/1.

The fluent-bit sidecar is automatically deployed for any cluster with "stream" in its roles. Without a fluentBit section, it defaults to stdout output. Add the fluentBit section only when you need to override the output destination for production.

Note: The Terraform module creates S3 buckets for log input and index storage, but not for Fluent Bit output. If using type: s3 for Fluent Bit, create the output bucket separately or reference an existing bucket.

Values injected automatically by the Terraform module (do not specify these):

• inputBucket and indexBucket (from S3 bucket configuration)
• indexQueueUrl, queryQueueUrl, subQueryQueueUrl, streamQueueUrl (from SQS)
• serviceAccount.create = false and serviceAccount.name (Terraform-managed)

Scheduled queries (periodic cron-based querying) are configured separately in Step 11.

Step 5: GitOps (optional)

Log10x uses GitOps to manage configuration centrally.

Setup steps:

1. Fork the Config Repository
2. Create a branch for your configuration
3. Edit the streamer app configuration

Add GitHub configuration to your Helm values file:

values/streamer.yaml

# GitHub token for authentication (required for GitOps)
githubToken: "YOUR-GITHUB-TOKEN"

# GitHub configuration for fetching pipeline config and compiled symbols
github:
  config:
    repo: "YOUR-ACCOUNT/REPO-NAME"
    branch: "my-cloud-streamer-config"
  symbols:
    repo: "YOUR-ACCOUNT/symbols-repo"
    branch: "main"
    path: "tenx/my-app/symbols"  # Optional subfolder

Sensitive token handling: The example above includes the token directly in the values file for simplicity. For production, consider using External Secrets Operator or creating a Kubernetes secret and referencing it via apiKeySecret.existingSecret.

Step 6: Configure Secrets

Store sensitive output destination credentials in Kubernetes Secrets.

Important: Only add secrets for outputs you've configured in your Storage Streamer app configuration.

Create the secret (example for Elasticsearch):

kubectl create secret generic cloud-streamer-credentials \
  --from-literal=elastic-username=YOUR_USERNAME \
  --from-literal=elastic-password=YOUR_PASSWORD \
  -n log10x-streamer

Add secret references to your Helm values file under each cluster's extraEnv:

values/streamer.yaml

clusters:
  - name: all-in-one
    roles: ["index", "query", "stream"]
    # ... other config ...

    # Secret references for this cluster
    extraEnv:
      - name: ELASTIC_USERNAME
        valueFrom:
          secretKeyRef:
            name: cloud-streamer-credentials
            key: elastic-username
      - name: ELASTIC_PASSWORD
        valueFrom:
          secretKeyRef:
            name: cloud-streamer-credentials
            key: elastic-password

# For Datadog output, add to clusters with "stream" role:
#   - name: DD_API_KEY
#     valueFrom:
#       secretKeyRef:
#         name: cloud-streamer-credentials
#         key: datadog-api-key

# For Splunk output, add to clusters with "stream" role:
#   - name: SPLUNK_HEC_TOKEN
#     valueFrom:
#       secretKeyRef:
#         name: cloud-streamer-credentials
#         key: splunk-hec-token

Step 7: Deploy

Ensure your AWS credentials are configured (e.g., AWS_PROFILE=my-profile or default credentials):

terraform init
terraform plan
terraform apply

Step 8: Verify Pods

kubectl get pods -n log10x-streamer

Pods with the "stream" role include a fluent-bit sidecar and show 2/2 READY. Pods with only "index" or "query" roles show 1/1 READY. In the all-in-one configuration (where all roles are combined), all pods show 2/2.

Check streamer console output for startup:

The streamer container name matches the cluster name in your values file. Select your topology:

All-in-oneSeparate clusters

kubectl logs -n log10x-streamer -l app=streamer-10x -c streamer-10x-all-in-one --tail=50

# Check any cluster (indexer, query-handler, or stream-worker)
kubectl logs -n log10x-streamer -l app=streamer-10x -c streamer-10x-indexer --tail=50

Expected startup output:

INFO  [com.log10x.ext.quarkus.executor.PipelineExecutor] (main) Initializing cloud accessor...
INFO  [com.log10x.ext.quarkus.access.aws.AWSAccessor] (main) AWS shared client cache initialized
INFO  [com.log10x.ext.quarkus.executor.PipelineExecutor] (main) Cloud accessor AWSAccessor initialized
INFO  [com.log10x.ext.quarkus.executor.PipelineExecutor] (main) Pipeline factory initialized
INFO  [io.quarkus] (main) run-quarkus 0.2.0 on JVM (powered by Quarkus 3.30.2) started in 1.873s. Listening on: http://0.0.0.0:8080
INFO  [io.quarkus] (main) Installed features: [amazon-sdk-sqs, cdi, rest, rest-jackson, scheduler, smallrye-context-propagation, smallrye-health, vertx]

Step 9: Verify Indexing

Upload a test file matching your trigger configuration to verify the S3→SQS→indexer pipeline:

# Create a test log file with current timestamp
echo "{\"timestamp\":\"$(date -u +%Y-%m-%dT%H:%M:%SZ)\",\"level\":\"ERROR\",\"message\":\"Test error\"}" > test.log

# Upload to the source bucket (must match trigger prefix/suffix)
aws s3 cp test.log s3://YOUR-LOGS-BUCKET/app/test.log

Check the streamer console output for indexing activity:

All-in-oneSeparate clusters

kubectl logs -n log10x-streamer -l app=streamer-10x -c streamer-10x-all-in-one --tail=50

kubectl logs -n log10x-streamer -l app=streamer-10x -c streamer-10x-indexer --tail=50

Expected indexing output:

INFO  [executor-thread-3] IndexTemplates - merged templates. templates.size: 1, rawTemplateFiles.size: 0, ...
INFO  [executor-thread-3] IndexFilterStats - index complete. Bytes: 21, filters.size: 1, probability historgram: [1=>1], key sizes: [0=>1], element counts: [50=>1], epochs: 1
INFO  [executor-thread-3] ExecutionPipeline - execution of: /etc/tenx/modules/pipelines/run/pipeline.yaml (myObjectStorageIndex) completed in: 7501ms

Verify index results were written to S3:

aws s3 ls s3://YOUR-INDEX-BUCKET/ --recursive

Index files are written to the path configured by tenx_streamer_index_results_path (defaults to the bucket root).

Step 10: Verify Querying

Submit a test query and verify events are streamed to the fluent-bit sidecar. The query JSON format is the same for both methods — see Defining Queries for the full parameter reference.

HTTP — All-in-oneHTTP — Separate clustersSQS (direct)

kubectl port-forward -n log10x-streamer svc/tenx-streamer-streamer-10x-all-in-one 8080:80 &

curl -X POST http://localhost:8080/streamer/query \
  -H "Content-Type: application/json" \
  -d '{"from": "now(\"-5m\")", "to": "now()", "search": "severity_level == \"ERROR\""}'

The endpoint returns HTTP 200 to acknowledge receipt.

kubectl port-forward -n log10x-streamer svc/tenx-streamer-streamer-10x-query-handler 8080:80 &

curl -X POST http://localhost:8080/streamer/query \
  -H "Content-Type: application/json" \
  -d '{"from": "now(\"-5m\")", "to": "now()", "search": "severity_level == \"ERROR\""}'

The endpoint returns HTTP 200 to acknowledge receipt.

Send the query JSON directly to the query SQS queue. No port-forward required — works from any machine with AWS credentials and access to the queue.

aws sqs send-message \
  --queue-url YOUR-QUERY-QUEUE-URL \
  --message-body '{"from": "now(\"-5m\")", "to": "now()", "search": "severity_level == \"ERROR\""}'

Get your query queue URL from the Terraform output:

terraform output query_queue_url

Check fluent-bit logs for streamed events:

Matched events are processed asynchronously via the SQS queues and streamed to the fluent-bit sidecar. Results typically appear within 10-30 seconds in production (up to 60 seconds in single-node setups).

kubectl logs -n log10x-streamer -l app=streamer-10x -c fluent-bit --tail=50

Expected output:

[0] com.log10x.ext.cloud.index.query.object.IndexObjectQueryReader0: [[1769076000.000000000, {}], {"timestamp"=>"2026-01-22T10:00:00Z", "level"=>"ERROR", "message"=>"Test error"}]

Once running, view your streaming analytics in the Storage Streamer Dashboard.

Step 11: Configure Scheduled Queries

The Helm chart can create Kubernetes CronJobs that periodically send query messages to the SQS query queue. Each job runs an AWS CLI container that submits one or more queries on a cron schedule.

Add a scheduledQueries section to your Helm values file:

values/streamer.yaml

scheduledQueries:
  enabled: true
  jobs:
    - name: hourly-errors
      schedule: "0 * * * *"  # Every hour (UTC)
      queries:
        - name: "error-logs"
          from: 'now("-1h")'
          to: 'now()'
          search: 'severity_level=="ERROR"'

    - name: daily-report
      schedule: "0 0 * * *"  # Daily at midnight (UTC)
      queries:
        - name: "daily-errors"
          from: 'now("-24h")'
          to: 'now()'
          search: 'severity_level=="ERROR"'
        - name: "daily-warnings"
          from: 'now("-24h")'
          to: 'now()'
          search: 'severity_level=="WARN"'

Each job creates a separate CronJob resource. A single job can contain multiple queries, which are submitted sequentially on each run.

Query Parameters

Parameter	Type	Required	Description
from	String	Yes	Start of the time range. Use now("-1h"), now("-24h"), now("-7d"), or a literal epoch.
to	String	Yes	End of the time range. Use now() for current time.
search	String	No	Search expression for pre-filtering via TenXTemplate filters.
processingTime	String	No	Max processing time per query (e.g., parseDuration("5m")). Defaults to 1 minute.
resultSize	String	No	Max result volume per query (e.g., parseBytes("100MB")).

Job Options

Parameter	Type	Default	Description
schedule	String	—	Cron expression (5 fields: minute hour day month weekday). Runs in UTC.
suspend	Boolean	false	Temporarily pause a job without deleting it.
concurrencyPolicy	String	Forbid	Prevents overlapping executions of the same job.
successfulJobsHistoryLimit	Integer	3	Number of successful job records to keep.
failedJobsHistoryLimit	Integer	1	Number of failed job records to keep.
restartPolicy	String	OnFailure	Pod restart policy for failed queries.

Managing scheduled queries:

# List all scheduled query CronJobs
kubectl get cronjobs -n log10x-streamer -l component=scheduled-query

# View recent job executions
kubectl get jobs -n log10x-streamer -l component=scheduled-query

# Manually trigger a scheduled query
kubectl create job --from=cronjob/tenx-streamer-streamer-10x-hourly-errors manual-run -n log10x-streamer

# Check logs for a specific job run
kubectl logs job/manual-run -n log10x-streamer

The CronJob pods use the same service account as the streamer, so they automatically inherit SQS permissions. Matched events are delivered to the fluent-bit sidecar output configured in Step 4.

Step 12: Teardown

To remove all Terraform-managed resources:

If the module created S3 buckets (create_s3_buckets = true, the default), you must empty them before destroying — S3 buckets cannot be deleted if they contain objects:

# Empty buckets first (skip if using create_s3_buckets = false)
aws s3 rm s3://YOUR-LOGS-BUCKET/ --recursive
aws s3 rm s3://YOUR-INDEX-BUCKET/ --recursive

# Destroy all Terraform-managed resources
terraform destroy

When using create_s3_buckets = false (see Step 3), Terraform does not manage the S3 buckets — only the S3→SQS event notification is created on them. This means terraform destroy will cleanly remove the notification, SQS queues, IAM role, and Kubernetes resources without needing to empty any buckets.

Quickstart Full Sample

main.tf - Complete Terraform configuration:

main.tf

provider "aws" {
  region = "us-west-2"
}

data "aws_eks_cluster" "cluster" {
  name = "my-production-cluster"
}

provider "kubernetes" {
  config_path    = "~/.kube/config"
  config_context = "arn:aws:eks:us-west-2:ACCOUNT:cluster/my-production-cluster"
}

provider "helm" {
  # Note: 'kubernetes' is an attribute (uses '='), not a block
  kubernetes = {
    config_path    = "~/.kube/config"
    config_context = "arn:aws:eks:us-west-2:ACCOUNT:cluster/my-production-cluster"
  }
}

# Get current AWS account ID
data "aws_caller_identity" "current" {}

locals {
  oidc_issuer = replace(data.aws_eks_cluster.cluster.identity[0].oidc[0].issuer, "https://", "")
}

module "tenx_streamer" {
  source  = "log-10x/tenx-streamer/aws"

  # Required: API key and OIDC provider
  tenx_api_key      = var.tenx_api_key
  oidc_provider_arn = "arn:aws:iam::${data.aws_caller_identity.current.account_id}:oidc-provider/${local.oidc_issuer}"
  oidc_provider     = local.oidc_issuer

  # Kubernetes
  namespace        = "log10x-streamer"
  create_namespace = true

  # Resource naming prefix
  resource_prefix = "prod-streamer"

  # S3 bucket configuration (names must be globally unique)
  tenx_streamer_index_source_bucket_name  = "prod-logs-${data.aws_caller_identity.current.account_id}"
  tenx_streamer_index_results_bucket_name = "prod-index-${data.aws_caller_identity.current.account_id}"

  # S3 trigger configuration
  tenx_streamer_index_trigger_prefix = "app/"
  tenx_streamer_index_trigger_suffix = ".log"

  # Application configuration via values file
  helm_values_file   = "${path.module}/values/streamer-production.yaml"

  # Tagging
  tags = {
    Environment = "production"
    Project     = "log10x"
    ManagedBy   = "terraform"
  }
}

variable "tenx_api_key" {
  description = "Log10x API key"
  type        = string
  sensitive   = true
}

values/streamer-production.yaml - Application configuration:

values/streamer-production.yaml

clusters:
  - name: all-in-one
    roles: ["index", "query", "stream"]
    replicaCount: 2
    maxParallelRequests: 10
    maxQueuedRequests: 1000
    readinessThresholdPercent: 90

    resources:
      requests:
        cpu: 1000m
        memory: 2Gi
      limits:
        memory: 4Gi

    autoscaling:
      enabled: true
      minReplicas: 2
      maxReplicas: 10
      targetCPUUtilizationPercentage: 70

fluentBit:
  output:
    type: s3
    config:
      s3:
        bucket: my-output-logs-bucket
        region: us-west-2

For separate clusters per role (index, query, stream), see the Separate clusters per role tab in Step 4.

Next: FAQ