Run

Test the Storage Streamer locally using minikube with either LocalStack (local AWS emulation) or real AWS services. Each step below provides tabs for both backends. For production deployment to AWS EKS, see the deployment guide.

Setup

Step 1: Prerequisites

Requirement	Description
Log10x License	Get your API key at console.log10x.com to enable analytics dashboards and full functionality.
minikube	Local Kubernetes cluster (install guide)
kubectl	Kubernetes CLI (install guide)
Helm	Kubernetes package manager (install guide)
Terraform	Infrastructure as code tool (install guide)
AWS CLI	For testing S3/SQS operations (install guide)
AWS Account	Required only for the AWS backend tab. An AWS account with permissions to create S3 buckets, SQS queues, and S3 event notifications. Not needed when using LocalStack.

Step 2: Start minikube

Start your local Kubernetes cluster:

minikube start --cpus=4 --memory=8192

Verify the cluster is running:

kubectl cluster-info

Step 3: Configure Storage Backend

LocalStack AWS

LocalStack provides local AWS service emulation (S3, SQS) running inside your Kubernetes cluster. No AWS account required.

Add the LocalStack Helm repository:

helm repo add localstack https://localstack.github.io/helm-charts
helm repo update

Create namespace and install LocalStack:

kubectl create namespace localstack

helm upgrade --install localstack localstack/localstack \
  --namespace localstack \
  --set service.type=ClusterIP \
  --set "extraEnvVars[0].name=SERVICES" \
  --set "extraEnvVars[0].value=s3\,sqs" \
  --wait --timeout 180s

Start port-forward to LocalStack (keep this running in a separate terminal):

kubectl port-forward -n localstack svc/localstack 4566:4566

Verify LocalStack is running:

curl http://localhost:4566/_localstack/health

Use real AWS S3 and SQS services. Terraform creates the infrastructure using your AWS credentials, and the pods in minikube access AWS via access keys passed as environment variables.

Set your AWS credentials so Terraform can provision resources:

export AWS_ACCESS_KEY_ID=YOUR_ACCESS_KEY
export AWS_SECRET_ACCESS_KEY=YOUR_SECRET_KEY
export AWS_DEFAULT_REGION=us-east-1

Verify access:

aws sts get-caller-identity

The same credentials are passed to the streamer pods in minikube (via extraEnv in the Helm chart) so they can reach S3 and SQS. This is appropriate for local testing — for production, use IAM Roles for Service Accounts (IRSA).

Step 4: Deploy with Terraform

LocalStack AWS

Download the Terraform configuration and deploy:

mkdir -p streamer-local && cd streamer-local
curl -sLO https://raw.githubusercontent.com/log-10x/terraform-aws-tenx-streamer/main/examples/local-localstack/main.tf
terraform init
terraform apply -var="tenx_api_key=YOUR_API_KEY"

Ensure LocalStack port-forward is running before applying (kubectl port-forward -n localstack svc/localstack 4566:4566).

Download the Terraform configuration and deploy:

mkdir -p streamer-local && cd streamer-local
curl -sLO https://raw.githubusercontent.com/log-10x/terraform-aws-tenx-streamer/main/examples/local-aws/main.tf
terraform init
terraform apply \
  -var="tenx_api_key=YOUR_API_KEY" \
  -var="aws_access_key_id=$AWS_ACCESS_KEY_ID" \
  -var="aws_secret_access_key=$AWS_SECRET_ACCESS_KEY"

AWS credentials are injected into pods so they can access real S3/SQS from minikube.

Verify deployment:

# Check pods are running (should show 2/2 READY for Fluent Bit sidecar)
kubectl get pods -n log10x-streamer

# View streamer logs
kubectl logs -n log10x-streamer -l app=streamer-10x -c streamer-10x-all-in-one --tail=50

Expected startup log output:

INFO  [com.log10x.ext.quarkus.executor.PipelineExecutor] (main) Initializing cloud accessor...
INFO  [com.log10x.ext.quarkus.access.aws.AWSAccessor] (main) AWS shared client cache initialized
INFO  [com.log10x.ext.quarkus.executor.PipelineExecutor] (main) Cloud accessor AWSAccessor initialized
INFO  [com.log10x.ext.quarkus.executor.PipelineExecutor] (main) Pipeline factory initialized
INFO  [io.quarkus] (main) run-quarkus 0.2.0 on JVM (powered by Quarkus 3.30.2) started in 1.873s. Listening on: http://0.0.0.0:8080
INFO  [io.quarkus] (main) Profile prod activated.
INFO  [io.quarkus] (main) Installed features: [amazon-sdk-sqs, cdi, rest, rest-jackson, scheduler, smallrye-context-propagation, smallrye-health, vertx]

Step 5: Test Indexing

Upload a test log file to trigger auto-indexing. Only files with .log extension trigger indexing (configured via tenx_streamer_index_trigger_suffix in Terraform).

LocalStack AWS

# Configure AWS CLI for LocalStack
export AWS_ACCESS_KEY_ID=test
export AWS_SECRET_ACCESS_KEY=test
export AWS_DEFAULT_REGION=us-east-1

# Create test log file with current timestamp
echo "{\"timestamp\":\"$(date -u +%Y-%m-%dT%H:%M:%SZ)\",\"level\":\"ERROR\",\"message\":\"Test error\"}" > test.log

# Upload to S3 (triggers auto-indexing via S3 notification)
aws --endpoint-url=http://localhost:4566 s3 cp test.log s3://streamer-logs/app/test.log

# Check index was created (within 5-15 seconds)
aws --endpoint-url=http://localhost:4566 s3 ls s3://streamer-index/indexed/ --recursive

# Create test log file with current timestamp
echo "{\"timestamp\":\"$(date -u +%Y-%m-%dT%H:%M:%SZ)\",\"level\":\"ERROR\",\"message\":\"Test error\"}" > test.log

# Upload to S3 (triggers auto-indexing via S3 notification)
aws s3 cp test.log s3://streamer-logs/app/test.log

# Check index was created (within 5-15 seconds)
aws s3 ls s3://streamer-index/indexed/ --recursive

Verify indexing completed — check the streamer logs for IndexTemplates and IndexFilterStats entries:

kubectl logs -n log10x-streamer -l app=streamer-10x -c streamer-10x-all-in-one --tail=50

Expected indexing log output:

INFO  [executor-thread-3] IndexTemplates - merged templates. templates.size: 1, rawTemplateFiles.size: 0, ...
INFO  [executor-thread-3] IndexFilterStats - index complete. Bytes: 21, filters.size: 1, probability historgram: [1=>1], key sizes: [0=>1], element counts: [50=>1], epochs: 1
INFO  [executor-thread-3] ExecutionPipeline - execution of: /etc/tenx/modules/pipelines/run/pipeline.yaml (myObjectStorageIndex) completed in: 7501ms

IndexTemplates confirms TenXTemplate files were merged and written to the index bucket. IndexFilterStats confirms Bloom filters were created for the indexed byte ranges.

Step 6: Test Querying

Start port-forward to streamer service (in a separate terminal):

kubectl port-forward -n log10x-streamer svc/streamer-streamer-10x-all-in-one 8080:80

Send a query via REST API:

# Query for ERROR events in the last 5 minutes
curl -X POST http://localhost:8080/streamer/query \
  -H "Content-Type: application/json" \
  -d '{
    "from": "now(\"-5m\")",
    "to": "now()",
    "search": "severity_level == \"ERROR\""
  }'

Query parameter formats:

from/to: Epoch milliseconds (not ISO 8601 dates). Use an epoch converter to convert human-readable dates. Multiply seconds by 1000 for milliseconds.
search: Search expression syntax supporting &&, ||, ==, and includes(). Use enriched field names like severity_level (derived automatically from level, severity, etc. — see initialize/level).

The query endpoint returns HTTP 200 to acknowledge receipt. Events are processed asynchronously and streamed to the Fluent Bit sidecar output (configured as stdout for local testing). In this single-node setup, expect results within 30-60 seconds as all worker roles share resources.

Check Fluent Bit logs for streamed events:

kubectl logs -n log10x-streamer -l app=streamer-10x -c fluent-bit --tail=50

Expected Fluent Bit output:

[0] tenx-cloud-streamer: [[<epoch>, {}], {"timestamp"=>"<ISO8601>", "level"=>"ERROR", "message"=>"Test error"}]

Step 7: Teardown

Remove all local testing resources when done.

LocalStack AWS

# Destroy Terraform resources (S3, SQS, Helm release)
cd streamer-local
terraform destroy -var="tenx_api_key=YOUR_API_KEY"

# Remove LocalStack
helm uninstall localstack -n localstack
kubectl delete namespace localstack

# Optional: Delete minikube cluster entirely
minikube delete

# Empty S3 buckets first (required before destroy)
aws s3 rm s3://streamer-logs/ --recursive
aws s3 rm s3://streamer-index/ --recursive

# Destroy Terraform resources
cd streamer-local
terraform destroy \
  -var="tenx_api_key=YOUR_API_KEY" \
  -var="aws_access_key_id=$AWS_ACCESS_KEY_ID" \
  -var="aws_secret_access_key=$AWS_SECRET_ACCESS_KEY"

# Optional: Delete minikube cluster
minikube delete

Important: S3 buckets cannot be deleted until empty. The aws s3 rm --recursive commands remove all objects before terraform destroy deletes the buckets.

Customization

Configuring Fluent Bit Output

The local test examples configure Fluent Bit to output to stdout for easy debugging. To send events to a SIEM or log analyzer, modify the fluentBit.output section in your Terraform configuration.

Edit your main.tf and update the fluentBit block inside the Helm values:

stdout Elasticsearch Splunk Datadog CloudWatch S3

Default for local testing — events print to container logs.

fluentBit = {
  output = { type = "stdout" }
}

View output: kubectl logs -n log10x-streamer -l app=streamer-10x -c fluent-bit

fluentBit = {
  output = {
    type = "elasticsearch"
    config = {
      elasticsearch = {
        host  = "elasticsearch.example.com"
        port  = 9200
        index = "logs"
        # httpUser = "elastic"        # Optional auth
        # httpPassword = "changeme"   # Use extraEnv with secretKeyRef instead
      }
    }
  }
}

fluentBit = {
  output = {
    type = "splunk"
    config = {
      splunk = {
        host  = "splunk-hec.example.com"
        port  = 8088
        token = "YOUR-HEC-TOKEN"  # Use extraEnv with secretKeyRef instead
        # index = "main"          # Optional index override
      }
    }
  }
}

fluentBit = {
  output = {
    type = "datadog"
    config = {
      datadog = {
        apiKey = "YOUR-DD-API-KEY"  # Use extraEnv with secretKeyRef instead
        # host = "http-intake.logs.datadoghq.com"  # Default US
        # host = "http-intake.logs.datadoghq.eu"   # EU region
      }
    }
  }
}

fluentBit = {
  output = {
    type = "cloudwatch"
    config = {
      cloudwatch = {
        region        = "us-east-1"
        logGroupName  = "/aws/streamer/events"
        logStreamPrefix = "stream-"
        # autoCreateGroup = true  # Create log group if missing
      }
    }
  }
}

Pods need AWS credentials to write to CloudWatch. For local testing, add AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY to extraEnv. For production EKS, use IRSA instead.

fluentBit = {
  output = {
    type = "s3"
    config = {
      s3 = {
        bucket = "my-streamed-events"
        region = "us-east-1"
        # totalFileSize = "100M"     # Rotate after 100MB
        # uploadTimeout = "1m"       # Flush interval
      }
    }
  }
}

Pods need AWS credentials to write to S3. For local testing, add AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY to extraEnv. For production EKS, use IRSA instead.

After modifying, run terraform apply to update the deployment.

Customizing Index Triggers

By default, only files with .log extension trigger indexing. To change this, modify the Terraform variables:

# In the streamer_infra module block
tenx_streamer_index_trigger_prefix = "app/"    # Only index files in app/ prefix
tenx_streamer_index_trigger_suffix = ".json"   # Only index .json files

Leave prefix empty ("") to match all paths. The suffix is required — S3 event notifications need at least one filter criterion.

Loading Local Config and Symbol Files

To test custom pipeline configs or symbol libraries locally without pushing to GitHub, you can mount a local directory into the streamer pods using a single Terraform variable. No code changes required — the example Terraform files already support this.

Step 1: Prepare your local config directory

Create a directory with the same structure as the streamer's built-in config:

my-tenx-config/
├── apps/                     # App-specific configs
├── pipelines/                # Custom pipeline YAML configs
└── data/
    └── shared/
        └── symbols/          # Symbol library files (.10x.tar)

Step 2: Mount the directory into minikube

In a separate terminal (keep it running):

Linux / macOS Windows

minikube mount ~/my-tenx-config:/mnt/tenx-config

minikube mount "C:/path/to/my-tenx-config":/mnt/tenx-config

Step 3: Deploy with the local_config_path variable

Add -var="local_config_path=/mnt/tenx-config" to your terraform apply command:

LocalStack AWS

terraform apply \
  -var="tenx_api_key=YOUR_API_KEY" \
  -var="local_config_path=/mnt/tenx-config"

terraform apply \
  -var="tenx_api_key=YOUR_API_KEY" \
  -var="aws_access_key_id=$AWS_ACCESS_KEY_ID" \
  -var="aws_secret_access_key=$AWS_SECRET_ACCESS_KEY" \
  -var="local_config_path=/mnt/tenx-config"

This mounts your local directory over the container's built-in /etc/tenx/config, replacing the default configuration. Omit the variable to use the built-in defaults.

Updating config after deployment

File changes on your host are reflected inside minikube immediately via the mount. However, the streamer pods need a restart to pick up the new files:

kubectl rollout restart deployment -n log10x-streamer -l app=streamer-10x

Configuring Metric Outputs (TSDBs)

Fluent Bit handles event streaming to log analyzers. For metric outputs to time-series databases (Prometheus, Datadog, CloudWatch), configure the metric output modules in your pipeline config.

Metric outputs aggregate events into metrics (counters, gauges, histograms) before publishing — different from streaming raw events via Fluent Bit.

Supported metric outputs:

Output	Config Reference
Prometheus Remote Write	remote-write
Prometheus Push Gateway	push-gateway
Datadog	datadog
CloudWatch	cloudwatch
Elastic	elastic
SignalFx	signalFx

To enable locally: Use the Loading Local Config approach above to mount custom pipeline configs with metric output modules enabled.

Example: Enable Prometheus Remote Write in your local pipeline config:

apps/cloud/streamer/stream/config.yaml

# Comment in the line enabling a custom remote write
- run/output/metric/prometheus/remote-write  # https://doc.log10x.com/run/output/metric/prometheus/remote-write

pipelines/run/output/metric/prometheus/remote-write/config.yaml

# Fill the endpoint host and credentials
- host: http://localhost:9090/api/v1/write # (REQUIRED)

  # 'user' for authentication
  user: my-username # (REQUIRED)

  # 'password' for authentication
  password: my-password # (REQUIRED)

See the Storage Streamer run guide for the full list of configurable modules.

Defining Queries

Queries select events from object storage by target app, time range, and search expression, then stream matching events to the Fluent Bit sidecar output.

Query Parameters

Parameter	Type	Description
`from`	Epoch ms	Start of the time range (inclusive). Use `now("-5m")` for relative time or a literal epoch like `1769076000000`.
`to`	Epoch ms	End of the time range (exclusive). Use `now()` for current time.
`search`	String	Search expression applied to storage via TenXTemplate filters for pre-filtering. Supports `&&`, `
`filters`	List	Optional JavaScript expressions for in-memory filtering after events are fetched from storage.
`processingTime`	Duration	Optional max processing time (e.g., `parseDuration("5m")`). Defaults to 1 minute.
`resultSize`	Bytes	Optional max result volume (e.g., `parseBytes("10MB")`).

Sample Queries

Query for ERROR events in the last 5 minutes

Query events by severity_level enriched from log level fields:

curl -X POST http://localhost:8080/streamer/query \
  -H "Content-Type: application/json" \
  -d '{
    "from": "now(\"-5m\")",
    "to": "now()",
    "search": "severity_level == \"ERROR\""
  }'

Query for ERROR or FATAL events containing POST

Combine severity_level with text search using includes():

curl -X POST http://localhost:8080/streamer/query \
  -H "Content-Type: application/json" \
  -d '{
    "from": "now(\"-15m\")",
    "to": "now()",
    "search": "(severity_level == \"ERROR\" || severity_level == \"FATAL\") && includes(text, \"POST\")"
  }'

Query with a fixed time range

Use epoch milliseconds for precise time ranges (multiply seconds by 1000):

# 2026-01-22T09:00:00Z = 1769072400000
# 2026-01-22T11:00:00Z = 1769079600000
curl -X POST http://localhost:8080/streamer/query \
  -H "Content-Type: application/json" \
  -d '{
    "from": "1769072400000",
    "to": "1769079600000",
    "search": "severity_level == \"ERROR\""
  }'

Query with extended processing time

Use processingTime for large time ranges or high-volume data:

curl -X POST http://localhost:8080/streamer/query \
  -H "Content-Type: application/json" \
  -d '{
    "from": "now(\"-1h\")",
    "to": "now()",
    "search": "severity_level == \"ERROR\"",
    "processingTime": "parseDuration(\"5m\")"
  }'

Query with result size limit

Use resultSize to cap the volume of returned data:

curl -X POST http://localhost:8080/streamer/query \
  -H "Content-Type: application/json" \
  -d '{
    "from": "now(\"-1h\")",
    "to": "now()",
    "search": "severity_level == \"ERROR\"",
    "resultSize": "parseBytes(\"100MB\")"
  }'

Query by country

Query events by country enriched from IP addresses via GeoIP lookup:

curl -X POST http://localhost:8080/streamer/query \
  -H "Content-Type: application/json" \
  -d '{
    "from": "now(\"-1h\")",
    "to": "now()",
    "search": "country == \"US\" && severity_level == \"ERROR\""
  }'

Query by HTTP status

Query events by http_code enriched via lookup tables:

curl -X POST http://localhost:8080/streamer/query \
  -H "Content-Type: application/json" \
  -d '{
    "from": "now(\"-1h\")",
    "to": "now()",
    "search": "http_code == \"500\" || http_code == \"503\""
  }'

Query by k8s namespace

Query events by k8s_namespace enriched from Kubernetes context:

curl -X POST http://localhost:8080/streamer/query \
  -H "Content-Type: application/json" \
  -d '{
    "from": "now(\"-30m\")",
    "to": "now()",
    "search": "k8s_namespace == \"production\" && severity_level == \"ERROR\""
  }'

The query endpoint returns HTTP 200 to acknowledge receipt. Matched events are streamed to the Fluent Bit sidecar. Check the Fluent Bit container logs for output:

kubectl logs -n log10x-streamer -l app=streamer-10x -c fluent-bit --tail=50

For the full query configuration reference, see Object Storage Query.

Troubleshooting

No events appearing in Fluent Bit logs after query

Wait 30-60 seconds — in this single-node setup, all worker roles (index, query, stream) share resources. Query processing completes once stream workers pick up results from the Stream SQS queue.
Check the time range — ensure your query's from/to range covers the timestamp of your uploaded log file. Using now("-5m") queries events from the last 5 minutes.

Verify query was accepted — check streamer logs for QueryWorker entries:

kubectl logs -n log10x-streamer -l app=streamer-10x -c streamer-10x-all-in-one --tail=100 | grep -i query

Extend processing timeout — add "processingTime": "parseDuration(\"5m\")" to your query request. See query API reference for details.

Query returns no results due to incorrect epoch timestamps or clock skew

Use the now() function for relative time ranges — it returns epoch milliseconds and accepts an offset string:

now() — current time
now("-5m") — 5 minutes ago
now("-1h") — 1 hour ago

If you must use literal epochs, ensure they're in milliseconds (not seconds). A common mistake is passing epoch seconds (e.g., 1769076000) instead of milliseconds (1769076000000). Multiply seconds by 1000.

Additionally, minikube's clock can drift from the host machine after sleep/hibernate. If the streamer's log timestamps don't match your wall clock, resync minikube's clock:

minikube ssh -- sudo chronyc makestep

If chronyc is not available, restart minikube instead: minikube stop && minikube start.

LocalStack connectivity issues (LocalStack only)

If AWS CLI commands fail, re-export the LocalStack environment variables:

export AWS_ACCESS_KEY_ID=test AWS_SECRET_ACCESS_KEY=test AWS_DEFAULT_REGION=us-east-1

Verify LocalStack health and resources:

# Verify LocalStack is healthy
curl http://localhost:4566/_localstack/health

# Check S3 buckets exist
aws --endpoint-url=http://localhost:4566 s3 ls

# Check SQS queues exist
aws --endpoint-url=http://localhost:4566 sqs list-queues

Terraform errors (AWS only)

S3 bucket name conflict:

Error: creating Amazon S3 Bucket: BucketAlreadyExists

S3 bucket names are globally unique across all AWS accounts. Change the resource_prefix variable in your main.tf to use a unique name:

variable "resource_prefix" {
  default = "streamer-myname-12345"  # Add your name/ID to ensure uniqueness
}

Insufficient permissions:

Error: creating SQS Queue: AccessDenied

Your AWS credentials need permissions for S3, SQS, and S3 event notifications. Verify your IAM user/role has the required policies:

s3:CreateBucket, s3:PutBucketNotification, s3:PutObject, s3:GetObject, s3:DeleteObject, s3:DeleteBucket
sqs:CreateQueue, sqs:GetQueueAttributes, sqs:SetQueueAttributes, sqs:DeleteQueue, sqs:SendMessage, sqs:ReceiveMessage

State lock error:

Error: Error acquiring the state lock

A previous Terraform run was interrupted. Remove the lock:

terraform force-unlock LOCK_ID

Or if using local state, delete .terraform.lock.hcl and retry.

Configuration

To configure the Storage Streamer app, Edit these settings:

Index Config

Below is the default configuration from: index/config.yaml.

Run

Setup

Customization

Defining Queries

Query Parameters

Sample Queries

Troubleshooting

Configuration

Index Config

Query Config

Stream Config

Object Storage Query

Initializers

message

level

group

httpCode

lookup

k8s

geoIP

Regulators

rate

policy

Event Forward Output

Metric Outputs

log10x

scrape

remote-write

push-gateway

datadog

cloudwatch

elastic

signalFx

log

Advanced Settings

Bootstrap

Github Sync

Symbols

Aggregators

index

stream

Template

Transform

timestamp

fields

symbol

group

parallelize