Skip to content

Log Analyzer Input

Analyzer inputs retrieve events from log analytics backends via REST APIs to transform them into typed TenXObjects.

The Cloud Reporter app launches analyzer inputs within scheduled tasks (e.g., k8s CronJob) to read a sample amount of events (e.g., 10Mb of events in the last 5min) to identify and report on the app/infra events that incur the highest costs.

Extensibility

The Apache Camel integration framework enables connectivity and event ingestion from 400+ data sources without requiring custom coding, using YAML Routes.

For a full example of integrating a log analyzer service, see the Datadog Logs route.

Debugging

Each route has an explicit logger that can be enabled for debugging.

Logger Name App/Input Description
splunkRoute Splunk Analyzer Splunk REST API requests/responses
elasticsearchRoute Elasticsearch Analyzer Elasticsearch query requests/responses
cloudwatchLogsRoute CloudWatch Logs Analyzer AWS CloudWatch API requests/responses
datadogLogsRoute Datadog Logs Analyzer Datadog API requests/responses

To enable trace-level logging for a specific route, add the logger to your log4j2.yaml:

loggers:
  logger:
    - name: splunkRoute
      level: trace

loggers:
  logger:
    - name: elasticsearchRoute
      level: trace

loggers:
  logger:
    - name: cloudwatchLogsRoute
      level: trace

loggers:
  logger:
    - name: datadogLogsRoute
      level: trace

What Debug Logs Show

When trace logging is enabled, the route logs include:

  • Request headers: Authentication tokens, content types
  • Request body: Query parameters, search filters
  • Response headers: Rate limits, pagination info
  • Response body: Full API response data

Example Debug Session

  1. Enable the logger in $TENX_CONFIG/log4j2.yaml:

    loggers:
  logger:
    - name: splunkRoute
      level: trace

    - name: org.apache.camel
      level: info

  root:
    level: info

  2. Run the app and check the log file:

    tail -f /var/log/tenx/tenx.log | grep -i splunk

  3. Look for error patterns:

    # Authentication errors
[TRACE] splunkRoute - Response: {"messages":[{"type":"ERROR","text":"Unauthorized"}]}

# Connection errors
[ERROR] org.apache.camel - Failed to connect to splunk.example.com:8089

# Query errors
[TRACE] splunkRoute - Response: {"messages":[{"type":"FATAL","text":"Search query is malformed"}]}

Modules

  • Splunk Inputs

    Read events from a Splunk Cloud/on-premises deployment.

    More info

  • ElasticSearch Inputs

    Read events from an ElasticSearch hosted/on-premises cluster.

    More info

  • Datadog Logs Inputs

    Read events from Datadog Logs.

    More info

  • AWS Cloudwatch Logs Inputs

    Read events from AWS CloudWatch Logs.

    More info


This module is defined in analyzer/module.yaml.