Skip to content

Troubleshooting

Common issues after putting 10x in front of Datadog: optimized logs not appearing, the bill not dropping, and Datadog Agent problems.

Troubleshooting

10x optimized logs aren't appearing in Datadog

Check these in order:

  1. The 10x sidecar is running. Check the sidecar container's logs: 

    kubectl logs <forwarder-pod> -c <10x-sidecar>   # the 10x sidecar container
# Or: docker logs <container-id>
# Or: systemctl status log10x

  2. The pipeline is healthy. Run log10x_doctor from your MCP client (or tenx @apps/dev locally) to confirm the engine is processing events and emitting metrics.

  3. The forwarder/Agent is reading 10x output. Confirm the Datadog Agent (or Fluent Bit) input points at the file or socket the Receiver writes to, and that the path in your config matches.

  4. Datadog is ingesting. Datadog → Logs → Live Tail, filter on your source / service. If it's empty, the forwarder isn't shipping.

  5. The API key is valid. echo $DD_API_KEY is non-empty, and the key is active under Organization Settings → API Keys.

Common causes:

Problem Solution
Agent can't read the 10x socket/file Check file permissions and the socket/file path in your config
Datadog API key invalid Verify the key in the Datadog UI, rotate if needed
Log pipeline filters out logs Check Log Pipelines for filters blocking your service
Network/firewall blocking Datadog intake Test curl -v https://http-intake.logs.datadoghq.com
My Datadog cost didn't go down after deploying 10x

Check these:

  1. The down-tier and offload split is happening. Run log10x_commitment_report (or query your TSDB) to confirm patterns are being routed to Flex and offloaded, with non-zero volume.

  2. Datadog usage metrics. Datadog → Dashboards → Usage Metrics. The down-tiered slice should now show up under the Flex index, not Standard.

  3. The Flex index is enabled and ordered first. Datadog indexes are first-match-wins, so the Flex index must sit ahead of your catch-all index, or the down-tiered events never reach it.

  4. Offloaded patterns reach S3, not Datadog. Confirm the offload bucket is filling.

Why costs might not drop:

  • New services added: if you added instrumentation or new apps, increased volume can offset the reduction
  • Flex index not enabled or mis-ordered: if Flex Logs isn't enabled, or the Flex index sits after the catch-all, the down-tiered slice stays in the Standard index
  • Metric lag: allow a day or two for usage metrics to settle
  • Error logs not down-tiered: if most growth is from new errors/alerts, those stay in the Standard index by design

Solution: confirm the down-tier and offload split with the Dev app on real logs and check the Flex index is enabled and ordered first. If still no change, contact support.

Datadog Agent is crashing after deploying the 10x sidecar

Check Agent logs:

tail -f /var/log/datadog/agent.log | grep -i error

Common causes & fixes:

Error Cause Fix
permission denied reading log file 10x output file has wrong permissions chmod 644 /path/to/log
connection refused to 10x socket Socket path doesn't exist or 10x isn't running Verify the sidecar is running, check the path in config
out of memory Heap pressure under load Raise the Receiver heap (-Xmx) in your pod resources
Agent reads non-standard lines Agent is tailing the compact S3-bound output, not the Datadog-bound output Point the Agent at the Receiver's Datadog-bound output (standard log format); the compact stream is for S3 only

If the Agent still crashes:

  1. Temporarily disable 10x (remove the sidecar) to isolate.
  2. Restart the Agent: systemctl restart datadog-agent.
  3. Verify the Agent is healthy: datadog-agent status.
  4. Re-enable 10x and check logs.