Skip to content

Integration

How 10x sits in front of the Datadog Agent, how the products fit together across the Standard index, Flex Logs, S3, and metrics, and the AWS Lambda path for CloudWatch-origin logs.

Datadog Agent Integration

How do I integrate 10x with Datadog Agent

10x operates before logs reach Datadog, as a sidecar or preprocessing step. There's no Datadog agent plugin; instead, 10x processes logs via standard forwarders.

Two Integration Paths:

Path 1: Fluent Bit / Fluentd sidecar (Recommended)

10x sits between your application and Datadog Agent:

Application logs → Fluent Bit (with log10x sidecar) → Datadog Agent → Datadog

Configuration (Fluent Bit output to Datadog):

[OUTPUT]
Name datadog
Match *
apikey ${DD_API_KEY}
json_date_key timestamp
dd_service myapp
dd_source myapp_logs

Path 2: Datadog Agent file input

10x writes optimized logs to a file; Datadog Agent reads from it:

logs:
  - type: file
    path: /var/log/app/optimized/*.log
    service: myapp
    source: myapp_logs
    # No special encoding config needed, logs are already optimized by 10x

Cost Impact:

Datadog meters log ingestion per GB, and separately meters indexing/retention per million events. tier_down moves low-value patterns into a Flex Logs index (cheaper, still queryable), cutting the index cost on the down-tiered slice; offload removes patterns from Datadog entirely. Savings depend on how much of your volume is low-value pattern noise.

Use Dev app to measure your specific reduction ratio before production deployment.

What is the performance overhead of 10x with Datadog Agent

The Receiver processes events at the edge before they reach Datadog, with resource use scaling with throughput. Memory is capped via -Xmx, CPU via threadPoolSize. For measured resource requirements, scaling tables, and architecture details, see Optimization.

Agent interaction: the Receiver writes to a file or socket; Datadog Agent reads from it asynchronously. No blocking.

Datadog Agent overhead: the agent ingests fewer bytes, since tier_down and offload remove low-value patterns from the Standard-index path, potentially reducing agent CPU.

No search-time overhead: events sent to Datadog (the hot path and the Flex tier) are standard log format. Compacted events live in S3, where the Retriever expands them, so there is no in-Datadog expansion cost.

Can I use 10x with Datadog's log sampling

Yes, they're complementary.

  • Datadog exclusion filters: query- and facet-based rules where you choose which logs to exclude from an index, at a sampling rate you configure from 0 to 100%. Excluded logs are still ingested and reduce indexing, not ingestion (lossy on the excluded slice for that index).
  • 10x: down-tier and offload are non-destructive levers. tier_down moves noisy patterns into the cheaper Flex Logs index (still queryable in Datadog); offload sends the rest to your own S3, where the Retriever fetches them back on demand. Both keep the data reachable. Filter mode (sample/drop) is a separate, lossy choice you opt into.
  • Together: down-tier and offload first, then add Datadog exclusion filters only if you still need to cut what gets indexed further.

Recommended: start with 10x. Re-evaluate exclusion filters after measuring actual reduction.

Architecture

How do the 10x products work together for Datadog

Data paths, deployed incrementally:

  1. Hot path, Receiver → Datadog Standard index (modeled 10-30% of volume, illustrative): ERRORs, security alerts, SLO signals, and events powering real-time monitors go directly to Datadog in their original format. Receiver passes critical events and caps the rest with severity-aware rate limits.

  2. Down-tier path, Receiver marks low-value patterns (routeState) → Datadog Flex Logs index: the primary in-Datadog cost lever. Receiver decides per pattern which logs are safe to down-tier and routes them into a Flex Logs index, cutting the dominant index cost on that slice while the logs stay queryable in Datadog's own Log Explorer with no rehydration.

  3. Offload path, Receiver → S3 (patterns removed from Datadog entirely): patterns you want out of Datadog go to your own S3 (compacted losslessly for storage). When you need them, Retriever expands and streams them back on-demand, paying Datadog ingestion only on what you fetch.

  4. Metrics, high-volume repetitive patterns → Datadog metrics: Retriever aggregates high-volume repetitive S3 events into metric data points published to Datadog's time-series API. Dashboard visibility at metric pricing, not log pricing.

  5. Lambda path, CloudWatch-origin logs (Lambda, ECS, EKS): the 10x Lambda replaces the Datadog Forwarder, processing events before they reach Datadog. Filtered and down-tiered events ship to the Datadog Logs API; to take patterns out of Datadog, the Receiver offload lever writes them to your own S3.

Datadog's Standard index receives only the events that justify it. See Retriever use cases for detailed workflows: incident investigation, dashboard population, compliance queries, and metric aggregation.

What log data should I send to Datadog vs S3

Datadog Standard index (via Receiver): events that need real-time indexing and monitoring, ERROR-level events, security alerts, SLO signals, and data powering active monitors and dashboards. Typically 10-30% of total volume.

Datadog Flex Logs (via tier_down): low-value patterns that still need to be searchable in Datadog but don't justify the Standard index. tier_down routes them to the cheaper Flex tier; they stay queryable in Log Explorer with no rehydration.

Your own S3 (via offload): patterns you want out of Datadog entirely, stored in your bucket. When you need specific data back in Datadog, Retriever streams it on-demand, paying Datadog ingestion only on what you fetch.

Most log data does not need to be in the Standard index at all times, it needs to be either in the cheaper Flex tier or available from S3 when you need it.

AWS Lambda

Can I optimize Datadog logs from AWS Lambda functions

Yes. Deploy the 10x Engine as a Lambda container image that subscribes to CloudWatch Log Groups via subscription filter, a drop-in replacement for the Datadog Forwarder Lambda.

How it works: CloudWatch delivers batched events to the 10x Lambda, which processes them through the same edge apps (Reporter, Receiver). Filtered and down-tiered events ship to the Datadog Logs API via the engine's Fluent Bit output. To take patterns out of Datadog, the Receiver offload lever writes them to your own S3, where the Retriever fetches them back on-demand.

Cost levers: Receiver caps noisy events with Filter mode, tier_down routes low-value patterns to a Datadog Flex Logs index (cheaper, still queryable), and offload sends the rest to your own S3. The compacted stream in S3 reduces storage there, not the Datadog bill.

Deployment guide