Optimization
Why compact events go to S3 rather than Datadog, the cost-insight metrics 10x publishes, what the Receiver regulates, querying offloaded logs, query overhead, and how to migrate.
Optimization
Can Receiver (Compact mode) output go directly to Datadog
No loss, but not Datadog-bound either. Receiver (Compact mode) uses lossless techniques only, like Protocol Buffers for logs. Structure is stored once per event type as a template, with only changing values shipped per event; all fields and values remain intact. Compact events contain template references instead of full repetitive structure.
Why they go to S3, not Datadog: Datadog has no plugin to expand 10x's template-plus-values wire form, so compact events cannot be queried natively in Datadog. They go to S3 (and to Splunk/Elasticsearch/ClickHouse, which can expand them). Compaction reduces S3 storage, not the Datadog bill.
To cut the Datadog bill in-platform, down-tier patterns to Flex Logs instead. tier_down routes low-value patterns straight to Datadog's Flex Logs index, where they stay queryable in Datadog at a lower cost, no S3 round-trip.
To get S3 data back into Datadog: Retriever reads compact events from S3, expands them back to standard format, and streams regulated events to Datadog on-demand.
Note: if you want to filter or drop logs (lossy), that's Receiver Filter mode, explicit policies you control.
Can I see what's driving my Datadog costs, inside Datadog
Yes. 10x publishes cost insight metrics directly to Datadog's time-series API: volume and bytes per event type, tagged by symbol identity. Which log patterns drive the most volume shows up on native Datadog dashboards.
This includes regulation visibility: when Receiver caps noisy events, the metrics show what passed vs. what was filtered, so cost control is monitored without leaving Datadog.
What types of logs typically see the best optimization
High-volume repetitive logs see the best results:
- Health checks and heartbeats
- Debug statements
- Access logs
- Container orchestration events
- Verbose application traces
On Datadog, the lever for these is tier_down (route them to the cheaper Flex Logs index, still queryable) or offload to your own S3. In many environments these patterns make up a modeled 60-80% of ingestion volume.
What happens to logs regulated by Receiver
Receiver identifies low-priority logs based on your configured budget and severity thresholds. You control the destination for regulated logs:
- Offload to S3 storage at $0.023/GB. Non-destructive: the data stays reachable, and Retriever fetches it back on-demand.
- Stream to a secondary destination for further analysis. Non-destructive.
- Drop (Filter mode). This is a lossy choice you opt into: dropped events are gone, so use it only for noise you are willing to lose.
The non-destructive levers (offload, stream) keep the data; drop is the one option that loses it.
Can I query offloaded logs after they're filtered
Yes. Retriever provides on-demand fetch of the events offloaded to your own S3.
When you need specific data back for investigation or compliance, stream specific time ranges and log types directly into Datadog, paying ingestion costs only when you actually need the data.
Performance & Migration
Is there any query overhead inside Datadog
No in-Datadog expansion overhead. Events that reach Datadog, the hot path to the Standard index and the down-tiered slice in the Flex Logs index, are standard log format. There is nothing for Datadog to expand at query time, so queries run exactly as they would on any Datadog logs.
Compacted events live in S3, not Datadog. When you need them back, the Retriever expands them on the S3 read path and streams the result to Datadog, the expansion happens in the Retriever, not inside a Datadog query.
Flex Logs queries are slower than Standard-index queries (that tradeoff is part of the Flex tier itself, set by the compute size you pick), but that is Datadog's tiering, not a 10x expansion cost.
How do I migrate from non-optimized to 10x optimization
Safe migration procedure:
Phase 1: Test (Week 1)
1. Deploy the Receiver with Filter mode + tier_down on 1-2 test nodes
(route low-value patterns to a Datadog Flex Logs index)
2. Route subset of traffic (5-10%) to 10x
3. Remainder bypasses 10x (standard Datadog flow)
4. Monitor:
- Check Datadog for missing logs
- Verify the down-tiered slice lands in the Flex index
- Verify Standard-index volume drops on the down-tiered patterns
- Look for errors in sidecar logs
(Use Compact mode only on the S3 offload path, not for Datadog.)
Phase 2: Gradual rollout (Weeks 2-3)
Increase traffic percentage:
- 10% -> 25% -> 50% -> 100%
At each stage:
- Monitor sidecar CPU/memory
- Verify Datadog shows consistent log volume across Standard + Flex
- Check that the down-tier and offload split stays consistent
Phase 3: Stabilization (Week 4+)
Once at 100%:
- Monitor weekly for consistency
- Set baseline metrics (GB/day moved to Flex, GB/day offloaded, bill reduction)
- Plan next step (add S3 offload for patterns to remove from Datadog entirely)
Rollback procedure (if needed):
# If issues arise, disable 10x immediately:
# Remove the Receiver sidecar from your forwarder pod spec
# and roll out the change (or revert the Helm values)
# Datadog Agent automatically routes to original sources
# No logs are lost (sidecar doesn't modify original files)
# Timeline: < 2 minutes to full bypass
What happens to logs already optimized?
If you optimize logs for 2 weeks then disable 10x: - Already-optimized logs remain in Datadog (still queryable) - New logs flow at full volume (standard ingestion) - No data loss, just temporary cost spike until old logs age out
Testing before production:
Use the Dev app to measure your actual reduction ratio. Point $TENX_CONFIG at your config, drop your log files into its sample input directory, and run:
Can I optimize only specific services or log sources
Yes. You can apply 10x selectively:
Option 1: Sidecar to specific pods
Add the Receiver as a sidecar only to the forwarder pods whose log sources you want optimized; leave the rest untouched.
apiVersion: v1
kind: Pod
spec:
containers:
- name: app
image: myapp:latest
- name: receiver
image: log10x/pipeline-10x:latest
Option 2: Different optimization for different services
Frontend logs (static HTML, JS) → Low repetition → Skip optimization
Backend logs (microservices) → High repetition → Apply optimization
Database logs (verbose) → Very high repetition → Aggressive optimization
Option 3: Route by log pattern
If using Fluent Bit or Fluentd:
- Match app.frontend → Standard routing (no optimization)
- Match app.backend → Through 10x sidecar
- Match app.database → Through 10x sidecar
Cost impact per service:
Once you deploy, use Datadog metrics to measure savings by service: