Datadog

Cut Datadog costs 50-80%. Go beyond Flex Logs and Archive Search -- index your S3, stream regulated data, or aggregate into metrics and only pay for what reaches Datadog.

Compatibility

Does 10x require changes to my Datadog configuration

No. Your Datadog organization, Log Pipelines, dashboards, alerts, and saved views all continue working exactly as before.

Edge apps (Reporter, Regulator, Optimizer) deploy as sidecars alongside your log forwarders or Datadog Agent -- processing events before they reach Datadog
Cloud Reporter reads from Datadog's API (read-only, no agent changes)
Storage Streamer reads from S3 archives (no agent involvement)

What log forwarders does log10x support

All major log forwarders are supported.

Native plugin integrations:

File relay integrations:

Deployment: Log10x Edge apps deploy as sidecars alongside your existing forwarders. Integration requires minimal configuration changes to your existing setup.

How does this work with Datadog Log Pipelines

Log Pipelines continue working normally.

The 10x Engine operates at the edge before logs reach Datadog. Optimized logs maintain their original structure and are parsed, enriched, and routed by your existing pipeline rules without any modifications needed.

Compatibility: Grok parsers, remappers, category processors, and all other pipeline processors work identically with optimized logs.

Getting Started

How do I test this on my Datadog environment

Dev — Run on your Datadog log files locally. One-line install, results in minutes. No account, no credit card.
Cloud Reporter — Connect to your Datadog account via API. See which event types cost the most — no agent changes.
Edge apps — Deploy optimizer and regulator via Helm chart alongside your forwarder. ~30 min setup.
Storage Streamer — Route events to S3, stream selected data to Datadog on-demand.

Each step is independent — start with Dev to see your reduction ratio, then move to production when ready.

Can I see which log types are costing the most

Yes. Cloud Reporter analyzes your existing Datadog logs to identify which event types, services, and sources generate the most volume and cost.

This visibility helps you prioritize optimization efforts for maximum ROI. See breakdowns by service, source, log level, and custom dimensions.

What's the typical ROI timeline

Day 1 (15 min): Deploy Cloud Reporter -- agentless cost analysis via Datadog API (read-only, no agent changes). Deploy via Helm CronJob or Docker. First cost report visible same day.
Week 1-2 (30 min per app): Deploy Edge Optimizer and Edge Regulator as sidecars alongside your forwarders via Helm. They begin processing immediately.
Week 3-4: Measure sustained reduction in Datadog ingestion metrics. Typical: 30-60% volume reduction (50%+ from Optimizer, additional 10-30% from Regulator).

Start first: Use Dev locally on your actual logs to validate savings potential before production deployment.

Is the deployment reversible? Can I roll back

Yes, 100% reversible. Rolling back takes minutes:

Configuration change: Point your Datadog Agent back to the original log folder (or remove the Fluent Bit sidecar)
No data migration: Your Datadog organization is unchanged -- same indexes, same pipelines
No downtime: Switch traffic with a config reload (no restart required)

Why it's safe: Log10x doesn't modify your Datadog organization settings, log pipelines, or dashboards. Edge apps process events before they reach Datadog; Cloud apps read via API or S3 (read-only).

We recommend keeping bypass routing configured for instant rollback during initial deployment.

Datadog Agent Integration

How do I integrate Log10x with Datadog Agent

Log10x operates before logs reach Datadog — as a sidecar or preprocessing step. There's no Datadog agent plugin; instead, log10x processes logs via standard forwarders.

Two Integration Paths:

Path 1: Fluent Bit / Fluentd sidecar (Recommended)

Log10x sits between your application and Datadog Agent:

Application logs → Fluent Bit (with log10x sidecar) → Datadog Agent → Datadog

Configuration (Fluent Bit output to Datadog):

[OUTPUT]
Name datadog
Match *
apikey ${DD_API_KEY}
provider gcp  # or aws, azure
json_date_key timestamp
dd_service myapp
dd_source myapp_logs

Path 2: Datadog Agent file input

Log10x writes optimized logs to a file; Datadog Agent reads from it:

logs:
  - type: file
    path: /var/log/app/optimized/*.log
    service: myapp
    source: myapp_logs
    # No special encoding config needed — logs are already optimized by log10x

Cost Impact:

Without log10x: 100 TB/month × $2.50/GB = $250,000/month
With log10x (70% reduction): 30 TB/month × $2.50/GB = $75,000/month
Monthly savings: $175,000 ($2.1M annually)

Actual savings depend on log verbosity. Use Dev app to measure your specific reduction ratio before production deployment.

What is the performance overhead of Log10x with Datadog Agent

Log10x processes at sub-millisecond per event on a single CPU core (2GB heap). For typical workloads:

CPU Impact:

10 GB/day: ~0.1 CPU cores
100 GB/day: ~1 CPU core
1 TB/day: ~10 CPU cores

Agent interaction: Log10x writes to a file or socket; Datadog Agent reads from it asynchronously. No blocking.

Datadog Agent overhead: Unchanged. The agent ingests fewer bytes (optimized logs are 50-80% smaller), potentially reducing agent CPU.

Can I use Log10x with Datadog's log sampling

Yes, they're complementary.

Datadog log sampling: Randomly drops 90% of logs to reduce ingestion volume (lossy)
Log10x optimization: Reduces repetitive structure without dropping any events (lossless)
Together: Optimize first (50-70% lossless), then sample if needed (additional lossy reduction)

Recommended: Start with Log10x optimization only. Re-evaluate log sampling after measuring actual reduction.

Architecture

How do the Log10x products work together for Datadog

Four data paths, deployed incrementally:

Hot path — Edge Regulator → Datadog (10-30% of volume): ERRORs, security alerts, SLO signals, and events powering real-time monitors go directly to Datadog in their original format. Edge Regulator passes critical events and caps the rest with severity-aware rate limits.
Warm path — Edge Optimizer → S3 (100% of volume): The complete dataset goes to S3, losslessly compact 50%+ by Edge Optimizer. S3 stores everything at $0.023/GB — this is the default destination for all log data.
On-demand — Storage Streamer → Datadog or Metrics API: When you need historical data, Storage Streamer queries S3 indexes and either streams regulated events to Datadog or aggregates them into metric data points published to Datadog's time-series API.
Lambda — 10x Lambda → Datadog + S3 (dual-write): For CloudWatch-origin logs (Lambda, ECS, EKS), the 10x Lambda replaces the Datadog Forwarder. Regulated events go to Datadog; the full optimized stream goes to S3 via Kinesis Firehose.

S3 is the default destination for all log data. Datadog receives only the events that justify $2.50/GB indexing. See Storage Streamer use cases for detailed workflows: incident investigation, dashboard population, compliance queries, and metric aggregation.

What is the processing overhead

Edge Regulator sends events to Datadog in standard log format — no expansion needed, no search-time overhead. For events routed to S3 via Edge Optimizer, Storage Streamer expands and streams to Datadog on-demand.

The 10x Engine processes events at sub-millisecond per event — 100+ GB/day on a single node (512 MB heap, 2 threads). Memory is capped via -Xmx, CPU via threadPoolSize. For resource requirements, scaling tables, and architecture details, see Performance FAQ.

What log data should I send to Datadog vs S3

Datadog (via Edge Regulator): Events that need real-time indexing — ERROR-level events, security alerts, SLO signals, and data powering active monitors and dashboards. Typically 10-30% of total volume.

S3 (via Edge Optimizer): Everything. Your complete dataset at $0.023/GB/month. When you need specific data in Datadog, Storage Streamer streams it on-demand — paying Datadog ingestion only on what you query.

S3 is the default destination. Datadog is for events that justify $2.50/GB real-time indexing. Most log data does not need to be in Datadog at all times — it needs to be available when you need it.

Optimization

How does optimization affect log data

No loss. Edge Optimizer uses lossless techniques only — like Protocol Buffers for logs.

Structure is stored once per event type as a template. Only changing values are shipped per event. All fields and values remain intact and fully queryable in Datadog.

Note: If you want to filter or drop logs (lossy), that's Edge Regulator -- a separate product with explicit policies you control.

Can Edge Optimizer output go directly to Datadog

No. Edge Optimizer output is compacted (like Protocol Buffers) — structure is stored once per event type as a template, with only changing values shipped per event. Datadog can parse the data, but compact events contain template hash references instead of full repetitive structure.

For Datadog-bound logs, use Edge Regulator. Regulator output is standard log format that Datadog ingests directly through its Logs API.

To get optimized data into Datadog: Storage Streamer reads compact events from S3, expands them back to standard format, and streams regulated events to Datadog on-demand.

Can I see what's driving my Datadog costs -- inside Datadog

Yes. Log10x publishes cost insight metrics directly to Datadog's time-series API -- volume and bytes per event type, tagged by symbol identity. You see exactly which log patterns are consuming your ingestion budget on native Datadog dashboards.

This includes regulation visibility: when Edge Regulator caps noisy events, the metrics show what passed vs. what was filtered -- so you monitor cost control without leaving Datadog.

What types of logs typically see the best optimization

High-volume repetitive logs see the best results:

Health checks and heartbeats
Debug statements
Access logs
Container orchestration events
Verbose application traces

These often represent 60-80% of ingestion volume while providing minimal analytical value.

What happens to logs regulated by Edge Regulator

Edge Regulator identifies low-priority logs based on your configured budget and severity thresholds. You control the destination for regulated logs:

Archive to S3 storage for compliance retention at $0.023/GB
Stream to a secondary destination for further analysis
Discard true noise (health checks, debug spam)

Storage Streamer lets you query archived logs on-demand when needed.

Can I query archived logs after they're filtered

Yes. Storage Streamer provides on-demand access to logs archived in S3.

When you need historical data for investigation or compliance, stream specific time ranges and log types directly into Datadog -- paying ingestion costs only when you actually need the data.

AWS Lambda

Can I optimize Datadog logs from AWS Lambda functions

Yes. Deploy the 10x Engine as a Lambda container image that subscribes to CloudWatch Log Groups via subscription filter -- a drop-in replacement for the Datadog Forwarder Lambda.

How it works: CloudWatch delivers batched events to the 10x Lambda, which processes them through the same edge apps (Reporter, Regulator, Optimizer). Regulated events ship directly to the Datadog Logs API. The full optimized stream also writes to S3 via Kinesis Firehose — a dual-write that ensures complete retention for on-demand querying via Storage Streamer.

Cost example: 1TB/month at $2.50/GB = $2,500. With Edge Regulator capping noisy events to Datadog and Edge Optimizer compacting the full stream to S3 (50%+ reduction) -- total savings of 60-80%.

Deployment guide

Does the Lambda approach work with all Datadog regions

Yes. Set the DD_SITE environment variable to your Datadog site (e.g. datadoghq.eu, us5.datadoghq.com). The handler ships logs to the corresponding regional Datadog intake endpoint.

Performance & Migration

What is the search/query latency impact of Log10x optimization

Query latency overhead: ~1-2% for optimized events

When you query Log10x-optimized events in Datadog, the optimization is transparent:

Baseline query: 200ms (on non-optimized logs)

Query on optimized logs: 200ms + 2-4ms overhead (decoding) = 202-204ms

Decoding breakdown: - Fetch compact events: 1ms - Inflate templates: 1-2ms - Return decoded results: 0.5-1ms

For dashboards and alerts: Overhead is negligible (< 1% slowdown). Datadog caches query results, so repeated queries on the same time range are instantaneous.

For large-scale incident response: Reduced ingestion volume means faster full-text searches across a smaller dataset. The latency savings from smaller queries often offset the decoding overhead.

Performance by query type:

Query Type	Non-Optimized	Optimized	Overhead
Point query (single event)	10ms	12ms	~20%
Time-range aggregate (1h, 1000 events)	50ms	52ms	~4%
Full-text search (1d, 10K events)	200ms	210ms	~5%
Dashboard (10 queries, cached)	100ms	100ms	0% (cached)

Bottom line: Query latency impact is minimal and often outweighed by the benefit of searching a smaller dataset.

How do I migrate from non-optimized to Log10x optimization

Safe migration procedure:

Phase 1: Test (Week 1)

1. Deploy Edge Optimizer sidecar on 1-2 test nodes
2. Route subset of traffic (5-10%) to Log10x
3. Remainder bypasses Log10x (standard Datadog flow)
4. Monitor for 24 hours:
   - Check Datadog for missing logs
   - Verify optimization ratio (target: 50%+)
   - Look for errors in sidecar logs

Phase 2: Gradual rollout (Weeks 2-3)

Increase traffic percentage:
- Day 1-2: 10%
- Day 3-4: 25%
- Day 5-6: 50%
- Day 7+: 100%

At each stage:
- Monitor sidecar CPU/memory (should be < 5% additional)
- Verify Datadog shows consistent log volume
- Check that optimization ratio stays consistent

Phase 3: Stabilization (Week 4+)

Once at 100%:
- Monitor weekly for consistency
- Set baseline metrics (GB/day reduction, cost savings)
- Plan next step (add Regulator for additional cost savings)

Rollback procedure (if needed):

# If issues arise, disable Log10x immediately:
kubectl delete daemonset log10x-optimizer  # Or equivalent for your platform

# Datadog Agent automatically routes to original sources
# No logs are lost (sidecar doesn't modify original files)
# Timeline: < 2 minutes to full bypass

What happens to logs already optimized?

If you optimize logs for 2 weeks then disable Log10x: - Already-optimized logs remain in Datadog (still queryable) - New logs flow at full volume (standard ingestion) - No data loss, just temporary cost spike until old logs age out

Testing before production:

Use the Dev app to measure your actual reduction ratio:

log10x dev --file /path/to/sample.log
# Output shows: current_size → optimized_size (reduction %)
# Use this to estimate actual cost savings before deploying

Can I optimize only specific services or log sources

Yes. You can apply Log10x selectively:

Option 1: Sidecar to specific pods

apiVersion: v1
kind: Pod
metadata:
  annotations:
    log10x/optimize: "true"  # Only pods with this annotation get sidecar
spec:
  containers:
  - name: app
    image: myapp:latest
  - name: log10x-optimizer
    image: log10x/optimizer:latest

Option 2: Different optimization for different services

Frontend logs (static HTML, JS) → Low repetition → Skip optimization
Backend logs (microservices) → High repetition → Apply optimization
Database logs (verbose) → Very high repetition → Aggressive optimization

Option 3: Route by log pattern

If using Fluent Bit or Fluentd:
- Match app.frontend → Standard routing (no optimization)
- Match app.backend → Through Log10x sidecar
- Match app.database → Through Log10x sidecar

Cost impact per service:

Once you deploy, use Datadog metrics to measure savings by service:

Datadog Dashboard:
| Logs → Summary → Group by service
| Look for: Ingested bytes/day per service
| Should see 50%+ reduction on high-volume services

Comparisons

How does Log10x compare to Datadog Flex Logs

The difference: Flex Logs makes it cheaper to store data you've already sent to Datadog. Storage Streamer means you never send it in the first place -- you stream specific time ranges and log types from S3 on-demand, paying ingestion costs only when you actually need the data.

Flex Logs: the $0.05 headline vs reality

$2.50/GB ingestion + 30-day retention -- mandatory for all data, regardless of storage tier
$0.05/million events -- storage only (the headline number)
$0.60/million events -- Flex Starter with compute bundled (12x the headline)
Compute for S/M/L tiers not publicly priced -- requires sales call to learn query costs
No monitors or alerting on Flex data -- must rehydrate to Standard tier (additional cost)

Log10x Storage Streamer: query S3, stream only what you need

Store everything in your S3 at $0.023/GB -- standard AWS pricing, your bucket, full portability
Query in-place via Bloom filter indexes -- locate events by app, timeframe, keywords without full scans
Stream regulated data to Datadog -- cost-aware sampling keeps all ERRORs, throttles DEBUG, enforces per-hour budgets. You control exactly what Datadog ingests
Or aggregate into metrics -- convert millions of S3 events into metric data points published to Datadog's time-series API. Dashboard visibility at metric pricing, not log pricing

Datadog pricing as of Jan 2026

How does Log10x compare to Datadog's Log Rehydration from archives

Rehydration charges you ingestion a second time. You pay $2.50/GB to re-index data you already paid to ingest -- and only data that was originally sent. Anything you filtered is gone. Storage Streamer keeps 100% in your S3 and queries in-place.

	Rehydration	Storage Streamer
Cost	$2.50/GB re-indexed	$0.023/GB stored in your S3
Data scope	Only logs that were already ingested	100% of original logs in S3
Query	Full time-range re-ingestion	Indexes locate events by app, timeframe, keywords
Output	Re-indexed into Datadog	Stream regulated events or aggregate into metrics
Control	All-or-nothing on time range	Cost-aware sampling per event type

Datadog pricing as of Jan 2026

Troubleshooting

Log10x optimized logs aren't appearing in Datadog

Check these in order:

Verify sidecar is running:

kubectl logs -l app=log10x-optimizer
# Or: docker logs <container-id>
# Or: systemctl status log10x

Check that logs are being optimized:

curl -s http://localhost:8080/health | jq .
# Should show events_processed > 0

Verify Datadog Agent is reading from log10x output:

Check: /var/log/datadog/agent.log
Look for: "tail_file.*optimized" or similar

Check Datadog ingestion: - Go to Datadog → Logs → Live Tail - Filter: source:myapp_logs (or your service) - If empty: Agent isn't forwarding
Verify API key: - echo $DD_API_KEY — should not be empty - Check Datadog org: Organization Settings → API Keys → Verify key is active

Common causes:

Problem	Solution
Agent can't read from log10x socket/file	Check file permissions, socket path in config
Datadog API key invalid	Verify key in Datadog UI, rotate if needed
Log pipeline filters out logs	Check Log Pipelines — verify no filters are blocking your service
Network/firewall blocking Datadog intake	Test: `curl -v https://http-intake.logs.datadoghq.com`

My Datadog cost didn't go down after deploying Log10x

Check these:

Verify logs are actually being optimized:

# Check Agent logs for encoding stats:
grep -i "optimized\|encoded\|reduced" /var/log/datadog/agent.log
# Should show non-zero events processed

Check Datadog ingestion metrics: - Datadog → Dashboards → Usage Metrics - Look for: Log ingestion volume (GB/day) - Should be noticeably lower than before deployment

Measure reduction ratio:

# Use Dev app to verify your specific logs:
log10x dev --file /path/to/sample.log
# Should show reduction% (typically 50-80%)

Check if logs are reaching Datadog: - Are you seeing expected volume in Datadog Logs? - Run: index:* query — how many events indexed in last hour? - Expected: Similar count as before, but ingested volume should be lower

Why costs might not drop:

New services added: If you added instrumentation or new apps, increased volume can offset optimization
Sampling interfering: If you're also using Datadog log sampling, it may drop logs before optimization
Buffering/lag: Can take 24-48 hours for usage metrics to settle
Error logs not optimized: If most growth is from new errors/alerts, optimization helps less

Solution: Measure your current reduction ratio with Dev app on real logs. If the ratio is good (50%+), give it 48 hours for metrics to settle. If still no improvement, contact support.

Datadog Agent is crashing after deploying log10x sidecar

Check Agent logs:

tail -f /var/log/datadog/agent.log | grep -i error

Common causes & fixes:

Error	Cause	Fix
`permission denied` reading log file	Log10x output file has wrong permissions	`chmod 644 /path/to/log`
`connection refused` to log10x socket	Socket path doesn't exist or log10x not running	Verify sidecar is running, check path config
`out of memory`	Too many logs buffered	Reduce `--buffer-size` flag in log10x
`json parsing error`	Corrupted log from log10x	Check log10x health: `curl localhost:8080/health`

If Agent still crashes:

Temporarily disable log10x (remove sidecar) to isolate
Restart Agent: systemctl restart datadog-agent
Verify Agent is healthy: datadog-agent status
Re-enable log10x and check logs

Decision Guide

Should I use Log10x + Datadog or just Datadog sampling

Datadog Log Sampling: Randomly drops 90% of logs (lossy, saves cost)

Log10x Optimizer: Reduces repetitive structure 50-80% (lossless, same data, less storage)

Factor	Log Sampling	Log10x	Recommendation
Data fidelity	90% lost	100% preserved	Log10x
Troubleshooting	Limited (missing data)	Full visibility	Log10x
Search latency	Same	+1-2%	Either
Cost reduction	10x data = same cost	50-80% reduction	Log10x
Ease of setup	Native to Datadog	Requires sidecar	Datadog sampling
ROI break-even	Never (you pay same)	30-60 TB/month	Log10x

Choose Log10x if: Your org has >30 TB/month of logs and can't accept lossy sampling.

Choose Sampling if: You have <10 TB/month and are OK discarding 90% for cost savings.

Choose Both if: You want maximum savings — optimize first (lossless), then sample if needed.

How does Log10x differ from Datadog Observability Pipelines

OP requires your team to write and maintain VRL routing rules for every log format your applications produce.

Log10x learns your log formats automatically from source code -- no rules to write, no regex to maintain. Powers three cost reduction apps:

Edge Optimizer: 50-80% volume reduction while keeping 100% of your data intact
Edge Regulator: Cap 80%+ of noisy events with rate policies you set once
Storage Streamer: Query S3 archives on-demand, stream only what you need into Datadog at a fraction of ingestion cost

How does Log10x compare to other tools

See What makes 10x different in the main FAQ for detailed comparisons with Cribl, OTel Collector, Vector, Tero, and log analytics platforms (Splunk, Datadog, Elastic).

What about Datadog CloudPrem

CloudPrem (announced DASH 2025) lets customers run Datadog's processing on their own infrastructure. It solves where data is processed -- but not how much you process.

Log10x solves the cost side:

Edge Optimizer: Reduces ingestion volume 50%+ before data reaches Datadog
Storage Streamer: Stores 100% in your S3, streams only what you query

These work regardless of whether Datadog runs in their cloud or yours.

CloudPrem + Log10x is complementary: CloudPrem for data sovereignty, Log10x for cost reduction. One addresses compliance, the other addresses your bill.