Configuration

Local Docker Compose setup for the L1ES plugin, sizing dmlSizeToSearch, and the fields parameter in Elasticsearch queries.

Docker Compose Setup for L1ES Plugin Testing

For local development and testing of the L1ES plugin, use this Docker Compose setup:

version: '3'
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.17.0
    environment:
      - discovery.type=single-node
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
      - xpack.security.enabled=false  # For dev only; enable in production
    ports:
      - "9200:9200"
    volumes:
      - ./l1es-plugin-0.3.0.es.8.17.0.zip:/tmp/l1es-plugin.zip:ro
      - es-data:/usr/share/elasticsearch/data

  setup:
    image: curlimages/curl:latest
    depends_on:
      elasticsearch:
        condition: service_started
    entrypoint: >
      sh -c "
      sleep 10 &&
      echo 'Installing L1ES plugin...' &&
      curl -X POST 'http://elasticsearch:9200/_plugins/_install?location=file:///tmp/l1es-plugin.zip' || true &&
      sleep 5 &&
      echo 'Setting up plugin indices...' &&
      curl -X POST 'http://elasticsearch:9200/_l1es/setup' &&
      echo 'Done!'
      "

volumes:
  es-data:

Usage:

docker-compose up -d

# Wait for setup to complete, then verify:
curl -s http://localhost:9200/_cat/indices?v | grep l1es

# Stop when done:
docker-compose down -v

What this sets up:

• Single-node Elasticsearch 8.17.0 with L1ES plugin
• Two internal indices: l1es_dml (templates) and l1es_dml_indices (field mappings)
• Plugin endpoints: _l1es/setup, _l1es/add-dml-index, _l1es/query, etc.
• Ready for development and testing without x-pack security

What does dmlSizeToSearch do and when do I need to increase it

dmlSizeToSearch limits how many templates the plugin scans per query. Default: 10,000.

How it works:

When you query an index with encoded events, the plugin: 1. Fetches templates from l1es_dml index (up to dmlSizeToSearch limit) 2. Matches incoming events against those templates 3. Expands matched events; leaves unmatched events as-is

What happens when you exceed the limit:

If you have 50,000 templates but dmlSizeToSearch=10000: - Plugin scans only first 10,000 templates - Events matching templates #10,001-50,000 won't expand (silent — no error) - Result: Partial expansion, some events remain encoded

When to increase it:

Monitor template count:

GET l1es_dml/_count

• Under 10,000: No action needed
• 10,000-50,000: Increase dmlSizeToSearch to 50000 in l1es.yml
• Over 50,000: Contact engineering; consider template partitioning

Trade-off: Increasing dmlSizeToSearch uses more memory and may slow queries. Recommended max: 100,000.

Query Semantics: Why do I need to include the 'fields' parameter

The fields parameter in Elasticsearch queries triggers the fetch phase, which is when L1ES expansion happens.

Without fields parameter:

GET my-logs/_search
{
  "query": { "match": { "message": "error" } }
}

Response: _id and _score only. Encoded events remain encoded (L1ES doesn't run).

With fields parameter:

GET my-logs/_search
{
  "query": { "match": { "message": "error" } },
  "fields": ["message", "timestamp", "level"]
}

Response: Specified fields are returned (AND L1ES plugin decodes encoded events during fetch phase).

Kibana automatically includes fields — if you're using Kibana, this is handled transparently. For direct API calls or custom applications, always include fields if you want expansion to happen.

Performance note: Including fields adds modest overhead (~1-2ms per event) for the fetch phase. This is where expansion happens.