Compiler

The compiler app produces symbol library files that enable the edge/cloud apps to transform input events into well-defined TenXObjects.

The compiler scans GitHub repos and container images to find symbol values (e.g., log statements, class names, enums), capturing their the source file and context within it.

The linker packages this information into symbol library files, similar to package artifacts (e.g., npm, .jar, .NET assembly).

Compile custom symbols if:

• Using 3rd party frameworks not already in the default library
• Want symbols from your specific application code
• Using proprietary or internal logging frameworks

Benefit

10x apps utilize symbol library files at runtime to assign a shared TenXTemplate (i.e., Hidden class) to incoming log events, creating a shared schema structure for each event type.

Operating on well-defined objects enables direct access to symbol and variable values, eliminating the overhead of parsing repetitive JSON structures and evaluating brittle, computationally expensive regex patterns for each event.

Out-of-the-Box Ready

The 10x runtime includes a built-in symbol library with industry-standard frameworks (Kubernetes, OpenTelemetry, Spring Boot, Django, and 30+ others). You can immediately test the Dev app with OpenTelemetry sample files or try it with your own logs for out-of-the-box optimization, even without custom symbol compilation.

Workflow

The Compiler app processes source code and binaries to generate symbol libraries that enable intelligent log processing and optimization.

graph LR
    A["<div style='font-size: 14px;'>🔄 CI/CD</div><div style='font-size: 10px; text-align: center;'>Trigger Process</div>"] --> B["<div style='font-size: 14px;'>📥 Pull</div><div style='font-size: 10px; text-align: center;'>Fetch Sources</div>"]
    B --> C["<div style='font-size: 14px;'>🔍 Scan</div><div style='font-size: 10px; text-align: center;'>Extract Symbols</div>"]
    C --> D["<div style='font-size: 14px;'>🔗 Link</div><div style='font-size: 10px; text-align: center;'>Build Library</div>"]
    D --> E["<div style='font-size: 14px;'>📤 Push</div><div style='font-size: 10px; text-align: center;'>Commit Output</div>"]
    E --> F["<div style='font-size: 14px;'>📦 Distribute</div><div style='font-size: 10px; text-align: center;'>to Runtime Apps</div>"]

    classDef cicd fill:#7c3aed88,stroke:#6d28d9,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef pull fill:#2563eb88,stroke:#1d4ed8,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef scan fill:#059669,stroke:#047857,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef link fill:#ea580c88,stroke:#c2410c,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef push fill:#16a34a88,stroke:#15803d,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef distribute fill:#f59e0b,stroke:#d97706,color:#ffffff,stroke-width:2px,rx:8,ry:8

    class A cicd
    class B pull
    class C scan
    class D link
    class E push
    class F distribute

🔄 Invoke: runs locally for development, periodically as k8s CronJob for continuous updates, or integrated as part of CI/CD pipelines (GitHub Actions) for automated symbol generation

📥 Pull: Fetches source code from GitHub, Docker, Helm, and existing symbol files for reuse

🔍 Scan: Extracts symbol values from source code and binaries across multiple programming languages to create symbol unit files

🔗 Link: Combines symbol unit files into a single symbol library file for runtime use by edge and cloud apps

📤 Push: Commits symbol library files to GitHub for distribution to runtime apps and reuse in future compilation runs

📦 Distribute: Edge and cloud apps pull symbol libraries via the @github launch macro at startup and poll the remote branch for changes (e.g., every 5 minutes). Between compilation runs, events are processed using the previous symbol library version — slightly less efficient for new log patterns, but no data is lost. The next run picks up the changes automatically.

Architecture

Compile Phase Runtime Phase

The compiler executes periodically (k8s CronJob) or as a CI/CD pipeline (GitHub Actions) step to pull modified repos, process changes, and push updated symbol files to GitHub.

⚙️ Pull source code and binary repos to generate symbol library files

10x apps running at the edge (alongside Fluentd/Fluent Bit) or in the cloud (k8s pods, serverless functions) pull symbol libraries when they start up or when libraries are updated.

▶️ 10x apps can pull symbol libraries for structured event processing

See the Compiler FAQ for complete details on symbol discovery, compilation workflows, and troubleshooting.

This app is defined in compiler/app.yaml.

Next: Test