Delete Index Objects

Deletes TenXTemplate Filters when a storage object (e.g., S3) is removed. Runs as a deletion trigger to clean up associated index objects.

Config Files

To configure the Delete index objects module, Edit these files.

Below is the default configuration from: delete/config.yaml.

Edit Online

# 🔟❎ 'run' Object storage delete index objects configuration

# Configure query AWS S3 bucket settings
# To learn more see https://doc.log10x.com/run/input/objectStorage/index/delete

# Set the 10x pipeline to 'run'
tenx: run

# =============================== Dependencies ================================

include: run/modules/input/objectStorage/index/delete

# ============================= AWS S3 Query Options ==========================

indexDelete:

  # 'objectStorageName' selects the storage accessor alias registered by 'objectStorageClassName' (AWS or Azure).
  #  Set TENX_OBJECT_STORAGE_NAME to Azure to target Azure Blob Storage. Defaults to AWS.
  #  To learn more about configuring Object storage, see: https://doc.log10x.com/run/input/objectStorage

  # AWS Authentication is made via the default provider credentials chain:
  # https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/auth/credentials/DefaultCredentialsProvider.html

  # AWS Region selection is made via the default region provider chain:
  # https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/regions/providers/DefaultAwsRegionProviderChain.html

  - objectStorageName: $=TenXEnv.get("TENX_OBJECT_STORAGE_NAME", "AWS")

    # 'objectStorageArgs' provides arguments to the constructor of the class defined by 'objectStorageName'.
    # The 'invoke' argument sets how the storage accessor routes pipeline invocations.
    #  AWS accepts sqs, lambda or http. Azure accepts queue (Storage Queue) or http.
    #  Set TENX_OBJECT_STORAGE_INVOKE to override the default of sqs. An unknown value fails the accessor at construction.
    objectStorageArgs:
      - invoke
      - $=TenXEnv.get("TENX_OBJECT_STORAGE_INVOKE", "sqs")

    # 'container' specifies the container in which index objects created by the 'index' module reside.
    #  On AWS the S3 event notification in TENX_LAMBDA_INPUT carries the bucket name at Records[0].s3.bucket.name.
    #  On Azure the Event Grid BlobCreated event in TENX_LAMBDA_INPUT carries the blob URL at data.url as
    #  https://{account}.blob.core.windows.net/{container}/{blob}. The container is the path segment after the host:
    #  the first '/' found from index 8 closes the host (index 8 skips the scheme), the next '/' closes the container.
    container: '$=TenXEnv.get("TENX_OBJECT_STORAGE_NAME", "AWS") == "Azure" ? TenXString.substring(TenXString.jsonpath(TenXEnv.get("TENX_LAMBDA_INPUT"), "$.data.url"), TenXString.indexOf(TenXString.jsonpath(TenXEnv.get("TENX_LAMBDA_INPUT"), "$.data.url"), "/", 8) + 1, TenXString.indexOf(TenXString.jsonpath(TenXEnv.get("TENX_LAMBDA_INPUT"), "$.data.url"), "/", TenXString.indexOf(TenXString.jsonpath(TenXEnv.get("TENX_LAMBDA_INPUT"), "$.data.url"), "/", 8) + 1)) : TenXString.jsonpath(TenXEnv.get("TENX_LAMBDA_INPUT"), "$.Records[0].s3.bucket.name")'

    # 'target' sets a logical name under which output index objects reside.
    #  This commonly refers to the name of the application/service which generated
    #  'targetObject' (e.g., 'my-service', 'my-broker')
    target: app

    # 'targetObject' points to the blob (e.g., log file) within being deleted
    #  On AWS this is Records[0].s3.object.key. On Azure it is the rest of data.url after the container segment.
    object: '$=TenXEnv.get("TENX_OBJECT_STORAGE_NAME", "AWS") == "Azure" ? TenXString.substring(TenXString.jsonpath(TenXEnv.get("TENX_LAMBDA_INPUT"), "$.data.url"), TenXString.indexOf(TenXString.jsonpath(TenXEnv.get("TENX_LAMBDA_INPUT"), "$.data.url"), "/", TenXString.indexOf(TenXString.jsonpath(TenXEnv.get("TENX_LAMBDA_INPUT"), "$.data.url"), "/", 8) + 1) + 1, TenXString.length(TenXString.jsonpath(TenXEnv.get("TENX_LAMBDA_INPUT"), "$.data.url"))) : TenXString.jsonpath(TenXEnv.get("TENX_LAMBDA_INPUT"), "$.Records[0].s3.object.key")'

Options

Specify the options below to configure multiple Delete index objects:

Name	Description	Category
indexDeleteObjectStorageName	Object storage logical name	General
indexDeleteContainer	Object storage container containing index objects	General
indexDeleteObject	The target object whose index objects to delete from storage	General
indexDeleteObjectStorageArgs	Custom object storage args	Advanced
indexDeleteTarget	Logical name identifying the origin of 'indexDeleteObject'	Output

General

`indexDeleteObjectStorageName`

Object storage logical name.

Type	Required	Category
String	✔	General

Identifies the object storage from which to delete the target index objects (e.g., 'AWS').

`indexDeleteContainer`

Object storage container containing index objects.

Type	Required	Category
String	✔	General

Specifies the object storage container (e.g., AWS S3 bucket) name from which to delete TenXTemplate Filters.

`indexDeleteObject`

The target object whose index objects to delete from storage.

Type	Required	Category
String	✔	General

Sets the name of the object (e.g., log file) whose associated index objects to delete from indexDeleteContainer.

Advanced

`indexDeleteObjectStorageArgs`

Custom object storage args.

Type	Default	Category
List	[]	Advanced

Custom arguments passed as a map to the constructor of the underlying object storage. This list is expected to hold pairs of key values (e.g., args: [key1, value1, key2, value2]).

Output

`indexDeleteTarget`

Logical name identifying the origin of 'indexDeleteObject'.

Type	Required	Category
String	✔	Output

Specifies a logical name to delete TenXTemplate Filters produced for indexDeleteTargetObject under. This name commonly specifies the app which generated the events enclosed within this blob (e.g. 'acme-client').

This module is defined in delete/module.yaml.