Créer un pipeline
https://api.datadoghq.com/api/v2/obs-pipelines/pipelineshttps://api.us3.datadoghq.com/api/v2/obs-pipelines/pipelineshttps://api.us5.datadoghq.com/api/v2/obs-pipelines/pipelineshttps://api.datadoghq.eu/api/v2/obs-pipelines/pipelineshttps://api.ap1.datadoghq.com/api/v2/obs-pipelines/pipelineshttps://api.ap2.datadoghq.com/api/v2/obs-pipelines/pipelineshttps://api.uk1.datadoghq.com/api/v2/obs-pipelines/pipelineshttps://api.ddog-gov.com/api/v2/obs-pipelines/pipelineshttps://api.us2.ddog-gov.com/api/v2/obs-pipelines/pipelinesPrésentation
Créez un pipeline.
Requête
Body Data (required)
data [required]Schéma d'entrée représentant une configuration de pipeline d'observabilité. Utilisé dans les requêtes de création et de validation.
attributes [required]Defines the pipeline’s name and its components (sources, processors, and destinations).
config [required]Specifies the pipeline's configuration, including its sources, processors, and destinations.
destinations [required]A list of destination components where processed logs are sent.
<type=elasticsearch>The elasticsearch destination writes logs or metrics to an Elasticsearch cluster.
Supported pipeline types: logs, metrics
api_versionThe Elasticsearch API version to use. Set to auto to auto-detect.
auto, v6, v7, v8authAuthentication settings for the Elasticsearch destination.
When strategy is basic, use username_key and password_key to reference credentials stored in environment variables or secrets.
password_keyName of the environment variable or secret that holds the Elasticsearch password (used when strategy is basic).
strategy [required]The authentication strategy to use.
basic, awsusername_keyName of the environment variable or secret that holds the Elasticsearch username (used when strategy is basic).
bufferConfiguration for buffer settings on destination components.
Object 1Options for configuring a disk buffer.
max_size [required]Maximum size of the disk buffer.
typeThe type of the buffer that will be configured, a disk buffer.
diskdiskwhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 2Options for configuring a memory buffer by byte size.
max_size [required]Maximum size of the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 3Options for configuring a memory buffer by queue length.
max_events [required]Maximum events for the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestbulk_indexThe name of the index to write events to in Elasticsearch.
compressionCompression configuration for the Elasticsearch destination.
algorithm [required]The compression algorithm applied when sending data to Elasticsearch.
none, gzip, zlib, zstd, snappylevelThe compression level. Only applicable for gzip, zlib, and zstd algorithms.
data_streamConfiguration options for writing to Elasticsearch Data Streams instead of a fixed index.
auto_routingWhen true, automatically routes events to the appropriate data stream based on the event content.
datasetThe data stream dataset. This groups events by their source or application.
dtypeThe data stream type. This determines how events are categorized within the data stream.
namespaceThe data stream namespace. This separates events into different environments or domains.
sync_fieldsWhen true, synchronizes data stream fields with the Elasticsearch index mapping.
endpoint_url_keyName of the environment variable or secret that holds the Elasticsearch endpoint URL.
id [required]The unique identifier for this component.
id_keyThe name of the field used as the document ID in Elasticsearch.
inputs [required]A list of component IDs whose output is used as the input for this component.
pipelineThe name of an Elasticsearch ingest pipeline to apply to events before indexing.
request_retry_partialWhen true, retries failed partial bulk requests when some events in a batch fail while others succeed.
tlsConfiguration for enabling TLS encryption between the pipeline component and external services.
ca_filePath to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_filePath to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
key_pass_keyName of the environment variable or secret that holds the passphrase for the private key file.
type [required]The destination type. The value should always be elasticsearch.
elasticsearchelasticsearch<type=http_client>The http_client destination sends data to an HTTP endpoint.
Supported pipeline types: logs, metrics
auth_strategyHTTP authentication strategy.
none, basic, bearer, custombufferConfiguration for buffer settings on destination components.
Object 1Options for configuring a disk buffer.
max_size [required]Maximum size of the disk buffer.
typeThe type of the buffer that will be configured, a disk buffer.
diskdiskwhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 2Options for configuring a memory buffer by byte size.
max_size [required]Maximum size of the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 3Options for configuring a memory buffer by queue length.
max_events [required]Maximum events for the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestcompressionCompression configuration for HTTP requests.
algorithm [required]Compression algorithm.
gzipcustom_keyName of the environment variable or secret that holds a custom header value (used with custom auth strategies).
encoding [required]Encoding format for log events.
jsonid [required]The unique identifier for this component.
inputs [required]A list of component IDs whose output is used as the input for this component.
password_keyName of the environment variable or secret that holds the password (used when auth_strategy is basic).
tlsConfiguration for enabling TLS encryption between the pipeline component and external services.
ca_filePath to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_filePath to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
key_pass_keyName of the environment variable or secret that holds the passphrase for the private key file.
server_nameServer name to use for Server Name Indication (SNI) and to verify against the certificate presented by the remote host. Use this when the address you connect to doesn't match the certificate's Common Name or Subject Alternative Name.
token_keyName of the environment variable or secret that holds the bearer token (used when auth_strategy is bearer).
type [required]The destination type. The value should always be http_client.
http_clienthttp_clienturi_keyName of the environment variable or secret that holds the HTTP endpoint URI.
username_keyName of the environment variable or secret that holds the username (used when auth_strategy is basic).
<type=amazon_opensearch>The amazon_opensearch destination writes logs to Amazon OpenSearch.
Supported pipeline types: logs
authAuthentication settings for the Amazon OpenSearch destination.
The strategy field determines whether basic or AWS-based authentication is used.
When strategy is basic, use username_key and password_key to reference credentials stored in environment variables or secrets.
assume_roleThe ARN of the role to assume (used with aws strategy).
aws_regionAWS region
external_idExternal ID for the assumed role (used with aws strategy).
password_keyName of the environment variable or secret that holds the Amazon OpenSearch password (used when strategy is basic).
session_nameSession name for the assumed role (used with aws strategy).
strategy [required]The authentication strategy to use.
basic, awsusername_keyName of the environment variable or secret that holds the Amazon OpenSearch username (used when strategy is basic).
bufferConfiguration for buffer settings on destination components.
Object 1Options for configuring a disk buffer.
max_size [required]Maximum size of the disk buffer.
typeThe type of the buffer that will be configured, a disk buffer.
diskdiskwhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 2Options for configuring a memory buffer by byte size.
max_size [required]Maximum size of the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 3Options for configuring a memory buffer by queue length.
max_events [required]Maximum events for the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestbulk_indexThe index to write logs to.
data_streamConfiguration options for writing to OpenSearch Data Streams instead of a fixed index.
datasetThe data stream dataset for your logs. This groups logs by their source or application.
dtypeThe data stream type for your logs. This determines how logs are categorized within the data stream.
namespaceThe data stream namespace for your logs. This separates logs into different environments or domains.
endpoint_url_keyName of the environment variable or secret that holds the Amazon OpenSearch endpoint URL.
id [required]The unique identifier for this component.
inputs [required]A list of component IDs whose output is used as the input for this component.
type [required]The destination type. The value should always be amazon_opensearch.
amazon_opensearchamazon_opensearch<type=amazon_s3>The amazon_s3 destination sends your logs in Datadog-rehydratable format to an Amazon S3 bucket for archiving.
Supported pipeline types: logs
authAWS authentication credentials used for accessing AWS services such as S3. If omitted, the system’s default credentials are used (for example, the IAM role and environment variables).
assume_roleThe Amazon Resource Name (ARN) of the role to assume.
external_idA unique identifier for cross-account role assumption.
session_nameA session identifier used for logging and tracing the assumed role session.
bucket [required]S3 bucket name.
bufferConfiguration for buffer settings on destination components.
Object 1Options for configuring a disk buffer.
max_size [required]Maximum size of the disk buffer.
typeThe type of the buffer that will be configured, a disk buffer.
diskdiskwhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 2Options for configuring a memory buffer by byte size.
max_size [required]Maximum size of the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 3Options for configuring a memory buffer by queue length.
max_events [required]Maximum events for the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestcompressionCompression configuration for archived logs. When omitted, logs are compressed with gzip for backward compatibility.
<algorithm=zstd>Zstd compression.
algorithm [required]The compression type. Always zstd.
zstdzstdlevel [required]Zstd compression level. Valid values range from 1 to 21.
<algorithm=gzip>Gzip compression.
algorithm [required]The compression type. Always gzip.
gzipgziplevel [required]Gzip compression level. Valid values range from 1 to 9.
id [required]Unique identifier for the destination component.
inputs [required]A list of component IDs whose output is used as the input for this component.
key_prefixOptional prefix for object keys.
region [required]AWS region of the S3 bucket.
server_side_encryptionServer-side encryption type for Amazon S3.
aws:kms, AES256ssekms_key_idThe AWS KMS key ID used for SSE-KMS encryption.
Only applies when server_side_encryption is set to aws:kms.
storage_class [required]S3 storage class.
STANDARD, REDUCED_REDUNDANCY, INTELLIGENT_TIERING, STANDARD_IA, EXPRESS_ONEZONE, ONEZONE_IA, GLACIER, GLACIER_IR, DEEP_ARCHIVEtlsConfiguration for enabling TLS encryption between the pipeline component and external services.
ca_filePath to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_filePath to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
key_pass_keyName of the environment variable or secret that holds the passphrase for the private key file.
type [required]The destination type. Always amazon_s3.
amazon_s3amazon_s3<type=amazon_s3_generic>The amazon_s3_generic destination sends your logs to an Amazon S3 bucket.
Supported pipeline types: logs
authAWS authentication credentials used for accessing AWS services such as S3. If omitted, the system’s default credentials are used (for example, the IAM role and environment variables).
assume_roleThe Amazon Resource Name (ARN) of the role to assume.
external_idA unique identifier for cross-account role assumption.
session_nameA session identifier used for logging and tracing the assumed role session.
batch_settingsEvent batching settings
batch_sizeMaximum batch size in bytes.
timeout_secsMaximum number of seconds to wait before flushing the batch.
bucket [required]S3 bucket name.
bufferConfiguration for buffer settings on destination components.
Object 1Options for configuring a disk buffer.
max_size [required]Maximum size of the disk buffer.
typeThe type of the buffer that will be configured, a disk buffer.
diskdiskwhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 2Options for configuring a memory buffer by byte size.
max_size [required]Maximum size of the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 3Options for configuring a memory buffer by queue length.
max_events [required]Maximum events for the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestcompression [required]Compression algorithm applied to encoded logs.
<algorithm=zstd>Zstd compression.
algorithm [required]The compression type. Always zstd.
zstdzstdlevel [required]Zstd compression level.
<algorithm=gzip>Gzip compression.
algorithm [required]The compression type. Always gzip.
gzipgziplevel [required]Gzip compression level.
<algorithm=snappy>Snappy compression.
algorithm [required]The compression type. Always snappy.
snappysnappyencoding [required]Encoding format for the destination.
<type=json>JSON encoding.
type [required]The encoding type. Always json.
jsonjson<type=parquet>Parquet encoding.
type [required]The encoding type. Always parquet.
parquetparquetid [required]Unique identifier for the destination component.
inputs [required]A list of component IDs whose output is used as the input for this component.
key_prefixOptional prefix for object keys.
region [required]AWS region of the S3 bucket.
server_side_encryptionServer-side encryption type for Amazon S3.
aws:kms, AES256ssekms_key_idThe AWS KMS key ID used for SSE-KMS encryption.
Only applies when server_side_encryption is set to aws:kms.
storage_class [required]S3 storage class.
STANDARD, REDUCED_REDUNDANCY, INTELLIGENT_TIERING, STANDARD_IA, EXPRESS_ONEZONE, ONEZONE_IA, GLACIER, GLACIER_IR, DEEP_ARCHIVEtype [required]The destination type. Always amazon_s3_generic.
amazon_s3_genericamazon_s3_generic<type=amazon_security_lake>The amazon_security_lake destination sends your logs to Amazon Security Lake.
Supported pipeline types: logs
authAWS authentication credentials used for accessing AWS services such as S3. If omitted, the system’s default credentials are used (for example, the IAM role and environment variables).
assume_roleThe Amazon Resource Name (ARN) of the role to assume.
external_idA unique identifier for cross-account role assumption.
session_nameA session identifier used for logging and tracing the assumed role session.
bucket [required]Name of the Amazon S3 bucket in Security Lake (3-63 characters).
bufferConfiguration for buffer settings on destination components.
Object 1Options for configuring a disk buffer.
max_size [required]Maximum size of the disk buffer.
typeThe type of the buffer that will be configured, a disk buffer.
diskdiskwhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 2Options for configuring a memory buffer by byte size.
max_size [required]Maximum size of the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 3Options for configuring a memory buffer by queue length.
max_events [required]Maximum events for the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestcustom_source_name [required]Custom source name for the logs in Security Lake.
id [required]Unique identifier for the destination component.
inputs [required]A list of component IDs whose output is used as the input for this component.
region [required]AWS region of the S3 bucket.
tlsConfiguration for enabling TLS encryption between the pipeline component and external services.
ca_filePath to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_filePath to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
key_pass_keyName of the environment variable or secret that holds the passphrase for the private key file.
type [required]The destination type. Always amazon_security_lake.
amazon_security_lakeamazon_security_lake<type=azure_storage>The azure_storage destination forwards logs to an Azure Blob Storage container.
Supported pipeline types: logs
blob_prefixOptional prefix for blobs written to the container.
bufferConfiguration for buffer settings on destination components.
Object 1Options for configuring a disk buffer.
max_size [required]Maximum size of the disk buffer.
typeThe type of the buffer that will be configured, a disk buffer.
diskdiskwhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 2Options for configuring a memory buffer by byte size.
max_size [required]Maximum size of the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 3Options for configuring a memory buffer by queue length.
max_events [required]Maximum events for the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestcompressionCompression configuration for archived logs. When omitted, logs are compressed with gzip for backward compatibility.
<algorithm=zstd>Zstd compression.
algorithm [required]The compression type. Always zstd.
zstdzstdlevel [required]Zstd compression level. Valid values range from 1 to 21.
<algorithm=gzip>Gzip compression.
algorithm [required]The compression type. Always gzip.
gzipgziplevel [required]Gzip compression level. Valid values range from 1 to 9.
connection_string_keyName of the environment variable or secret that holds the Azure Storage connection string.
container_name [required]The name of the Azure Blob Storage container to store logs in.
id [required]The unique identifier for this component.
inputs [required]A list of component IDs whose output is used as the input for this component.
type [required]The destination type. The value should always be azure_storage.
azure_storageazure_storage<type=azure_data_explorer>The azure_data_explorer destination sends log events to an Azure Data Explorer table.
Supported pipeline types: logs
auth [required]Authentication configuration for Azure Data Explorer. The azure_credential_kind field selects the credential type.
<azure_credential_kind=azure_cli>Authenticate using the Azure CLI credentials available in the environment.
azure_credential_kind [required]The Azure credential kind. The value should always be azure_cli.
azure_cliazure_cli<azure_credential_kind=client_secret_credential>Authenticate using a Microsoft Entra application client secret.
azure_client_id [required]The Microsoft Entra application (client) ID.
azure_client_secret_key [required]Name of the environment variable or secret that holds the Microsoft Entra application client secret.
azure_credential_kind [required]The Azure credential kind. The value should always be client_secret_credential.
client_secret_credentialclient_secret_credentialazure_tenant_id [required]The Microsoft Entra tenant ID.
<azure_credential_kind=client_certificate_credential>Authenticate using a Microsoft Entra application client certificate.
azure_client_id [required]The Microsoft Entra application (client) ID.
azure_credential_kind [required]The Azure credential kind. The value should always be client_certificate_credential.
client_certificate_credentialclient_certificate_credentialazure_tenant_id [required]The Microsoft Entra tenant ID.
certificate_password_keyName of the environment variable or secret that holds the password for the client certificate.
certificate_path [required]Path to the .pfx client certificate file on the Worker.
<azure_credential_kind=managed_identity>Authenticate using an Azure managed identity.
azure_credential_kind [required]The Azure credential kind. The value should always be managed_identity.
managed_identitymanaged_identityuser_assigned_managed_identity_idThe ID of the user-assigned managed identity. If omitted, the system-assigned managed identity is used.
user_assigned_managed_identity_id_typeThe type of the user-assigned managed identity ID.
client_id, object_id, resource_id<azure_credential_kind=managed_identity_client_assertion>Authenticate using a managed identity as a client assertion for a Microsoft Entra application.
azure_credential_kind [required]The Azure credential kind. The value should always be managed_identity_client_assertion.
managed_identity_client_assertionmanaged_identity_client_assertionclient_assertion_client_id [required]The client ID of the Microsoft Entra application that trusts the managed identity.
client_assertion_tenant_id [required]The tenant ID of the Microsoft Entra application that trusts the managed identity.
user_assigned_managed_identity_idThe ID of the user-assigned managed identity. If omitted, the system-assigned managed identity is used.
user_assigned_managed_identity_id_typeThe type of the user-assigned managed identity ID.
client_id, object_id, resource_id<azure_credential_kind=workload_identity>Authenticate using Azure Workload Identity (for example, on Kubernetes).
azure_credential_kind [required]The Azure credential kind. The value should always be workload_identity.
workload_identityworkload_identityclient_idThe client ID of the Microsoft Entra application. If omitted, it is read from the environment.
tenant_idThe Microsoft Entra tenant ID. If omitted, it is read from the environment.
token_file_pathPath to the federated token file. If omitted, it is read from the environment.
batchEvent batching settings for Azure Data Explorer ingestion.
max_bytesMaximum batch size in bytes.
max_eventsMaximum number of events per batch before it is flushed.
timeout_secsMaximum number of seconds to wait before flushing a partial batch.
bufferConfiguration for buffer settings on destination components.
Object 1Options for configuring a disk buffer.
max_size [required]Maximum size of the disk buffer.
typeThe type of the buffer that will be configured, a disk buffer.
diskdiskwhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 2Options for configuring a memory buffer by byte size.
max_size [required]Maximum size of the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 3Options for configuring a memory buffer by queue length.
max_events [required]Maximum events for the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestcompressionGzip compression.
algorithm [required]The compression type. Always gzip.
gzipgziplevel [required]Gzip compression level. Valid values range from 1 to 9.
database [required]The name of the Azure Data Explorer database to ingest into. Supports template syntax.
id [required]The unique identifier for this component.
ingestion_endpoint_keyName of the environment variable or secret that holds the Azure Data Explorer ingestion endpoint URL.
Defaults to DESTINATION_AZURE_DATA_EXPLORER_INGESTION_ENDPOINT (prefixed with DD_OP_ at runtime).
inputs [required]A list of component IDs whose output is used as the input for this component.
mapping_referenceThe name of a pre-created ingestion mapping on the table used to map incoming events to columns. Supports template syntax.
table [required]The name of the Azure Data Explorer table to ingest into. Supports template syntax.
token_scopeThe OAuth scope requested when acquiring an access token for Azure Data Explorer.
Defaults to https://kusto.kusto.windows.net/.default.
type [required]The destination type. The value should always be azure_data_explorer.
azure_data_explorerazure_data_explorer<type=clickhouse>The clickhouse destination sends log events to a ClickHouse database table over HTTP.
Supported pipeline types: logs.
authHTTP Basic Authentication credentials for the ClickHouse destination.
When strategy is basic, provide username_key and password_key that reference environment variables or secrets containing the credentials.
password_keyName of the environment variable or secret that contains the ClickHouse password.
strategy [required]The authentication strategy for ClickHouse HTTP requests. Only basic is supported.
basicusername_keyName of the environment variable or secret that contains the ClickHouse username.
batchBatching configuration for ClickHouse inserts.
max_eventsMaximum number of events per batch before it is flushed.
timeout_secsMaximum number of seconds to wait before flushing a partial batch.
batch_encodingBatch encoding configuration for the ClickHouse destination.
Required when format is arrow_stream. The codec field must be set to arrow_stream.
allow_nullable_fieldsWhen true, null values are allowed for non-nullable fields in the ClickHouse schema.
When false (default), missing values for non-nullable columns cause encoding errors.
codec [required]The codec used for batch encoding. Only arrow_stream is supported.
arrow_streambufferConfiguration for buffer settings on destination components.
Object 1Options for configuring a disk buffer.
max_size [required]Maximum size of the disk buffer.
typeThe type of the buffer that will be configured, a disk buffer.
diskdiskwhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 2Options for configuring a memory buffer by byte size.
max_size [required]Maximum size of the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 3Options for configuring a memory buffer by queue length.
max_events [required]Maximum events for the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestcompressionCompression setting for outbound HTTP requests to ClickHouse.
Can be specified as a shorthand string ("gzip" or "none") or as an object
with an algorithm field and an optional level (gzip only, 1–9).
Object 1The compression algorithm applied to outbound HTTP requests.
gzip, noneObject 2Structured compression configuration for the ClickHouse destination.
Use algorithm to specify the compression type and level (optional, gzip only) to control compression strength.
algorithm [required]The compression algorithm applied to outbound HTTP requests.
gzip, nonelevelCompression level (1–9). Only applicable when algorithm is gzip.
databaseOptional ClickHouse database name. If omitted, the user's default database on the ClickHouse server is used.
date_time_best_effortWhen true, enables flexible DateTime parsing on the ClickHouse server side.
endpoint_url_keyName of the environment variable or secret that contains the ClickHouse HTTP endpoint URL.
Defaults to DESTINATION_CLICKHOUSE_ENDPOINT_URL (prefixed with DD_OP_ at runtime).
formatInsert format for events sent to ClickHouse.
json_each_row: Maps event fields to columns by name (ClickHouseJSONEachRow).json_as_object: Inserts each event into a singleObject('json')/JSONcolumn (ClickHouseJSONAsObject).json_as_string: Inserts each event into a singleString-typed column as raw JSON (ClickHouseJSONAsString).arrow_stream: Batches events using Apache Arrow IPC streaming format. Requiresbatch_encoding.
json_each_row, json_as_object, json_as_string, arrow_streamid [required]The unique identifier for this component.
inputs [required]A list of component IDs whose output is used as the input for this component.
skip_unknown_fieldsWhen true, fields not present in the target table schema are dropped instead of causing insert errors.
When unset, the ClickHouse server's own input_format_skip_unknown_fields setting applies.
table [required]Target ClickHouse table name. Events are inserted into this table.
tlsConfiguration for enabling TLS encryption between the pipeline component and external services.
ca_filePath to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_filePath to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
key_pass_keyName of the environment variable or secret that holds the passphrase for the private key file.
type [required]The destination type. The value must be clickhouse.
clickhouseclickhouse<type=cloud_prem>The cloud_prem destination sends logs to Datadog CloudPrem.
Supported pipeline types: logs
bufferConfiguration for buffer settings on destination components.
Object 1Options for configuring a disk buffer.
max_size [required]Maximum size of the disk buffer.
typeThe type of the buffer that will be configured, a disk buffer.
diskdiskwhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 2Options for configuring a memory buffer by byte size.
max_size [required]Maximum size of the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 3Options for configuring a memory buffer by queue length.
max_events [required]Maximum events for the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestendpoint_url_keyName of the environment variable or secret that holds the CloudPrem endpoint URL.
id [required]The unique identifier for this component.
inputs [required]A list of component IDs whose output is used as the input for this component.
tlsConfiguration for TLS encryption.
ca_filePath to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_filePath to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
key_pass_keyName of the environment variable or secret that holds the passphrase for the private key file.
server_nameServer name to use for Server Name Indication (SNI) and to verify against the certificate presented by the remote host. Use this when the address you connect to doesn't match the certificate's Common Name or Subject Alternative Name.
type [required]The destination type. The value should always be cloud_prem.
cloud_premcloud_prem<type=crowdstrike_next_gen_siem>The crowdstrike_next_gen_siem destination forwards logs to CrowdStrike Next Gen SIEM.
Supported pipeline types: logs
bufferConfiguration for buffer settings on destination components.
Object 1Options for configuring a disk buffer.
max_size [required]Maximum size of the disk buffer.
typeThe type of the buffer that will be configured, a disk buffer.
diskdiskwhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 2Options for configuring a memory buffer by byte size.
max_size [required]Maximum size of the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 3Options for configuring a memory buffer by queue length.
max_events [required]Maximum events for the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestcompressionCompression configuration for log events.
algorithm [required]Compression algorithm for log events.
gzip, zliblevelCompression level.
encoding [required]Encoding format for log events.
json, raw_messageendpoint_url_keyName of the environment variable or secret that holds the CrowdStrike endpoint URL.
id [required]The unique identifier for this component.
inputs [required]A list of component IDs whose output is used as the input for this component.
tlsConfiguration for enabling TLS encryption between the pipeline component and external services.
ca_filePath to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_filePath to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
key_pass_keyName of the environment variable or secret that holds the passphrase for the private key file.
token_keyName of the environment variable or secret that holds the CrowdStrike API token.
type [required]The destination type. The value should always be crowdstrike_next_gen_siem.
crowdstrike_next_gen_siemcrowdstrike_next_gen_siem<type=datadog_logs>The datadog_logs destination forwards logs to Datadog Log Management.
Supported pipeline types: logs
bufferConfiguration for buffer settings on destination components.
Object 1Options for configuring a disk buffer.
max_size [required]Maximum size of the disk buffer.
typeThe type of the buffer that will be configured, a disk buffer.
diskdiskwhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 2Options for configuring a memory buffer by byte size.
max_size [required]Maximum size of the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 3Options for configuring a memory buffer by queue length.
max_events [required]Maximum events for the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestid [required]The unique identifier for this component.
inputs [required]A list of component IDs whose output is used as the input for this component.
routesA list of routing rules that forward matching logs to Datadog using dedicated API keys.
api_key_keyName of the environment variable or secret that stores the Datadog API key used by this route.
includeA Datadog search query that determines which logs are forwarded using this route.
route_idUnique identifier for this route within the destination.
siteDatadog site where matching logs are sent (for example, us1).
type [required]The destination type. The value should always be datadog_logs.
datadog_logsdatadog_logs<type=google_chronicle>The google_chronicle destination sends logs to Google Chronicle.
Supported pipeline types: logs
authGoogle Cloud credentials used to authenticate with Google Cloud Storage.
credentials_file [required]Path to the Google Cloud service account key file.
bufferConfiguration for buffer settings on destination components.
Object 1Options for configuring a disk buffer.
max_size [required]Maximum size of the disk buffer.
typeThe type of the buffer that will be configured, a disk buffer.
diskdiskwhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 2Options for configuring a memory buffer by byte size.
max_size [required]Maximum size of the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 3Options for configuring a memory buffer by queue length.
max_events [required]Maximum events for the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestcustomer_id [required]The Google Chronicle customer ID.
encodingThe encoding format for the logs sent to Chronicle.
json, raw_messageendpoint_url_keyName of the environment variable or secret that holds the Google Chronicle endpoint URL.
id [required]The unique identifier for this component.
inputs [required]A list of component IDs whose output is used as the input for this component.
log_typeThe log type metadata associated with the Chronicle destination.
type [required]The destination type. The value should always be google_chronicle.
google_chroniclegoogle_chronicle<type=google_cloud_storage>The google_cloud_storage destination stores logs in a Google Cloud Storage (GCS) bucket.
It requires a bucket name, Google Cloud authentication, and metadata fields.
Supported pipeline types: logs
aclAccess control list setting for objects written to the bucket.
private, project-private, public-read, authenticated-read, bucket-owner-read, bucket-owner-full-controlauthGoogle Cloud credentials used to authenticate with Google Cloud Storage.
credentials_file [required]Path to the Google Cloud service account key file.
bucket [required]Name of the GCS bucket.
bufferConfiguration for buffer settings on destination components.
Object 1Options for configuring a disk buffer.
max_size [required]Maximum size of the disk buffer.
typeThe type of the buffer that will be configured, a disk buffer.
diskdiskwhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 2Options for configuring a memory buffer by byte size.
max_size [required]Maximum size of the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 3Options for configuring a memory buffer by queue length.
max_events [required]Maximum events for the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestcompressionCompression configuration for archived logs. When omitted, logs are compressed with gzip for backward compatibility.
<algorithm=zstd>Zstd compression.
algorithm [required]The compression type. Always zstd.
zstdzstdlevel [required]Zstd compression level. Valid values range from 1 to 21.
<algorithm=gzip>Gzip compression.
algorithm [required]The compression type. Always gzip.
gzipgziplevel [required]Gzip compression level. Valid values range from 1 to 9.
id [required]Unique identifier for the destination component.
inputs [required]A list of component IDs whose output is used as the input for this component.
key_prefixOptional prefix for object keys within the GCS bucket.
metadataCustom metadata to attach to each object uploaded to the GCS bucket.
name [required]The metadata key.
value [required]The metadata value.
storage_class [required]Storage class used for objects stored in GCS.
STANDARD, NEARLINE, COLDLINE, ARCHIVEtype [required]The destination type. Always google_cloud_storage.
google_cloud_storagegoogle_cloud_storage<type=google_pubsub>The google_pubsub destination publishes logs to a Google Cloud Pub/Sub topic.
Supported pipeline types: logs
authGoogle Cloud credentials used to authenticate with Google Cloud Storage.
credentials_file [required]Path to the Google Cloud service account key file.
bufferConfiguration for buffer settings on destination components.
Object 1Options for configuring a disk buffer.
max_size [required]Maximum size of the disk buffer.
typeThe type of the buffer that will be configured, a disk buffer.
diskdiskwhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 2Options for configuring a memory buffer by byte size.
max_size [required]Maximum size of the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 3Options for configuring a memory buffer by queue length.
max_events [required]Maximum events for the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestencoding [required]Encoding format for log events.
json, raw_messageendpoint_url_keyName of the environment variable or secret that holds the Google Cloud Pub/Sub endpoint URL.
id [required]The unique identifier for this component.
inputs [required]A list of component IDs whose output is used as the input for this component.
project [required]The Google Cloud project ID that owns the Pub/Sub topic.
tlsConfiguration for enabling TLS encryption between the pipeline component and external services.
ca_filePath to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_filePath to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
key_pass_keyName of the environment variable or secret that holds the passphrase for the private key file.
topic [required]The Pub/Sub topic name to publish logs to.
type [required]The destination type. The value should always be google_pubsub.
google_pubsubgoogle_pubsub<type=kafka>The kafka destination sends logs to Apache Kafka topics.
Supported pipeline types: logs
bootstrap_servers_keyName of the environment variable or secret that holds the Kafka bootstrap servers list.
bufferConfiguration for buffer settings on destination components.
Object 1Options for configuring a disk buffer.
max_size [required]Maximum size of the disk buffer.
typeThe type of the buffer that will be configured, a disk buffer.
diskdiskwhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 2Options for configuring a memory buffer by byte size.
max_size [required]Maximum size of the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 3Options for configuring a memory buffer by queue length.
max_events [required]Maximum events for the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestcompressionCompression codec for Kafka messages.
none, gzip, snappy, lz4, zstdencoding [required]Encoding format for log events.
json, raw_messageheaders_keyThe field name to use for Kafka message headers.
id [required]The unique identifier for this component.
inputs [required]A list of component IDs whose output is used as the input for this component.
key_fieldThe field name to use as the Kafka message key.
librdkafka_optionsOptional list of advanced Kafka producer configuration options, defined as key-value pairs.
name [required]The name of the librdkafka configuration option to set.
value [required]The value assigned to the specified librdkafka configuration option.
message_timeout_msMaximum time in milliseconds to wait for message delivery confirmation.
rate_limit_duration_secsDuration in seconds for the rate limit window.
rate_limit_numMaximum number of messages allowed per rate limit duration.
saslSpecifies the SASL mechanism for authenticating with a Kafka cluster.
mechanismSASL mechanism used for Kafka authentication.
PLAIN, SCRAM-SHA-256, SCRAM-SHA-512password_keyName of the environment variable or secret that holds the SASL password.
username_keyName of the environment variable or secret that holds the SASL username.
socket_timeout_msSocket timeout in milliseconds for network requests.
tlsConfiguration for enabling TLS encryption between the pipeline component and external services.
ca_filePath to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_filePath to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
key_pass_keyName of the environment variable or secret that holds the passphrase for the private key file.
topic [required]The Kafka topic name to publish logs to.
type [required]The destination type. The value should always be kafka.
kafkakafka<type=microsoft_sentinel>The microsoft_sentinel destination forwards logs to Microsoft Sentinel.
Supported pipeline types: logs
bufferConfiguration for buffer settings on destination components.
Object 1Options for configuring a disk buffer.
max_size [required]Maximum size of the disk buffer.
typeThe type of the buffer that will be configured, a disk buffer.
diskdiskwhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 2Options for configuring a memory buffer by byte size.
max_size [required]Maximum size of the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 3Options for configuring a memory buffer by queue length.
max_events [required]Maximum events for the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestclient_id [required]Azure AD client ID used for authentication.
client_secret_keyName of the environment variable or secret that holds the Azure AD client secret.
dce_uri_keyName of the environment variable or secret that holds the Data Collection Endpoint (DCE) URI.
dcr_immutable_id [required]The immutable ID of the Data Collection Rule (DCR).
id [required]The unique identifier for this component.
inputs [required]A list of component IDs whose output is used as the input for this component.
table [required]The name of the Log Analytics table where logs are sent.
tenant_id [required]Azure AD tenant ID.
type [required]The destination type. The value should always be microsoft_sentinel.
microsoft_sentinelmicrosoft_sentinel<type=new_relic>The new_relic destination sends logs to the New Relic platform.
Supported pipeline types: logs
account_id_keyName of the environment variable or secret that holds the New Relic account ID.
bufferConfiguration for buffer settings on destination components.
Object 1Options for configuring a disk buffer.
max_size [required]Maximum size of the disk buffer.
typeThe type of the buffer that will be configured, a disk buffer.
diskdiskwhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 2Options for configuring a memory buffer by byte size.
max_size [required]Maximum size of the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 3Options for configuring a memory buffer by queue length.
max_events [required]Maximum events for the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestid [required]The unique identifier for this component.
inputs [required]A list of component IDs whose output is used as the input for this component.
license_key_keyName of the environment variable or secret that holds the New Relic license key.
region [required]The New Relic region.
us, eutype [required]The destination type. The value should always be new_relic.
new_relicnew_relic<type=opensearch>The opensearch destination writes logs to an OpenSearch cluster.
Supported pipeline types: logs
authAuthentication settings for the Elasticsearch destination.
When strategy is basic, use username_key and password_key to reference credentials stored in environment variables or secrets.
password_keyName of the environment variable or secret that holds the Elasticsearch password (used when strategy is basic).
strategy [required]The authentication strategy to use.
basic, awsusername_keyName of the environment variable or secret that holds the Elasticsearch username (used when strategy is basic).
bufferConfiguration for buffer settings on destination components.
Object 1Options for configuring a disk buffer.
max_size [required]Maximum size of the disk buffer.
typeThe type of the buffer that will be configured, a disk buffer.
diskdiskwhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 2Options for configuring a memory buffer by byte size.
max_size [required]Maximum size of the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 3Options for configuring a memory buffer by queue length.
max_events [required]Maximum events for the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestbulk_indexThe index to write logs to.
data_streamConfiguration options for writing to OpenSearch Data Streams instead of a fixed index.
datasetThe data stream dataset for your logs. This groups logs by their source or application.
dtypeThe data stream type for your logs. This determines how logs are categorized within the data stream.
namespaceThe data stream namespace for your logs. This separates logs into different environments or domains.
endpoint_url_keyName of the environment variable or secret that holds the OpenSearch endpoint URL.
id [required]The unique identifier for this component.
inputs [required]A list of component IDs whose output is used as the input for this component.
type [required]The destination type. The value should always be opensearch.
opensearchopensearch<type=rsyslog>The rsyslog destination forwards logs to an external rsyslog server over TCP or UDP using the syslog protocol.
Supported pipeline types: logs
bufferConfiguration for buffer settings on destination components.
Object 1Options for configuring a disk buffer.
max_size [required]Maximum size of the disk buffer.
typeThe type of the buffer that will be configured, a disk buffer.
diskdiskwhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 2Options for configuring a memory buffer by byte size.
max_size [required]Maximum size of the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 3Options for configuring a memory buffer by queue length.
max_events [required]Maximum events for the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestendpoint_url_keyName of the environment variable or secret that holds the syslog server endpoint URL.
id [required]The unique identifier for this component.
inputs [required]A list of component IDs whose output is used as the input for this component.
keepaliveOptional socket keepalive duration in milliseconds.
tlsConfiguration for enabling TLS encryption between the pipeline component and external services.
ca_filePath to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_filePath to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
key_pass_keyName of the environment variable or secret that holds the passphrase for the private key file.
type [required]The destination type. The value should always be rsyslog.
rsyslogrsyslog<type=sentinel_one>The sentinel_one destination sends logs to SentinelOne.
Supported pipeline types: logs
bufferConfiguration for buffer settings on destination components.
Object 1Options for configuring a disk buffer.
max_size [required]Maximum size of the disk buffer.
typeThe type of the buffer that will be configured, a disk buffer.
diskdiskwhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 2Options for configuring a memory buffer by byte size.
max_size [required]Maximum size of the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 3Options for configuring a memory buffer by queue length.
max_events [required]Maximum events for the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestid [required]The unique identifier for this component.
inputs [required]A list of component IDs whose output is used as the input for this component.
region [required]The SentinelOne region to send logs to.
us, eu, ca, data_set_ustoken_keyName of the environment variable or secret that holds the SentinelOne API token.
type [required]The destination type. The value should always be sentinel_one.
sentinel_onesentinel_one<type=socket>The socket destination sends logs over TCP or UDP to a remote server.
Supported pipeline types: logs
address_keyName of the environment variable or secret that holds the socket address (host:port).
bufferConfiguration for buffer settings on destination components.
Object 1Options for configuring a disk buffer.
max_size [required]Maximum size of the disk buffer.
typeThe type of the buffer that will be configured, a disk buffer.
diskdiskwhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 2Options for configuring a memory buffer by byte size.
max_size [required]Maximum size of the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 3Options for configuring a memory buffer by queue length.
max_events [required]Maximum events for the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestencoding [required]Encoding format for log events.
json, raw_messageframing [required]Framing method configuration.
<method=newline_delimited>Each log event is delimited by a newline character.
method [required]The definition of ObservabilityPipelineSocketDestinationFramingNewlineDelimitedMethod object.
newline_delimited<method=bytes>Event data is not delimited at all.
method [required]The definition of ObservabilityPipelineSocketDestinationFramingBytesMethod object.
bytes<method=character_delimited>Each log event is separated using the specified delimiter character.
delimiter [required]A single ASCII character used as a delimiter.
method [required]The definition of ObservabilityPipelineSocketDestinationFramingCharacterDelimitedMethod object.
character_delimitedid [required]The unique identifier for this component.
inputs [required]A list of component IDs whose output is used as the input for this component.
mode [required]Protocol used to send logs.
tcp, udptlsTLS configuration. Relevant only when mode is tcp.
ca_filePath to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_filePath to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
key_pass_keyName of the environment variable or secret that holds the passphrase for the private key file.
server_nameServer name to use for Server Name Indication (SNI) and to verify against the certificate presented by the remote host. Use this when the address you connect to doesn't match the certificate's Common Name or Subject Alternative Name.
type [required]The destination type. The value should always be socket.
socketsocket<type=splunk_hec>The splunk_hec destination forwards logs to Splunk using the HTTP Event Collector (HEC).
Supported pipeline types: logs
auto_extract_timestampIf true, Splunk tries to extract timestamps from incoming log events.
If false, Splunk assigns the time the event was received.
Only applies when endpoint_target is event; cannot be true when endpoint_target is raw.
bufferConfiguration for buffer settings on destination components.
Object 1Options for configuring a disk buffer.
max_size [required]Maximum size of the disk buffer.
typeThe type of the buffer that will be configured, a disk buffer.
diskdiskwhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 2Options for configuring a memory buffer by byte size.
max_size [required]Maximum size of the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 3Options for configuring a memory buffer by queue length.
max_events [required]Maximum events for the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestencodingEncoding format for log events.
json, raw_messageendpoint_targetThe Splunk HEC endpoint to send events to. Use event to send structured events to the /event endpoint, or raw to send the raw message to the /raw endpoint.
event, rawendpoint_url_keyName of the environment variable or secret that holds the Splunk HEC endpoint URL.
id [required]The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the input to downstream components).
indexOptional name of the Splunk index where logs are written.
indexed_fieldsList of log field names to send as indexed fields to Splunk HEC. Available only when encoding is json.
inputs [required]A list of component IDs whose output is used as the input for this component.
sourcetypeThe Splunk sourcetype to assign to log events.
token_keyName of the environment variable or secret that holds the Splunk HEC token.
token_strategyControls how the Splunk HEC token is supplied. Use custom to provide a token with token_key, or from_source to forward the token received from an upstream Splunk HEC source.
custom, from_sourcetype [required]The destination type. Always splunk_hec.
splunk_hecsplunk_hec<type=sumo_logic>The sumo_logic destination forwards logs to Sumo Logic.
Supported pipeline types: logs
bufferConfiguration for buffer settings on destination components.
Object 1Options for configuring a disk buffer.
max_size [required]Maximum size of the disk buffer.
typeThe type of the buffer that will be configured, a disk buffer.
diskdiskwhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 2Options for configuring a memory buffer by byte size.
max_size [required]Maximum size of the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 3Options for configuring a memory buffer by queue length.
max_events [required]Maximum events for the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestencodingThe output encoding format.
json, raw_message, logfmtendpoint_url_keyName of the environment variable or secret that holds the Sumo Logic HTTP endpoint URL.
header_custom_fieldsA list of custom headers to include in the request to Sumo Logic.
name [required]The header field name.
value [required]The header field value.
header_host_nameOptional override for the host name header.
header_source_categoryOptional override for the source category header.
header_source_nameOptional override for the source name header.
id [required]The unique identifier for this component.
inputs [required]A list of component IDs whose output is used as the input for this component.
type [required]The destination type. The value should always be sumo_logic.
sumo_logicsumo_logic<type=syslog_ng>The syslog_ng destination forwards logs to an external syslog-ng server over TCP or UDP using the syslog protocol.
Supported pipeline types: logs
bufferConfiguration for buffer settings on destination components.
Object 1Options for configuring a disk buffer.
max_size [required]Maximum size of the disk buffer.
typeThe type of the buffer that will be configured, a disk buffer.
diskdiskwhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 2Options for configuring a memory buffer by byte size.
max_size [required]Maximum size of the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 3Options for configuring a memory buffer by queue length.
max_events [required]Maximum events for the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestendpoint_url_keyName of the environment variable or secret that holds the syslog-ng server endpoint URL.
id [required]The unique identifier for this component.
inputs [required]A list of component IDs whose output is used as the input for this component.
keepaliveOptional socket keepalive duration in milliseconds.
tlsConfiguration for enabling TLS encryption between the pipeline component and external services.
ca_filePath to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_filePath to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
key_pass_keyName of the environment variable or secret that holds the passphrase for the private key file.
server_nameServer name to use for Server Name Indication (SNI) and to verify against the certificate presented by the remote host. Use this when the address you connect to doesn't match the certificate's Common Name or Subject Alternative Name.
type [required]The destination type. The value should always be syslog_ng.
syslog_ngsyslog_ng<type=databricks_zerobus>The databricks_zerobus destination sends logs to Databricks using the Zerobus ingestion API, streaming data directly into your Databricks Lakehouse.
Supported pipeline types: Logs, rehydration
auth [required]OAuth credentials for authenticating with the Databricks Zerobus ingestion API.
client_id [required]Your service principal application ID (UUID).
client_secret_keyName of the environment variable or secret that holds the OAuth client secret used to authenticate with the Databricks ingestion endpoint.
bufferConfiguration for buffer settings on destination components.
Object 1Options for configuring a disk buffer.
max_size [required]Maximum size of the disk buffer.
typeThe type of the buffer that will be configured, a disk buffer.
diskdiskwhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 2Options for configuring a memory buffer by byte size.
max_size [required]Maximum size of the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 3Options for configuring a memory buffer by queue length.
max_events [required]Maximum events for the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestid [required]The unique identifier for this component.
ingestion_endpoint_keyName of the environment variable or the secret identifier that references the Databricks Zerobus ingestion endpoint, which is used to stream data directly into your Databricks Lakehouse.
inputs [required]A list of component IDs whose output is used as the input for this component.
table_name [required]The fully qualified name of your target Databricks table. Make sure this table already exists in your Databricks workspace before deploying.
type [required]The destination type. The value must be databricks_zerobus.
databricks_zerobusdatabricks_zerobusunity_catalog_endpoint_keyName of the environment variable or the secret identifier that references your Databricks workspace URL, which is used to communicate with the Unity Catalog API.
<type=datadog_metrics>The datadog_metrics destination forwards metrics to Datadog.
Supported pipeline types: metrics
id [required]The unique identifier for this component.
inputs [required]A list of component IDs whose output is used as the input for this component.
type [required]The destination type. The value should always be datadog_metrics.
datadog_metricsdatadog_metrics<type=prometheus_remote_write>The prometheus_remote_write destination forwards metrics to an endpoint that supports the Prometheus Remote Write protocol.
Supported pipeline types: metrics
auth_strategyThe authentication strategy to use for outgoing Prometheus Remote Write requests.
none, basic, bearerbufferConfiguration for buffer settings on destination components.
Object 1Options for configuring a disk buffer.
max_size [required]Maximum size of the disk buffer.
typeThe type of the buffer that will be configured, a disk buffer.
diskdiskwhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 2Options for configuring a memory buffer by byte size.
max_size [required]Maximum size of the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 3Options for configuring a memory buffer by queue length.
max_events [required]Maximum events for the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestdefault_namespaceThe default namespace to add as a prefix to metric names that do not already have one.
endpoint_url_keyName of the environment variable or secret that holds the Prometheus Remote Write endpoint URL.
Defaults to DESTINATION_PROMETHEUS_REMOTE_WRITE_ENDPOINT_URL (prefixed with DD_OP_ at runtime).
id [required]The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the input to downstream components).
inputs [required]A list of component IDs whose output is used as the input for this component.
password_keyName of the environment variable or secret that holds the password (used when auth_strategy is basic).
tenant_idThe tenant ID to include with outgoing requests. Used by multi-tenant Prometheus Remote Write receivers.
tlsConfiguration for enabling TLS encryption between the pipeline component and external services.
ca_filePath to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_filePath to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
key_pass_keyName of the environment variable or secret that holds the passphrase for the private key file.
server_nameServer name to use for Server Name Indication (SNI) and to verify against the certificate presented by the remote host. Use this when the address you connect to doesn't match the certificate's Common Name or Subject Alternative Name.
token_keyName of the environment variable or secret that holds the bearer token (used when auth_strategy is bearer).
type [required]The destination type. The value should always be prometheus_remote_write.
prometheus_remote_writeprometheus_remote_writeusername_keyName of the environment variable or secret that holds the username (used when auth_strategy is basic).
<type=splunk_hec_metrics>The splunk_hec_metrics destination forwards metrics to Splunk using the HTTP Event Collector (HEC).
Supported pipeline types: metrics
bufferConfiguration for buffer settings on destination components.
Object 1Options for configuring a disk buffer.
max_size [required]Maximum size of the disk buffer.
typeThe type of the buffer that will be configured, a disk buffer.
diskdiskwhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 2Options for configuring a memory buffer by byte size.
max_size [required]Maximum size of the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 3Options for configuring a memory buffer by queue length.
max_events [required]Maximum events for the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestcompressionCompression algorithm applied when sending metrics to Splunk HEC.
nonenone, gzipdefault_namespaceOptional default namespace for metrics sent to Splunk HEC.
endpoint_url_keyName of the environment variable or secret that holds the Splunk HEC endpoint URL.
id [required]The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the input to downstream components).
indexOptional name of the Splunk index where metrics are written.
inputs [required]A list of component IDs whose output is used as the input for this component.
sourceThe Splunk source field value for metric events.
sourcetypeThe Splunk sourcetype to assign to metric events.
tlsConfiguration for enabling TLS encryption between the pipeline component and external services.
ca_filePath to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_filePath to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
key_pass_keyName of the environment variable or secret that holds the passphrase for the private key file.
token_keyName of the environment variable or secret that holds the Splunk HEC token.
type [required]The destination type. Always splunk_hec_metrics.
splunk_hec_metricssplunk_hec_metrics<type=opentelemetry>The opentelemetry destination forwards metrics or traces using the OpenTelemetry Protocol (OTLP) over HTTP.
Supported pipeline types: metrics, traces
bufferConfiguration for buffer settings on destination components.
Object 1Options for configuring a disk buffer.
max_size [required]Maximum size of the disk buffer.
typeThe type of the buffer that will be configured, a disk buffer.
diskdiskwhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 2Options for configuring a memory buffer by byte size.
max_size [required]Maximum size of the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 3Options for configuring a memory buffer by queue length.
max_events [required]Maximum events for the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newesthttp_client_uri_keyEnvironment variable name containing the URI of the OTLP HTTP endpoint to send metrics or traces to.
DESTINATION_OTEL_HTTP_CLIENT_URIid [required]The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the input to downstream components).
inputs [required]A list of component IDs whose output is used as the input for this component.
tlsConfiguration for enabling TLS encryption between the pipeline component and external services.
ca_filePath to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_filePath to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
key_pass_keyName of the environment variable or secret that holds the passphrase for the private key file.
type [required]The destination type. Always opentelemetry.
opentelemetryopentelemetry<type=datadog_traces>The datadog_traces destination forwards traces to Datadog.
Supported pipeline types: traces
bufferConfiguration for buffer settings on destination components.
Object 1Options for configuring a disk buffer.
max_size [required]Maximum size of the disk buffer.
typeThe type of the buffer that will be configured, a disk buffer.
diskdiskwhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 2Options for configuring a memory buffer by byte size.
max_size [required]Maximum size of the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 3Options for configuring a memory buffer by queue length.
max_events [required]Maximum events for the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestid [required]The unique identifier for this component.
inputs [required]A list of component IDs whose output is used as the input for this component.
type [required]The destination type. The value should always be datadog_traces.
datadog_tracesdatadog_tracespipeline_typeThe type of data being ingested. Defaults to logs if not specified.
logslogs, metrics, tracesprocessor_groupsA list of processor groups that transform or enrich log data.
display_nameThe display name for a component.
enabled [required]Whether this processor group is enabled.
id [required]The unique identifier for the processor group.
include [required]Conditional expression for when this processor group should execute.
inputs [required]A list of IDs for components whose output is used as the input for this processor group.
processors [required]Processors applied sequentially within this group. Events flow through each processor in order.
<type=filter>The filter processor allows conditional processing of logs, metrics, or traces based on a Datadog search query. Events that match the include query are passed through; others are discarded.
Supported pipeline types: logs, metrics, traces
display_nameThe display name for a component.
enabled [required]Indicates whether the processor is enabled.
id [required]The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the input to downstream components).
include [required]A Datadog search query used to determine which events should pass through the filter. Events that match this query continue to downstream components; others are dropped.
type [required]The processor type. The value should always be filter.
filterfilter<type=add_env_vars>The add_env_vars processor adds environment variable values to log events.
Supported pipeline types: logs
display_nameThe display name for a component.
enabled [required]Indicates whether the processor is enabled.
id [required]The unique identifier for this component. Used to reference this processor in the pipeline.
include [required]A Datadog search query used to determine which logs this processor targets.
type [required]The processor type. The value should always be add_env_vars.
add_env_varsadd_env_varsvariables [required]A list of environment variable mappings to apply to log fields.
field [required]The target field in the log event.
name [required]The name of the environment variable to read.
<type=add_fields>The add_fields processor adds static key-value fields to logs.
Supported pipeline types: logs
display_nameThe display name for a component.
enabled [required]Indicates whether the processor is enabled.
fields [required]A list of static fields (key-value pairs) that is added to each log event processed by this component.
name [required]The field name.
value [required]The field value.
id [required]The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the input to downstream components).
include [required]A Datadog search query used to determine which logs this processor targets.
type [required]The processor type. The value should always be add_fields.
add_fieldsadd_fields<type=add_hostname>The add_hostname processor adds the hostname to log events.
Supported pipeline types: logs
display_nameThe display name for a component.
enabled [required]Indicates whether the processor is enabled.
id [required]The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the input to downstream components).
include [required]A Datadog search query used to determine which logs this processor targets.
type [required]The processor type. The value should always be add_hostname.
add_hostnameadd_hostname<type=datadog_tags>The datadog_tags processor includes or excludes specific Datadog tags in your logs.
Supported pipeline types: logs
action [required]The action to take on tags with matching keys.
include, excludedisplay_nameThe display name for a component.
enabled [required]Indicates whether the processor is enabled.
id [required]The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the input to downstream components).
include [required]A Datadog search query used to determine which logs this processor targets.
keys [required]A list of tag keys.
mode [required]The processing mode.
filtertype [required]The processor type. The value should always be datadog_tags.
datadog_tagsdatadog_tags<type=dedupe>The dedupe processor removes duplicate fields in log events.
Supported pipeline types: logs
cacheConfiguration for the cache used to detect duplicates.
num_events [required]The number of events to cache for duplicate detection.
display_nameThe display name for a component.
enabled [required]Indicates whether the processor is enabled.
fields [required]A list of log field paths to check for duplicates.
id [required]The unique identifier for this processor.
include [required]A Datadog search query used to determine which logs this processor targets.
mode [required]The deduplication mode to apply to the fields.
match, ignoretype [required]The processor type. The value should always be dedupe.
dedupededupe<type=enrichment_table>The enrichment_table processor enriches logs using a static CSV file, GeoIP database, or reference table. Exactly one of file, geoip, or reference_table must be configured.
Supported pipeline types: logs
display_nameThe display name for a component.
enabled [required]Indicates whether the processor is enabled.
fileDefines a static enrichment table loaded from a CSV file.
encoding [required]File encoding format.
delimiter [required]The encoding delimiter.
includes_headers [required]The encoding includes_headers.
type [required]Specifies the encoding format (e.g., CSV) used for enrichment tables.
csvkey [required]Key fields used to look up enrichment values.
column [required]The items column.
comparison [required]Defines how to compare key fields for enrichment table lookups.
equalsfield [required]Specifies the source of the key value used for enrichment table lookups.
Can be a plain field path string or an object specifying event, vrl, or secret.
Object 1A plain field path in the log event, used as the lookup key.
Object 2Looks up a value from a field path in the log event.
event [required]The path to the field in the log event to use as the lookup key.
Object 3Evaluates a VRL expression to produce the lookup key.
vrl [required]A VRL expression that returns the value to use as the lookup key.
Object 4Looks up a value stored as a pipeline secret.
secret [required]The name of the secret containing the lookup key value.
path [required]Path to the CSV file.
schema [required]Schema defining column names and their types.
column [required]The items column.
type [required]Declares allowed data types for enrichment table columns.
string, boolean, integer, float, date, timestampgeoipUses a GeoIP database to enrich logs based on an IP field.
key_field [required]Path to the IP field in the log.
locale [required]Locale used to resolve geographical names.
path [required]Path to the GeoIP database file.
id [required]The unique identifier for this processor.
include [required]A Datadog search query used to determine which logs this processor targets.
reference_tableUses a Datadog reference table to enrich logs.
app_key_keyName of the environment variable or secret that holds the Datadog application key used to access the reference table.
columnsList of column names to include from the reference table. If not provided, all columns are included.
key_field [required]Path to the field in the log event to match against the reference table.
table_id [required]The unique identifier of the reference table.
target [required]Path where enrichment results should be stored in the log.
type [required]The processor type. The value should always be enrichment_table.
enrichment_tableenrichment_table<type=generate_datadog_metrics>DEPRECATED The generate_datadog_metrics processor creates custom metrics from logs and sends them to Datadog.
Metrics can be counters, gauges, or distributions and optionally grouped by log fields.
Supported pipeline types: logs
Deprecated: This processor is deprecated. Use the generate_metrics processor instead.
display_nameThe display name for a component.
enabled [required]Indicates whether the processor is enabled.
id [required]The unique identifier for this component. Used to reference this component in other parts of the pipeline.
includeA Datadog search query used to determine which logs this processor targets.
metricsConfiguration for generating individual metrics.
group_byOptional fields used to group the metric series.
include [required]Datadog filter query to match events for metric generation.
metric_type [required]Type of metric to create.
count, gauge, distributionname [required]Name of the custom metric to be created.
value [required]Specifies how the value of the generated metric is computed.
<strategy=increment_by_one>Strategy that increments a generated metric by one for each matching event.
strategy [required]Increments the metric by 1 for each matching event.
increment_by_one<strategy=increment_by_field>Strategy that increments a generated metric based on the value of an event field.
field [required]Name of the field containing the numeric value to increment the metric by.
strategy [required]Uses a numeric field in the log event as the metric increment.
increment_by_fieldtype [required]The processor type. Always generate_datadog_metrics.
generate_datadog_metricsgenerate_datadog_metrics<type=ocsf_mapper>The ocsf_mapper processor transforms logs into the OCSF schema using a predefined mapping configuration.
Supported pipeline types: logs
display_nameThe display name for a component.
enabled [required]Indicates whether the processor is enabled.
id [required]The unique identifier for this component. Used to reference this component in other parts of the pipeline.
include [required]A Datadog search query used to determine which logs this processor targets.
keep_unmatchedWhether to keep an event that does not match any of the mapping filters.
mappings [required]A list of mapping rules to convert events to the OCSF format.
include [required]A Datadog search query used to select the logs that this mapping should apply to.
mapping [required]Defines a single mapping rule for transforming logs into the OCSF schema.
Object 1Predefined library mappings for common log formats.
CloudTrail Account Change, GCP Cloud Audit CreateBucket, GCP Cloud Audit CreateSink, GCP Cloud Audit SetIamPolicy, GCP Cloud Audit UpdateSink, Github Audit Log API Activity, Google Workspace Admin Audit addPrivilege, Microsoft 365 Defender Incident, Microsoft 365 Defender UserLoggedIn, Okta System Log AuthenticationShow 1 more
Palo Alto Networks Firewall TrafficObject 2Custom OCSF mapping configuration for transforming logs.
mapping [required]A list of field mapping rules for transforming log fields to OCSF schema fields.
defaultThe default value to use if the source field is missing or empty.
dest [required]The destination OCSF field path.
lookupLookup table configuration for mapping source values to destination values.
sourceThe source field path from the log event.
sourcesMultiple source field paths for combined mapping.
valueA static value to use for the destination field.
metadata [required]Metadata for the custom OCSF mapping.
class [required]The OCSF event class name.
profilesA list of OCSF profiles to apply.
version [required]The OCSF schema version.
version [required]The version of the custom mapping configuration.
type [required]The processor type. The value should always be ocsf_mapper.
ocsf_mapperocsf_mapper<type=parse_grok>The parse_grok processor extracts structured fields from unstructured log messages using Grok patterns.
Supported pipeline types: logs
disable_library_rulesIf set to true, disables the default Grok rules provided by Datadog.
falsedisplay_nameThe display name for a component.
enabled [required]Indicates whether the processor is enabled.
fieldThe log field to parse with the Grok rules.
messageid [required]A unique identifier for this processor.
include [required]A Datadog search query used to determine which logs this processor targets.
rules [required]The list of Grok parsing rules selected by either source field or include query.
Object 1A Grok parsing rule used in the parse_grok processor. Each rule defines how to extract structured fields
from a specific log field using Grok patterns.
match_rules [required]A list of Grok parsing rules that define how to extract fields from the source field. Each rule must contain a name and a valid Grok pattern.
name [required]The name of the rule.
rule [required]The definition of the Grok rule.
source [required]The value of the source field in log events to be processed by the Grok rules.
support_rulesA list of Grok helper rules that can be referenced by the parsing rules.
name [required]The name of the Grok helper rule.
rule [required]The definition of the Grok helper rule.
Object 2A Grok parsing rule selected using the include query. Each rule defines how to extract structured fields
from logs matching a Datadog search query.
include [required]A Datadog search query used to determine which logs this Grok rule targets.
match_rules [required]A list of Grok parsing rules that define how to extract fields from matching logs. Each rule must contain a name and a valid Grok pattern.
name [required]The name of the rule.
rule [required]The definition of the Grok rule.
support_rulesA list of Grok helper rules that can be referenced by the parsing rules.
name [required]The name of the Grok helper rule.
rule [required]The definition of the Grok helper rule.
type [required]The processor type. The value should always be parse_grok.
parse_grokparse_grok<type=parse_json>The parse_json processor extracts JSON from a specified field and flattens it into the event. This is useful when logs contain embedded JSON as a string.
Supported pipeline types: logs
display_nameThe display name for a component.
enabled [required]Indicates whether the processor is enabled.
field [required]The name of the log field that contains a JSON string.
id [required]A unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
include [required]A Datadog search query used to determine which logs this processor targets.
type [required]The processor type. The value should always be parse_json.
parse_jsonparse_json<type=parse_xml>The parse_xml processor parses XML from a specified field and extracts it into the event.
Supported pipeline types: logs
always_use_text_keyWhether to always use a text key for element content.
attr_prefixThe prefix to use for XML attributes in the parsed output.
display_nameThe display name for a component.
enabled [required]Indicates whether the processor is enabled.
field [required]The name of the log field that contains an XML string.
id [required]The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the input to downstream components).
include [required]A Datadog search query used to determine which logs this processor targets.
include_attrWhether to include XML attributes in the parsed output.
parse_boolWhether to parse boolean values from strings.
parse_nullWhether to parse null values.
parse_numberWhether to parse numeric values from strings.
text_keyThe key name to use for text content within XML elements. Must be at least 1 character if specified.
type [required]The processor type. The value should always be parse_xml.
parse_xmlparse_xml<type=quota>The quota processor measures logging traffic for logs that match a specified filter. When the configured daily quota is met, the processor can drop or alert.
Supported pipeline types: logs
display_nameThe display name for a component.
drop_eventsIf set to true, logs that match the quota filter and are sent after the quota is exceeded are dropped. Logs that do not match the filter continue through the pipeline. Note: You can set either drop_events or overflow_action, but not both.
enabled [required]Indicates whether the processor is enabled.
id [required]The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the input to downstream components).
ignore_when_missing_partitionsIf true, the processor skips quota checks when partition fields are missing from the logs.
include [required]A Datadog search query used to determine which logs this processor targets.
limit [required]The maximum amount of data or number of events allowed before the quota is enforced. Can be specified in bytes or events.
enforce [required]Unit for quota enforcement in bytes for data size or events for count.
bytes, eventslimit [required]The limit for quota enforcement.
name [required]Name of the quota.
overflow_actionThe action to take when the quota or bucket limit is exceeded. Options:
drop: Drop the event.no_action: Let the event pass through.overflow_routing: Route to an overflow destination.
When overflow_routing is used, there must be a destination whose inputs reference this processor with the <processor-id>.overflow_events suffix to route the overflowing events. Only the following destination types support overflow inputs: amazon_s3_generic, amazon_s3, google_cloud_storage, and azure_storage.
drop, no_action, overflow_routingoverridesA list of alternate quota rules that apply to specific sets of events, identified by matching field values. Each override can define a custom limit.
fields [required]A list of field matchers used to apply a specific override. If an event matches all listed key-value pairs, the corresponding override limit is enforced.
name [required]The field name.
value [required]The field value.
limit [required]The maximum amount of data or number of events allowed before the quota is enforced. Can be specified in bytes or events.
enforce [required]Unit for quota enforcement in bytes for data size or events for count.
bytes, eventslimit [required]The limit for quota enforcement.
partition_fieldsA list of fields used to segment log traffic for quota enforcement. Quotas are tracked independently by unique combinations of these field values.
too_many_buckets_actionThe action to take when the quota or bucket limit is exceeded. Options:
drop: Drop the event.no_action: Let the event pass through.overflow_routing: Route to an overflow destination.
When overflow_routing is used, there must be a destination whose inputs reference this processor with the <processor-id>.overflow_events suffix to route the overflowing events. Only the following destination types support overflow inputs: amazon_s3_generic, amazon_s3, google_cloud_storage, and azure_storage.
drop, no_action, overflow_routingtype [required]The processor type. The value should always be quota.
quotaquota<type=reduce>The reduce processor aggregates and merges logs based on matching keys and merge strategies.
Supported pipeline types: logs
display_nameThe display name for a component.
enabled [required]Indicates whether the processor is enabled.
group_by [required]A list of fields used to group log events for merging.
id [required]The unique identifier for this processor.
include [required]A Datadog search query used to determine which logs this processor targets.
merge_strategies [required]List of merge strategies defining how values from grouped events should be combined.
path [required]The field path in the log event.
strategy [required]The merge strategy to apply.
discard, retain, sum, max, min, array, concat, concat_newline, concat_raw, shortest_arrayShow 2 more
longest_array, flat_uniquetype [required]The processor type. The value should always be reduce.
reducereduce<type=remove_fields>The remove_fields processor deletes specified fields from logs.
Supported pipeline types: logs
display_nameThe display name for a component.
enabled [required]Indicates whether the processor is enabled.
fields [required]A list of field names to be removed from each log event.
id [required]The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the input to downstream components).
include [required]A Datadog search query used to determine which logs this processor targets.
type [required]The processor type. The value should always be remove_fields.
remove_fieldsremove_fields<type=rename_fields>The rename_fields processor changes field names.
Supported pipeline types: logs
display_nameThe display name for a component.
enabled [required]Indicates whether the processor is enabled.
fields [required]A list of rename rules specifying which fields to rename in the event, what to rename them to, and whether to preserve the original fields.
destination [required]The field name to assign the renamed value to.
preserve_source [required]Indicates whether the original field, that is received from the source, should be kept (true) or removed (false) after renaming.
source [required]The original field name in the log event that should be renamed.
id [required]A unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
include [required]A Datadog search query used to determine which logs this processor targets.
type [required]The processor type. The value should always be rename_fields.
rename_fieldsrename_fields<type=split_array>The split_array processor splits array fields into separate events based on configured rules.
Supported pipeline types: logs
arrays [required]A list of array split configurations.
field [required]The path to the array field to split.
include [required]A Datadog search query used to determine which logs this array split operation targets.
display_nameThe display name for a component.
enabled [required]Indicates whether the processor is enabled.
id [required]The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the input to downstream components).
include [required]A Datadog search query used to determine which logs this processor targets. For split_array, this should typically be *.
type [required]The processor type. The value should always be split_array.
split_arraysplit_array<type=throttle>The throttle processor limits the number of events that pass through over a given time window.
Supported pipeline types: logs
display_nameThe display name for a component.
enabled [required]Indicates whether the processor is enabled.
group_byOptional list of fields used to group events before the threshold has been reached.
id [required]The unique identifier for this processor.
include [required]A Datadog search query used to determine which logs this processor targets.
threshold [required]the number of events allowed in a given time window. Events sent after the threshold has been reached, are dropped.
type [required]The processor type. The value should always be throttle.
throttlethrottlewindow [required]The time window in seconds over which the threshold applies.
<type=custom_processor>The custom_processor processor transforms events using Vector Remap Language (VRL) scripts with advanced filtering capabilities.
Supported pipeline types: logs, traces
display_nameThe display name for a component.
enabled [required]Indicates whether the processor is enabled.
id [required]The unique identifier for this processor.
include [required]A Datadog search query used to determine which events this processor targets. This field should always be set to * for the custom_processor processor.
*remaps [required]Array of VRL remap rules.
drop_on_error [required]Whether to drop events that caused errors during processing.
enabledWhether this remap rule is enabled.
include [required]A Datadog search query used to filter events for this specific remap rule.
name [required]A descriptive name for this remap rule.
source [required]The VRL script source code that defines the processing logic.
type [required]The processor type. The value should always be custom_processor.
custom_processorcustom_processor<type=generate_metrics>The generate_metrics processor creates custom metrics from logs or traces.
Metrics can be counters, gauges, or distributions and optionally grouped by event fields.
There must be a destination whose inputs reference this processor with the <processor-id>.metrics suffix to route the generated metrics. All destination types normally supported for metrics pipelines are also supported as metrics destinations in logs and traces pipelines.
Supported pipeline types: logs, traces
display_nameThe display name for a component.
enabled [required]Indicates whether the processor is enabled.
enabled_metric_aggregation[Recommended] Indicates whether generated metrics are automatically aggregated and enriched with the worker_uuid tag before being sent to the metric destination.
trueid [required]The unique identifier for this component. Used to reference this component in other parts of the pipeline.
includeA Datadog search query used to determine which events this processor targets.
metricsConfiguration for generating individual metrics.
group_byOptional fields used to group the metric series.
include [required]Datadog filter query to match events for metric generation.
metric_type [required]Type of metric to create.
count, gauge, distributionname [required]Name of the custom metric to be created.
value [required]Specifies how the value of the generated metric is computed.
<strategy=increment_by_one>Strategy that increments a generated metric by one for each matching event.
strategy [required]Increments the metric by 1 for each matching event.
increment_by_one<strategy=increment_by_field>Strategy that increments a generated metric based on the value of an event field.
field [required]Name of the field containing the numeric value to increment the metric by.
strategy [required]Uses a numeric field in the log event as the metric increment.
increment_by_fieldtype [required]The processor type. Always generate_metrics.
generate_metricsgenerate_metrics<type=sample>The sample processor allows probabilistic sampling of logs or traces at a fixed rate.
Supported pipeline types: logs, traces
display_nameThe display name for a component.
enabled [required]Indicates whether the processor is enabled.
group_byOptional list of fields to group events by. Each group is sampled independently.
id [required]The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the input to downstream components).
include [required]A Datadog search query used to determine which events this processor targets.
percentage [required]The percentage of events to sample.
type [required]The processor type. The value should always be sample.
samplesample<type=sensitive_data_scanner>The sensitive_data_scanner processor detects and optionally redacts sensitive data in log or trace events.
Supported pipeline types: logs, traces
display_nameThe display name for a component.
enabled [required]Indicates whether the processor is enabled.
id [required]The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the input to downstream components).
include [required]A Datadog search query used to determine which events this processor targets.
rules [required]A list of rules for identifying and acting on sensitive data patterns.
keyword_optionsConfiguration for keywords used to reinforce sensitive data pattern detection.
keywords [required]A list of keywords to match near the sensitive pattern.
proximity [required]Maximum number of tokens between a keyword and a sensitive value match.
name [required]A name identifying the rule.
on_match [required]Defines what action to take when sensitive data is matched.
<action=redact>Configuration for completely redacting matched sensitive data.
action [required]Action type that completely replaces the matched sensitive data with a fixed replacement string to remove all visibility.
redactoptions [required]Configuration for fully redacting sensitive data.
replace [required]The string used to replace matched sensitive data (for example, "***" or "[REDACTED]").
<action=hash>Configuration for hashing matched sensitive values.
action [required]Action type that replaces the matched sensitive data with a hashed representation, preserving structure while securing content.
hashoptionsOptional settings for the hash action. When omitted or empty, matched sensitive data is replaced with a deterministic hashed value that preserves structure for analytics while protecting the original content. Reserved for future hash configuration (for example, algorithm or salt).
<action=partial_redact>Configuration for partially redacting matched sensitive data.
action [required]Action type that redacts part of the sensitive data while preserving a configurable number of characters, typically used for masking purposes (e.g., show last 4 digits of a credit card).
partial_redactoptions [required]Controls how partial redaction is applied, including character count and direction.
characters [required]Number of characters to leave visible from the start or end of the matched value; the rest are redacted.
direction [required]Indicates whether to redact characters from the first or last part of the matched value.
first, lastpattern [required]Pattern detection configuration for identifying sensitive data using either a custom regex or a library reference.
<type=custom>Defines a custom regex-based pattern for identifying sensitive data in logs.
options [required]Options for defining a custom regex pattern.
descriptionHuman-readable description providing context about a sensitive data scanner rule
rule [required]A regular expression used to detect sensitive values. Must be a valid regex.
type [required]Indicates a custom regular expression is used for matching.
custom<type=library>Specifies a pattern from Datadog’s sensitive data detection library to match known sensitive data types.
options [required]Options for selecting a predefined library pattern and enabling keyword support.
descriptionHuman-readable description providing context about a sensitive data scanner rule
id [required]Identifier for a predefined pattern from the sensitive data scanner pattern library.
use_recommended_keywordsWhether to augment the pattern with recommended keywords (optional).
type [required]Indicates that a predefined library pattern is used.
libraryscope [required]Determines which parts of the log the pattern-matching rule should be applied to.
<target=include>Includes only specific fields for sensitive data scanning.
options [required]Fields to which the scope rule applies.
fields [required]List of log attribute names (field paths) to which the scope applies. Only these fields are included in or excluded from pattern matching.
target [required]Applies the rule only to included fields.
include<target=exclude>Excludes specific fields from sensitive data scanning.
options [required]Fields to which the scope rule applies.
fields [required]List of log attribute names (field paths) to which the scope applies. Only these fields are included in or excluded from pattern matching.
target [required]Excludes specific fields from processing.
exclude<target=all>Applies scanning across all available fields.
target [required]Applies the rule to all fields.
alltagsTags assigned to this rule for filtering and classification.
type [required]The processor type. The value should always be sensitive_data_scanner.
sensitive_data_scannersensitive_data_scanner<type=add_metric_tags>The add_metric_tags processor adds static tags to metrics.
Supported pipeline types: metrics
display_nameThe display name for a component.
enabled [required]Indicates whether the processor is enabled.
id [required]The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the input to downstream components).
include [required]A Datadog search query used to determine which metrics this processor targets.
tags [required]A list of static tags (key-value pairs) added to each metric processed by this component.
name [required]The field name.
value [required]The field value.
type [required]The processor type. The value must be add_metric_tags.
add_metric_tagsadd_metric_tags<type=aggregate>The aggregate processor combines metrics that share the same name and tags into a single metric over a configurable interval.
Supported pipeline types: metrics
aggregation_timingConfigures how metrics are assigned to aggregation windows. When omitted, metrics are grouped using system time.
allowed_lateness_secsGrace period, in seconds, for late-arriving metrics when using event time. Defaults to 10 seconds when omitted.
type [required]Determines whether metrics are assigned to aggregation windows based on when they are processed or their timestamps.
system_time, event_timedisplay_nameThe display name for a component.
enabled [required]Indicates whether the processor is enabled.
id [required]The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the input to downstream components).
include [required]A Datadog search query used to determine which metrics this processor targets.
interval_secs [required]The interval, in seconds, over which metrics are aggregated.
mode [required]The aggregation mode applied to metrics that share the same name and tags within the interval.
auto, sum, latest, count, max, min, meantype [required]The processor type. The value must be aggregate.
aggregateaggregateObject 25The enrichment_table processor enriches metrics with tags from a static CSV file or a Datadog reference table.
It looks up a row using the metric name or a metric tag value. It then adds each column of the matching row as a
metric tag, overwriting any existing tag with the same key. Exactly one of file or reference_table must be
configured.
Supported pipeline types: metrics
Object 1An enrichment_table processor that enriches metrics using a static CSV file.
display_nameThe display name for a component.
enabled [required]Indicates whether the processor is enabled.
file [required]Defines a static enrichment table loaded from a CSV file for metric enrichment.
encoding [required]File encoding format.
delimiter [required]The encoding delimiter.
includes_headers [required]The encoding includes_headers.
type [required]Specifies the encoding format (e.g., CSV) used for enrichment tables.
csvkey [required]Defines how to map a metric lookup value to a CSV column during enrichment table lookups.
column [required]The CSV column name or index to match against the lookup value.
source [required]Specifies the source of the key value used for metric enrichment table lookups. The lookup key can be either the metric name or a metric tag.
<type=metric_name>Uses the metric name as the lookup key for enrichment table matching.
<type=tag>Uses a metric tag as the lookup key for enrichment table matching.
path [required]Path to the CSV file.
id [required]The unique identifier for this component. Used in other parts of the pipeline to reference this component
(for example, as the input to downstream components).
include [required]A Datadog search query used to determine which metrics this processor targets.
type [required]The processor type. The value should always be enrichment_table.
enrichment_tableenrichment_tableObject 2An enrichment_table processor that enriches metrics using a Datadog reference table.
display_nameThe display name for a component.
enabled [required]Indicates whether the processor is enabled.
id [required]The unique identifier for this component. Used in other parts of the pipeline to reference this component
(for example, as the input to downstream components).
include [required]A Datadog search query used to determine which metrics this processor targets.
reference_table [required]Uses a Datadog reference table to enrich metrics.
app_key_keyThe name of the environment variable or secret that holds the Datadog application key used to access the reference table.
columnsA list of column names to include from the reference table. If not provided, all columns are included.
key [required]Defines the metric lookup value used as the reference-table row ID.
source [required]Specifies the source of the key value used for metric enrichment table lookups. The lookup key can be either the metric name or a metric tag.
<type=metric_name>Uses the metric name as the lookup key for enrichment table matching.
<type=tag>Uses a metric tag as the lookup key for enrichment table matching.
table_id [required]The unique identifier of the reference table.
type [required]The processor type. The value should always be enrichment_table.
enrichment_tableenrichment_table<type=metric_tags>The metric_tags processor filters metrics based on their tags using Datadog tag key patterns.
Supported pipeline types: metrics
display_nameThe display name for a component.
enabled [required]Indicates whether the processor is enabled.
id [required]The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the input to downstream components).
include [required]A Datadog search query that determines which metrics the processor targets.
rules [required]A list of rules for filtering metric tags.
action [required]The action to take on tags with matching keys.
include, excludeinclude [required]A Datadog search query used to determine which metrics this rule targets.
keys [required]A list of tag keys to include or exclude.
mode [required]The processing mode for tag filtering.
filtertype [required]The processor type. The value should always be metric_tags.
metric_tagsmetric_tags<type=rename_metric_tags>The rename_metric_tags processor changes the keys of tags on metrics.
Supported pipeline types: metrics
display_nameThe display name for a component.
enabled [required]Indicates whether the processor is enabled.
id [required]The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the input to downstream components).
include [required]A Datadog search query used to determine which metrics this processor targets.
tags [required]A list of rename rules specifying which tag keys to rename on each metric.
rename_to [required]The new tag key to assign in place of the original.
tag [required]The original tag key on the metric event.
type [required]The processor type. The value must be rename_metric_tags.
rename_metric_tagsrename_metric_tags<type=tag_cardinality_limit>The tag_cardinality_limit processor caps the number of distinct tag value combinations on metrics, dropping tags or events once the limit is exceeded.
Supported pipeline types: metrics
display_nameThe display name for a component.
enabled [required]Indicates whether the processor is enabled.
id [required]The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the input to downstream components).
include [required]A Datadog search query used to determine which metrics this processor targets.
limit_exceeded_action [required]The action to take when the cardinality limit is exceeded.
drop_tag, drop_eventper_metric_limitsA list of per-metric cardinality overrides that take precedence over the default value_limit.
limit_exceeded_actionThe action to take when the cardinality limit is exceeded.
drop_tag, drop_eventmetric_name [required]The name of the metric this override applies to.
override_type [required]How the override is applied. limit_override enforces a custom limit; excluded omits the metric or tag from cardinality tracking.
limit_override, excludedper_tag_limitsA list of per-tag cardinality overrides that apply within this metric. Must be omitted when override_type is excluded.
override_type [required]How the override is applied. limit_override enforces a custom limit; excluded omits the metric or tag from cardinality tracking.
limit_override, excludedtag_key [required]The tag key this override applies to.
value_limitThe maximum number of distinct values allowed for this tag. Required when override_type is limit_override. Must be omitted when override_type is excluded.
value_limitThe maximum number of distinct tag value combinations allowed for this metric. Required when override_type is limit_override. Must be omitted when override_type is excluded.
tracking_mode [required]Controls whether the processor uses exact or probabilistic tag tracking.
mode [required]The cardinality tracking algorithm to use.
exact_fingerprint, probabilistictype [required]The processor type. The value must be tag_cardinality_limit.
tag_cardinality_limittag_cardinality_limitvalue_limit [required]The default maximum number of distinct tag value combinations allowed per metric.
processorsDEPRECATED A list of processor groups that transform or enrich log data.
Deprecated: This field is deprecated, you should now use the processor_groups field.
display_nameThe display name for a component.
enabled [required]Whether this processor group is enabled.
id [required]The unique identifier for the processor group.
include [required]Conditional expression for when this processor group should execute.
inputs [required]A list of IDs for components whose output is used as the input for this processor group.
processors [required]Processors applied sequentially within this group. Events flow through each processor in order.
<type=filter>The filter processor allows conditional processing of logs, metrics, or traces based on a Datadog search query. Events that match the include query are passed through; others are discarded.
Supported pipeline types: logs, metrics, traces
display_nameThe display name for a component.
enabled [required]Indicates whether the processor is enabled.
id [required]The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the input to downstream components).
include [required]A Datadog search query used to determine which events should pass through the filter. Events that match this query continue to downstream components; others are dropped.
type [required]The processor type. The value should always be filter.
filterfilter<type=add_env_vars>The add_env_vars processor adds environment variable values to log events.
Supported pipeline types: logs
display_nameThe display name for a component.
enabled [required]Indicates whether the processor is enabled.
id [required]The unique identifier for this component. Used to reference this processor in the pipeline.
include [required]A Datadog search query used to determine which logs this processor targets.
type [required]The processor type. The value should always be add_env_vars.
add_env_varsadd_env_varsvariables [required]A list of environment variable mappings to apply to log fields.
field [required]The target field in the log event.
name [required]The name of the environment variable to read.
<type=add_fields>The add_fields processor adds static key-value fields to logs.
Supported pipeline types: logs
display_nameThe display name for a component.
enabled [required]Indicates whether the processor is enabled.
fields [required]A list of static fields (key-value pairs) that is added to each log event processed by this component.
name [required]The field name.
value [required]The field value.
id [required]The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the input to downstream components).
include [required]A Datadog search query used to determine which logs this processor targets.
type [required]The processor type. The value should always be add_fields.
add_fieldsadd_fields<type=add_hostname>The add_hostname processor adds the hostname to log events.
Supported pipeline types: logs
display_nameThe display name for a component.
enabled [required]Indicates whether the processor is enabled.
id [required]The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the input to downstream components).
include [required]A Datadog search query used to determine which logs this processor targets.
type [required]The processor type. The value should always be add_hostname.
add_hostnameadd_hostname<type=datadog_tags>The datadog_tags processor includes or excludes specific Datadog tags in your logs.
Supported pipeline types: logs
action [required]The action to take on tags with matching keys.
include, excludedisplay_nameThe display name for a component.
enabled [required]Indicates whether the processor is enabled.
id [required]The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the input to downstream components).
include [required]A Datadog search query used to determine which logs this processor targets.
keys [required]A list of tag keys.
mode [required]The processing mode.
filtertype [required]The processor type. The value should always be datadog_tags.
datadog_tagsdatadog_tags<type=dedupe>The dedupe processor removes duplicate fields in log events.
Supported pipeline types: logs
cacheConfiguration for the cache used to detect duplicates.
num_events [required]The number of events to cache for duplicate detection.
display_nameThe display name for a component.
enabled [required]Indicates whether the processor is enabled.
fields [required]A list of log field paths to check for duplicates.
id [required]The unique identifier for this processor.
include [required]A Datadog search query used to determine which logs this processor targets.
mode [required]The deduplication mode to apply to the fields.
match, ignoretype [required]The processor type. The value should always be dedupe.
dedupededupe<type=enrichment_table>The enrichment_table processor enriches logs using a static CSV file, GeoIP database, or reference table. Exactly one of file, geoip, or reference_table must be configured.
Supported pipeline types: logs
display_nameThe display name for a component.
enabled [required]Indicates whether the processor is enabled.
fileDefines a static enrichment table loaded from a CSV file.
encoding [required]File encoding format.
delimiter [required]The encoding delimiter.
includes_headers [required]The encoding includes_headers.
type [required]Specifies the encoding format (e.g., CSV) used for enrichment tables.
csvkey [required]Key fields used to look up enrichment values.
column [required]The items column.
comparison [required]Defines how to compare key fields for enrichment table lookups.
equalsfield [required]Specifies the source of the key value used for enrichment table lookups.
Can be a plain field path string or an object specifying event, vrl, or secret.
Object 1A plain field path in the log event, used as the lookup key.
Object 2Looks up a value from a field path in the log event.
event [required]The path to the field in the log event to use as the lookup key.
Object 3Evaluates a VRL expression to produce the lookup key.
vrl [required]A VRL expression that returns the value to use as the lookup key.
Object 4Looks up a value stored as a pipeline secret.
secret [required]The name of the secret containing the lookup key value.
path [required]Path to the CSV file.
schema [required]Schema defining column names and their types.
column [required]The items column.
type [required]Declares allowed data types for enrichment table columns.
string, boolean, integer, float, date, timestampgeoipUses a GeoIP database to enrich logs based on an IP field.
key_field [required]Path to the IP field in the log.
locale [required]Locale used to resolve geographical names.
path [required]Path to the GeoIP database file.
id [required]The unique identifier for this processor.
include [required]A Datadog search query used to determine which logs this processor targets.
reference_tableUses a Datadog reference table to enrich logs.
app_key_keyName of the environment variable or secret that holds the Datadog application key used to access the reference table.
columnsList of column names to include from the reference table. If not provided, all columns are included.
key_field [required]Path to the field in the log event to match against the reference table.
table_id [required]The unique identifier of the reference table.
target [required]Path where enrichment results should be stored in the log.
type [required]The processor type. The value should always be enrichment_table.
enrichment_tableenrichment_table<type=generate_datadog_metrics>DEPRECATED The generate_datadog_metrics processor creates custom metrics from logs and sends them to Datadog.
Metrics can be counters, gauges, or distributions and optionally grouped by log fields.
Supported pipeline types: logs
Deprecated: This processor is deprecated. Use the generate_metrics processor instead.
display_nameThe display name for a component.
enabled [required]Indicates whether the processor is enabled.
id [required]The unique identifier for this component. Used to reference this component in other parts of the pipeline.
includeA Datadog search query used to determine which logs this processor targets.
metricsConfiguration for generating individual metrics.
group_byOptional fields used to group the metric series.
include [required]Datadog filter query to match events for metric generation.
metric_type [required]Type of metric to create.
count, gauge, distributionname [required]Name of the custom metric to be created.
value [required]Specifies how the value of the generated metric is computed.
<strategy=increment_by_one>Strategy that increments a generated metric by one for each matching event.
strategy [required]Increments the metric by 1 for each matching event.
increment_by_one<strategy=increment_by_field>Strategy that increments a generated metric based on the value of an event field.
field [required]Name of the field containing the numeric value to increment the metric by.
strategy [required]Uses a numeric field in the log event as the metric increment.
increment_by_fieldtype [required]The processor type. Always generate_datadog_metrics.
generate_datadog_metricsgenerate_datadog_metrics<type=ocsf_mapper>The ocsf_mapper processor transforms logs into the OCSF schema using a predefined mapping configuration.
Supported pipeline types: logs
display_nameThe display name for a component.
enabled [required]Indicates whether the processor is enabled.
id [required]The unique identifier for this component. Used to reference this component in other parts of the pipeline.
include [required]A Datadog search query used to determine which logs this processor targets.
keep_unmatchedWhether to keep an event that does not match any of the mapping filters.
mappings [required]A list of mapping rules to convert events to the OCSF format.
include [required]A Datadog search query used to select the logs that this mapping should apply to.
mapping [required]Defines a single mapping rule for transforming logs into the OCSF schema.
Object 1Predefined library mappings for common log formats.
CloudTrail Account Change, GCP Cloud Audit CreateBucket, GCP Cloud Audit CreateSink, GCP Cloud Audit SetIamPolicy, GCP Cloud Audit UpdateSink, Github Audit Log API Activity, Google Workspace Admin Audit addPrivilege, Microsoft 365 Defender Incident, Microsoft 365 Defender UserLoggedIn, Okta System Log AuthenticationShow 1 more
Palo Alto Networks Firewall TrafficObject 2Custom OCSF mapping configuration for transforming logs.
mapping [required]A list of field mapping rules for transforming log fields to OCSF schema fields.
defaultThe default value to use if the source field is missing or empty.
dest [required]The destination OCSF field path.
lookupLookup table configuration for mapping source values to destination values.
sourceThe source field path from the log event.
sourcesMultiple source field paths for combined mapping.
valueA static value to use for the destination field.
metadata [required]Metadata for the custom OCSF mapping.
class [required]The OCSF event class name.
profilesA list of OCSF profiles to apply.
version [required]The OCSF schema version.
version [required]The version of the custom mapping configuration.
type [required]The processor type. The value should always be ocsf_mapper.
ocsf_mapperocsf_mapper<type=parse_grok>The parse_grok processor extracts structured fields from unstructured log messages using Grok patterns.
Supported pipeline types: logs
disable_library_rulesIf set to true, disables the default Grok rules provided by Datadog.
falsedisplay_nameThe display name for a component.
enabled [required]Indicates whether the processor is enabled.
fieldThe log field to parse with the Grok rules.
messageid [required]A unique identifier for this processor.
include [required]A Datadog search query used to determine which logs this processor targets.
rules [required]The list of Grok parsing rules selected by either source field or include query.
Object 1A Grok parsing rule used in the parse_grok processor. Each rule defines how to extract structured fields
from a specific log field using Grok patterns.
match_rules [required]A list of Grok parsing rules that define how to extract fields from the source field. Each rule must contain a name and a valid Grok pattern.
name [required]The name of the rule.
rule [required]The definition of the Grok rule.
source [required]The value of the source field in log events to be processed by the Grok rules.
support_rulesA list of Grok helper rules that can be referenced by the parsing rules.
name [required]The name of the Grok helper rule.
rule [required]The definition of the Grok helper rule.
Object 2A Grok parsing rule selected using the include query. Each rule defines how to extract structured fields
from logs matching a Datadog search query.
include [required]A Datadog search query used to determine which logs this Grok rule targets.
match_rules [required]A list of Grok parsing rules that define how to extract fields from matching logs. Each rule must contain a name and a valid Grok pattern.
name [required]The name of the rule.
rule [required]The definition of the Grok rule.
support_rulesA list of Grok helper rules that can be referenced by the parsing rules.
name [required]The name of the Grok helper rule.
rule [required]The definition of the Grok helper rule.
type [required]The processor type. The value should always be parse_grok.
parse_grokparse_grok<type=parse_json>The parse_json processor extracts JSON from a specified field and flattens it into the event. This is useful when logs contain embedded JSON as a string.
Supported pipeline types: logs
display_nameThe display name for a component.
enabled [required]Indicates whether the processor is enabled.
field [required]The name of the log field that contains a JSON string.
id [required]A unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
include [required]A Datadog search query used to determine which logs this processor targets.
type [required]The processor type. The value should always be parse_json.
parse_jsonparse_json<type=parse_xml>The parse_xml processor parses XML from a specified field and extracts it into the event.
Supported pipeline types: logs
always_use_text_keyWhether to always use a text key for element content.
attr_prefixThe prefix to use for XML attributes in the parsed output.
display_nameThe display name for a component.
enabled [required]Indicates whether the processor is enabled.
field [required]The name of the log field that contains an XML string.
id [required]The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the input to downstream components).
include [required]A Datadog search query used to determine which logs this processor targets.
include_attrWhether to include XML attributes in the parsed output.
parse_boolWhether to parse boolean values from strings.
parse_nullWhether to parse null values.
parse_numberWhether to parse numeric values from strings.
text_keyThe key name to use for text content within XML elements. Must be at least 1 character if specified.
type [required]The processor type. The value should always be parse_xml.
parse_xmlparse_xml<type=quota>The quota processor measures logging traffic for logs that match a specified filter. When the configured daily quota is met, the processor can drop or alert.
Supported pipeline types: logs
display_nameThe display name for a component.
drop_eventsIf set to true, logs that match the quota filter and are sent after the quota is exceeded are dropped. Logs that do not match the filter continue through the pipeline. Note: You can set either drop_events or overflow_action, but not both.
enabled [required]Indicates whether the processor is enabled.
id [required]The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the input to downstream components).
ignore_when_missing_partitionsIf true, the processor skips quota checks when partition fields are missing from the logs.
include [required]A Datadog search query used to determine which logs this processor targets.
limit [required]The maximum amount of data or number of events allowed before the quota is enforced. Can be specified in bytes or events.
enforce [required]Unit for quota enforcement in bytes for data size or events for count.
bytes, eventslimit [required]The limit for quota enforcement.
name [required]Name of the quota.
overflow_actionThe action to take when the quota or bucket limit is exceeded. Options:
drop: Drop the event.no_action: Let the event pass through.overflow_routing: Route to an overflow destination.
When overflow_routing is used, there must be a destination whose inputs reference this processor with the <processor-id>.overflow_events suffix to route the overflowing events. Only the following destination types support overflow inputs: amazon_s3_generic, amazon_s3, google_cloud_storage, and azure_storage.
drop, no_action, overflow_routingoverridesA list of alternate quota rules that apply to specific sets of events, identified by matching field values. Each override can define a custom limit.
fields [required]A list of field matchers used to apply a specific override. If an event matches all listed key-value pairs, the corresponding override limit is enforced.
name [required]The field name.
value [required]The field value.
limit [required]The maximum amount of data or number of events allowed before the quota is enforced. Can be specified in bytes or events.
enforce [required]Unit for quota enforcement in bytes for data size or events for count.
bytes, eventslimit [required]The limit for quota enforcement.
partition_fieldsA list of fields used to segment log traffic for quota enforcement. Quotas are tracked independently by unique combinations of these field values.
too_many_buckets_actionThe action to take when the quota or bucket limit is exceeded. Options:
drop: Drop the event.no_action: Let the event pass through.overflow_routing: Route to an overflow destination.
When overflow_routing is used, there must be a destination whose inputs reference this processor with the <processor-id>.overflow_events suffix to route the overflowing events. Only the following destination types support overflow inputs: amazon_s3_generic, amazon_s3, google_cloud_storage, and azure_storage.
drop, no_action, overflow_routingtype [required]The processor type. The value should always be quota.
quotaquota<type=reduce>The reduce processor aggregates and merges logs based on matching keys and merge strategies.
Supported pipeline types: logs
display_nameThe display name for a component.
enabled [required]Indicates whether the processor is enabled.
group_by [required]A list of fields used to group log events for merging.
id [required]The unique identifier for this processor.
include [required]A Datadog search query used to determine which logs this processor targets.
merge_strategies [required]List of merge strategies defining how values from grouped events should be combined.
path [required]The field path in the log event.
strategy [required]The merge strategy to apply.
discard, retain, sum, max, min, array, concat, concat_newline, concat_raw, shortest_arrayShow 2 more
longest_array, flat_uniquetype [required]The processor type. The value should always be reduce.
reducereduce<type=remove_fields>The remove_fields processor deletes specified fields from logs.
Supported pipeline types: logs
display_nameThe display name for a component.
enabled [required]Indicates whether the processor is enabled.
fields [required]A list of field names to be removed from each log event.
id [required]The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the input to downstream components).
include [required]A Datadog search query used to determine which logs this processor targets.
type [required]The processor type. The value should always be remove_fields.
remove_fieldsremove_fields<type=rename_fields>The rename_fields processor changes field names.
Supported pipeline types: logs
display_nameThe display name for a component.
enabled [required]Indicates whether the processor is enabled.
fields [required]A list of rename rules specifying which fields to rename in the event, what to rename them to, and whether to preserve the original fields.
destination [required]The field name to assign the renamed value to.
preserve_source [required]Indicates whether the original field, that is received from the source, should be kept (true) or removed (false) after renaming.
source [required]The original field name in the log event that should be renamed.
id [required]A unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
include [required]A Datadog search query used to determine which logs this processor targets.
type [required]The processor type. The value should always be rename_fields.
rename_fieldsrename_fields<type=split_array>The split_array processor splits array fields into separate events based on configured rules.
Supported pipeline types: logs
arrays [required]A list of array split configurations.
field [required]The path to the array field to split.
include [required]A Datadog search query used to determine which logs this array split operation targets.
display_nameThe display name for a component.
enabled [required]Indicates whether the processor is enabled.
id [required]The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the input to downstream components).
include [required]A Datadog search query used to determine which logs this processor targets. For split_array, this should typically be *.
type [required]The processor type. The value should always be split_array.
split_arraysplit_array<type=throttle>The throttle processor limits the number of events that pass through over a given time window.
Supported pipeline types: logs
display_nameThe display name for a component.
enabled [required]Indicates whether the processor is enabled.
group_byOptional list of fields used to group events before the threshold has been reached.
id [required]The unique identifier for this processor.
include [required]A Datadog search query used to determine which logs this processor targets.
threshold [required]the number of events allowed in a given time window. Events sent after the threshold has been reached, are dropped.
type [required]The processor type. The value should always be throttle.
throttlethrottlewindow [required]The time window in seconds over which the threshold applies.
<type=custom_processor>The custom_processor processor transforms events using Vector Remap Language (VRL) scripts with advanced filtering capabilities.
Supported pipeline types: logs, traces
display_nameThe display name for a component.
enabled [required]Indicates whether the processor is enabled.
id [required]The unique identifier for this processor.
include [required]A Datadog search query used to determine which events this processor targets. This field should always be set to * for the custom_processor processor.
*remaps [required]Array of VRL remap rules.
drop_on_error [required]Whether to drop events that caused errors during processing.
enabledWhether this remap rule is enabled.
include [required]A Datadog search query used to filter events for this specific remap rule.
name [required]A descriptive name for this remap rule.
source [required]The VRL script source code that defines the processing logic.
type [required]The processor type. The value should always be custom_processor.
custom_processorcustom_processor<type=generate_metrics>The generate_metrics processor creates custom metrics from logs or traces.
Metrics can be counters, gauges, or distributions and optionally grouped by event fields.
There must be a destination whose inputs reference this processor with the <processor-id>.metrics suffix to route the generated metrics. All destination types normally supported for metrics pipelines are also supported as metrics destinations in logs and traces pipelines.
Supported pipeline types: logs, traces
display_nameThe display name for a component.
enabled [required]Indicates whether the processor is enabled.
enabled_metric_aggregation[Recommended] Indicates whether generated metrics are automatically aggregated and enriched with the worker_uuid tag before being sent to the metric destination.
trueid [required]The unique identifier for this component. Used to reference this component in other parts of the pipeline.
includeA Datadog search query used to determine which events this processor targets.
metricsConfiguration for generating individual metrics.
group_byOptional fields used to group the metric series.
include [required]Datadog filter query to match events for metric generation.
metric_type [required]Type of metric to create.
count, gauge, distributionname [required]Name of the custom metric to be created.
value [required]Specifies how the value of the generated metric is computed.
<strategy=increment_by_one>Strategy that increments a generated metric by one for each matching event.
strategy [required]Increments the metric by 1 for each matching event.
increment_by_one<strategy=increment_by_field>Strategy that increments a generated metric based on the value of an event field.
field [required]Name of the field containing the numeric value to increment the metric by.
strategy [required]Uses a numeric field in the log event as the metric increment.
increment_by_fieldtype [required]The processor type. Always generate_metrics.
generate_metricsgenerate_metrics<type=sample>The sample processor allows probabilistic sampling of logs or traces at a fixed rate.
Supported pipeline types: logs, traces
display_nameThe display name for a component.
enabled [required]Indicates whether the processor is enabled.
group_byOptional list of fields to group events by. Each group is sampled independently.
id [required]The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the input to downstream components).
include [required]A Datadog search query used to determine which events this processor targets.
percentage [required]The percentage of events to sample.
type [required]The processor type. The value should always be sample.
samplesample<type=sensitive_data_scanner>The sensitive_data_scanner processor detects and optionally redacts sensitive data in log or trace events.
Supported pipeline types: logs, traces
display_nameThe display name for a component.
enabled [required]Indicates whether the processor is enabled.
id [required]The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the input to downstream components).
include [required]A Datadog search query used to determine which events this processor targets.
rules [required]A list of rules for identifying and acting on sensitive data patterns.
keyword_optionsConfiguration for keywords used to reinforce sensitive data pattern detection.
keywords [required]A list of keywords to match near the sensitive pattern.
proximity [required]Maximum number of tokens between a keyword and a sensitive value match.
name [required]A name identifying the rule.
on_match [required]Defines what action to take when sensitive data is matched.
<action=redact>Configuration for completely redacting matched sensitive data.
action [required]Action type that completely replaces the matched sensitive data with a fixed replacement string to remove all visibility.
redactoptions [required]Configuration for fully redacting sensitive data.
replace [required]The string used to replace matched sensitive data (for example, "***" or "[REDACTED]").
<action=hash>Configuration for hashing matched sensitive values.
action [required]Action type that replaces the matched sensitive data with a hashed representation, preserving structure while securing content.
hashoptionsOptional settings for the hash action. When omitted or empty, matched sensitive data is replaced with a deterministic hashed value that preserves structure for analytics while protecting the original content. Reserved for future hash configuration (for example, algorithm or salt).
<action=partial_redact>Configuration for partially redacting matched sensitive data.
action [required]Action type that redacts part of the sensitive data while preserving a configurable number of characters, typically used for masking purposes (e.g., show last 4 digits of a credit card).
partial_redactoptions [required]Controls how partial redaction is applied, including character count and direction.
characters [required]Number of characters to leave visible from the start or end of the matched value; the rest are redacted.
direction [required]Indicates whether to redact characters from the first or last part of the matched value.
first, lastpattern [required]Pattern detection configuration for identifying sensitive data using either a custom regex or a library reference.
<type=custom>Defines a custom regex-based pattern for identifying sensitive data in logs.
options [required]Options for defining a custom regex pattern.
descriptionHuman-readable description providing context about a sensitive data scanner rule
rule [required]A regular expression used to detect sensitive values. Must be a valid regex.
type [required]Indicates a custom regular expression is used for matching.
custom<type=library>Specifies a pattern from Datadog’s sensitive data detection library to match known sensitive data types.
options [required]Options for selecting a predefined library pattern and enabling keyword support.
descriptionHuman-readable description providing context about a sensitive data scanner rule
id [required]Identifier for a predefined pattern from the sensitive data scanner pattern library.
use_recommended_keywordsWhether to augment the pattern with recommended keywords (optional).
type [required]Indicates that a predefined library pattern is used.
libraryscope [required]Determines which parts of the log the pattern-matching rule should be applied to.
<target=include>Includes only specific fields for sensitive data scanning.
options [required]Fields to which the scope rule applies.
fields [required]List of log attribute names (field paths) to which the scope applies. Only these fields are included in or excluded from pattern matching.
target [required]Applies the rule only to included fields.
include<target=exclude>Excludes specific fields from sensitive data scanning.
options [required]Fields to which the scope rule applies.
fields [required]List of log attribute names (field paths) to which the scope applies. Only these fields are included in or excluded from pattern matching.
target [required]Excludes specific fields from processing.
exclude<target=all>Applies scanning across all available fields.
target [required]Applies the rule to all fields.
alltagsTags assigned to this rule for filtering and classification.
type [required]The processor type. The value should always be sensitive_data_scanner.
sensitive_data_scannersensitive_data_scanner<type=add_metric_tags>The add_metric_tags processor adds static tags to metrics.
Supported pipeline types: metrics
display_nameThe display name for a component.
enabled [required]Indicates whether the processor is enabled.
id [required]The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the input to downstream components).
include [required]A Datadog search query used to determine which metrics this processor targets.
tags [required]A list of static tags (key-value pairs) added to each metric processed by this component.
name [required]The field name.
value [required]The field value.
type [required]The processor type. The value must be add_metric_tags.
add_metric_tagsadd_metric_tags<type=aggregate>The aggregate processor combines metrics that share the same name and tags into a single metric over a configurable interval.
Supported pipeline types: metrics
aggregation_timingConfigures how metrics are assigned to aggregation windows. When omitted, metrics are grouped using system time.
allowed_lateness_secsGrace period, in seconds, for late-arriving metrics when using event time. Defaults to 10 seconds when omitted.
type [required]Determines whether metrics are assigned to aggregation windows based on when they are processed or their timestamps.
system_time, event_timedisplay_nameThe display name for a component.
enabled [required]Indicates whether the processor is enabled.
id [required]The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the input to downstream components).
include [required]A Datadog search query used to determine which metrics this processor targets.
interval_secs [required]The interval, in seconds, over which metrics are aggregated.
mode [required]The aggregation mode applied to metrics that share the same name and tags within the interval.
auto, sum, latest, count, max, min, meantype [required]The processor type. The value must be aggregate.
aggregateaggregateObject 25The enrichment_table processor enriches metrics with tags from a static CSV file or a Datadog reference table.
It looks up a row using the metric name or a metric tag value. It then adds each column of the matching row as a
metric tag, overwriting any existing tag with the same key. Exactly one of file or reference_table must be
configured.
Supported pipeline types: metrics
Object 1An enrichment_table processor that enriches metrics using a static CSV file.
display_nameThe display name for a component.
enabled [required]Indicates whether the processor is enabled.
file [required]Defines a static enrichment table loaded from a CSV file for metric enrichment.
encoding [required]File encoding format.
delimiter [required]The encoding delimiter.
includes_headers [required]The encoding includes_headers.
type [required]Specifies the encoding format (e.g., CSV) used for enrichment tables.
csvkey [required]Defines how to map a metric lookup value to a CSV column during enrichment table lookups.
column [required]The CSV column name or index to match against the lookup value.
source [required]Specifies the source of the key value used for metric enrichment table lookups. The lookup key can be either the metric name or a metric tag.
<type=metric_name>Uses the metric name as the lookup key for enrichment table matching.
<type=tag>Uses a metric tag as the lookup key for enrichment table matching.
path [required]Path to the CSV file.
id [required]The unique identifier for this component. Used in other parts of the pipeline to reference this component
(for example, as the input to downstream components).
include [required]A Datadog search query used to determine which metrics this processor targets.
type [required]The processor type. The value should always be enrichment_table.
enrichment_tableenrichment_tableObject 2An enrichment_table processor that enriches metrics using a Datadog reference table.
display_nameThe display name for a component.
enabled [required]Indicates whether the processor is enabled.
id [required]The unique identifier for this component. Used in other parts of the pipeline to reference this component
(for example, as the input to downstream components).
include [required]A Datadog search query used to determine which metrics this processor targets.
reference_table [required]Uses a Datadog reference table to enrich metrics.
app_key_keyThe name of the environment variable or secret that holds the Datadog application key used to access the reference table.
columnsA list of column names to include from the reference table. If not provided, all columns are included.
key [required]Defines the metric lookup value used as the reference-table row ID.
source [required]Specifies the source of the key value used for metric enrichment table lookups. The lookup key can be either the metric name or a metric tag.
<type=metric_name>Uses the metric name as the lookup key for enrichment table matching.
<type=tag>Uses a metric tag as the lookup key for enrichment table matching.
table_id [required]The unique identifier of the reference table.
type [required]The processor type. The value should always be enrichment_table.
enrichment_tableenrichment_table<type=metric_tags>The metric_tags processor filters metrics based on their tags using Datadog tag key patterns.
Supported pipeline types: metrics
display_nameThe display name for a component.
enabled [required]Indicates whether the processor is enabled.
id [required]The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the input to downstream components).
include [required]A Datadog search query that determines which metrics the processor targets.
rules [required]A list of rules for filtering metric tags.
action [required]The action to take on tags with matching keys.
include, excludeinclude [required]A Datadog search query used to determine which metrics this rule targets.
keys [required]A list of tag keys to include or exclude.
mode [required]The processing mode for tag filtering.
filtertype [required]The processor type. The value should always be metric_tags.
metric_tagsmetric_tags<type=rename_metric_tags>The rename_metric_tags processor changes the keys of tags on metrics.
Supported pipeline types: metrics
display_nameThe display name for a component.
enabled [required]Indicates whether the processor is enabled.
id [required]The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the input to downstream components).
include [required]A Datadog search query used to determine which metrics this processor targets.
tags [required]A list of rename rules specifying which tag keys to rename on each metric.
rename_to [required]The new tag key to assign in place of the original.
tag [required]The original tag key on the metric event.
type [required]The processor type. The value must be rename_metric_tags.
rename_metric_tagsrename_metric_tags<type=tag_cardinality_limit>The tag_cardinality_limit processor caps the number of distinct tag value combinations on metrics, dropping tags or events once the limit is exceeded.
Supported pipeline types: metrics
display_nameThe display name for a component.
enabled [required]Indicates whether the processor is enabled.
id [required]The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the input to downstream components).
include [required]A Datadog search query used to determine which metrics this processor targets.
limit_exceeded_action [required]The action to take when the cardinality limit is exceeded.
drop_tag, drop_eventper_metric_limitsA list of per-metric cardinality overrides that take precedence over the default value_limit.
limit_exceeded_actionThe action to take when the cardinality limit is exceeded.
drop_tag, drop_eventmetric_name [required]The name of the metric this override applies to.
override_type [required]How the override is applied. limit_override enforces a custom limit; excluded omits the metric or tag from cardinality tracking.
limit_override, excludedper_tag_limitsA list of per-tag cardinality overrides that apply within this metric. Must be omitted when override_type is excluded.
override_type [required]How the override is applied. limit_override enforces a custom limit; excluded omits the metric or tag from cardinality tracking.
limit_override, excludedtag_key [required]The tag key this override applies to.
value_limitThe maximum number of distinct values allowed for this tag. Required when override_type is limit_override. Must be omitted when override_type is excluded.
value_limitThe maximum number of distinct tag value combinations allowed for this metric. Required when override_type is limit_override. Must be omitted when override_type is excluded.
tracking_mode [required]Controls whether the processor uses exact or probabilistic tag tracking.
mode [required]The cardinality tracking algorithm to use.
exact_fingerprint, probabilistictype [required]The processor type. The value must be tag_cardinality_limit.
tag_cardinality_limittag_cardinality_limitvalue_limit [required]The default maximum number of distinct tag value combinations allowed per metric.
sources [required]A list of configured data sources for the pipeline.
<type=datadog_agent>The datadog_agent source collects logs, metrics, or traces from the Datadog Agent.
Supported pipeline types: logs, metrics, traces
address_keyName of the environment variable or secret that holds the listen address for the Datadog Agent source.
id [required]The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the input to downstream components).
tlsConfiguration for enabling TLS encryption between the pipeline component and external services.
ca_filePath to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_filePath to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
key_pass_keyName of the environment variable or secret that holds the passphrase for the private key file.
type [required]The source type. The value should always be datadog_agent.
datadog_agentdatadog_agent<type=opentelemetry>The opentelemetry source receives telemetry data using the OpenTelemetry Protocol (OTLP) over gRPC and HTTP.
Supported pipeline types: logs, metrics, traces
grpc_address_keyEnvironment variable name containing the gRPC server address for receiving OTLP data. Must be a valid environment variable name (alphanumeric characters and underscores only).
http_address_keyEnvironment variable name containing the HTTP server address for receiving OTLP data. Must be a valid environment variable name (alphanumeric characters and underscores only).
id [required]The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the input to downstream components).
tlsConfiguration for enabling TLS encryption between the pipeline component and external connecting clients.
ca_filePath to the Certificate Authority (CA) file used to validate connecting clients' TLS certificates.
crt_file [required]Path to the TLS server certificate file used to used to identify the pipeline component to connecting clients.
key_filePath to the private key file associated with the TLS server certificate.
key_pass_keyName of the environment variable or secret that holds the passphrase for the private key file.
verify_certificateWhen true, requires client connections to present a valid certificate, enabling mutual TLS authentication.
type [required]The source type. The value should always be opentelemetry.
opentelemetryopentelemetry<type=amazon_data_firehose>The amazon_data_firehose source ingests logs from AWS Data Firehose.
Supported pipeline types: logs
address_keyName of the environment variable or secret that holds the Firehose delivery stream address.
authAWS authentication credentials used for accessing AWS services such as S3. If omitted, the system’s default credentials are used (for example, the IAM role and environment variables).
assume_roleThe Amazon Resource Name (ARN) of the role to assume.
external_idA unique identifier for cross-account role assumption.
session_nameA session identifier used for logging and tracing the assumed role session.
id [required]The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the input to downstream components).
tlsConfiguration for enabling TLS encryption between the pipeline component and external services.
ca_filePath to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_filePath to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
key_pass_keyName of the environment variable or secret that holds the passphrase for the private key file.
type [required]The source type. The value should always be amazon_data_firehose.
amazon_data_firehoseamazon_data_firehose<type=amazon_s3>The amazon_s3 source ingests logs from an Amazon S3 bucket.
It supports AWS authentication, TLS encryption, and configurable compression.
Supported pipeline types: logs
authAWS authentication credentials used for accessing AWS services such as S3. If omitted, the system’s default credentials are used (for example, the IAM role and environment variables).
assume_roleThe Amazon Resource Name (ARN) of the role to assume.
external_idA unique identifier for cross-account role assumption.
session_nameA session identifier used for logging and tracing the assumed role session.
compressionCompression format for objects retrieved from the S3 bucket. Use auto to detect compression from the object's Content-Encoding header or file extension.
auto, none, gzip, zstdid [required]The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the input to downstream components).
region [required]AWS region where the S3 bucket resides.
tlsConfiguration for enabling TLS encryption between the pipeline component and external services.
ca_filePath to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_filePath to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
key_pass_keyName of the environment variable or secret that holds the passphrase for the private key file.
type [required]The source type. Always amazon_s3.
amazon_s3amazon_s3url_keyName of the environment variable or secret that holds the S3 bucket URL.
<type=fluent_bit>The fluent_bit source ingests logs from Fluent Bit.
Supported pipeline types: logs
address_keyName of the environment variable or secret that holds the listen address for the Fluent Bit receiver.
id [required]The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the input to downstream components).
tlsConfiguration for enabling TLS encryption between the pipeline component and external connecting clients.
ca_filePath to the Certificate Authority (CA) file used to validate connecting clients' TLS certificates.
crt_file [required]Path to the TLS server certificate file used to used to identify the pipeline component to connecting clients.
key_filePath to the private key file associated with the TLS server certificate.
key_pass_keyName of the environment variable or secret that holds the passphrase for the private key file.
verify_certificateWhen true, requires client connections to present a valid certificate, enabling mutual TLS authentication.
type [required]The source type. The value should always be fluent_bit.
fluent_bitfluent_bit<type=fluentd>The fluentd source ingests logs from a Fluentd-compatible service.
Supported pipeline types: logs
address_keyName of the environment variable or secret that holds the listen address for the Fluent receiver.
id [required]The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the input to downstream components).
tlsConfiguration for enabling TLS encryption between the pipeline component and external connecting clients.
ca_filePath to the Certificate Authority (CA) file used to validate connecting clients' TLS certificates.
crt_file [required]Path to the TLS server certificate file used to used to identify the pipeline component to connecting clients.
key_filePath to the private key file associated with the TLS server certificate.
key_pass_keyName of the environment variable or secret that holds the passphrase for the private key file.
verify_certificateWhen true, requires client connections to present a valid certificate, enabling mutual TLS authentication.
type [required]The source type. The value should always be `fluentd.
fluentdfluentd<type=google_pubsub>The google_pubsub source ingests logs from a Google Cloud Pub/Sub subscription.
Supported pipeline types: logs
authGoogle Cloud credentials used to authenticate with Google Cloud Storage.
credentials_file [required]Path to the Google Cloud service account key file.
decoding [required]The decoding format used to interpret incoming logs.
bytes, gelf, json, syslogid [required]The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the input to downstream components).
project [required]The Google Cloud project ID that owns the Pub/Sub subscription.
subscription [required]The Pub/Sub subscription name from which messages are consumed.
tlsConfiguration for enabling TLS encryption between the pipeline component and external services.
ca_filePath to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_filePath to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
key_pass_keyName of the environment variable or secret that holds the passphrase for the private key file.
type [required]The source type. The value should always be google_pubsub.
google_pubsubgoogle_pubsub<type=http_client>The http_client source scrapes logs from HTTP endpoints at regular intervals.
Supported pipeline types: logs
auth_strategyOptional authentication strategy for HTTP requests.
none, basic, bearer, customcustom_keyName of the environment variable or secret that holds a custom header value (used with custom auth strategies).
decoding [required]The decoding format used to interpret incoming logs.
bytes, gelf, json, syslogendpoint_url_keyName of the environment variable or secret that holds the HTTP endpoint URL to scrape.
id [required]The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the input to downstream components).
password_keyName of the environment variable or secret that holds the password (used when auth_strategy is basic).
scrape_interval_secsThe interval (in seconds) between HTTP scrape requests.
scrape_timeout_secsThe timeout (in seconds) for each scrape request.
tlsConfiguration for enabling TLS encryption between the pipeline component and external services.
ca_filePath to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_filePath to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
key_pass_keyName of the environment variable or secret that holds the passphrase for the private key file.
server_nameServer name to use for Server Name Indication (SNI) and to verify against the certificate presented by the remote host. Use this when the address you connect to doesn't match the certificate's Common Name or Subject Alternative Name.
token_keyName of the environment variable or secret that holds the bearer token (used when auth_strategy is bearer).
type [required]The source type. The value should always be http_client.
http_clienthttp_clientusername_keyName of the environment variable or secret that holds the username (used when auth_strategy is basic).
<type=http_server>The http_server source collects logs over HTTP POST from external services.
Supported pipeline types: logs
address_keyName of the environment variable or secret that holds the listen address for the HTTP server.
auth_strategy [required]HTTP authentication method.
none, plaincustom_keyName of the environment variable or secret that holds a custom header value (used with custom auth strategies).
decoding [required]The decoding format used to interpret incoming logs.
bytes, gelf, json, syslogid [required]Unique ID for the HTTP server source.
password_keyName of the environment variable or secret that holds the password (used when auth_strategy is plain).
tlsConfiguration for enabling TLS encryption between the pipeline component and external connecting clients.
ca_filePath to the Certificate Authority (CA) file used to validate connecting clients' TLS certificates.
crt_file [required]Path to the TLS server certificate file used to used to identify the pipeline component to connecting clients.
key_filePath to the private key file associated with the TLS server certificate.
key_pass_keyName of the environment variable or secret that holds the passphrase for the private key file.
verify_certificateWhen true, requires client connections to present a valid certificate, enabling mutual TLS authentication.
type [required]The source type. The value should always be http_server.
http_serverhttp_serverusername_keyName of the environment variable or secret that holds the username (used when auth_strategy is plain).
valid_tokensA list of tokens that are accepted for authenticating incoming HTTP requests. When set,
the source rejects any request whose token does not match an enabled entry in this list.
Cannot be combined with the plain auth strategy.
enabledIndicates whether this token is currently accepted. Disabled tokens are rejected without being removed from the configuration.
truefield_to_addAn optional metadata field that is attached to every event authenticated by the
associated token. Both key and value must match ^[A-Za-z0-9_]+$.
key [required]The metadata field name to add to incoming events.
value [required]The metadata field value to add to incoming events.
path_to_tokenSpecifies where the worker extracts the token from in the incoming HTTP request.
This can be either a built-in location (path or address) or an HTTP header object.
Object 1Built-in token location on the incoming HTTP request.
path, addressObject 2Extract the token from a specific HTTP request header.
header [required]The name of the HTTP header that carries the token.
token_key [required]Name of the environment variable or secret that holds the expected token value.
<type=kafka>The kafka source ingests data from Apache Kafka topics.
Supported pipeline types: logs
bootstrap_servers_keyName of the environment variable or secret that holds the Kafka bootstrap servers list.
group_id [required]Consumer group ID used by the Kafka client.
id [required]The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the input to downstream components).
librdkafka_optionsOptional list of advanced Kafka client configuration options, defined as key-value pairs.
name [required]The name of the librdkafka configuration option to set.
value [required]The value assigned to the specified librdkafka configuration option.
saslSpecifies the SASL mechanism for authenticating with a Kafka cluster.
mechanismSASL mechanism used for Kafka authentication.
PLAIN, SCRAM-SHA-256, SCRAM-SHA-512password_keyName of the environment variable or secret that holds the SASL password.
username_keyName of the environment variable or secret that holds the SASL username.
tlsConfiguration for enabling TLS encryption between the pipeline component and external services.
ca_filePath to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_filePath to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
key_pass_keyName of the environment variable or secret that holds the passphrase for the private key file.
topics [required]A list of Kafka topic names to subscribe to. The source ingests messages from each topic specified.
type [required]The source type. The value should always be kafka.
kafkakafka<type=logstash>The logstash source ingests logs from a Logstash forwarder.
Supported pipeline types: logs
address_keyName of the environment variable or secret that holds the listen address for the Logstash receiver.
id [required]The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the input to downstream components).
tlsConfiguration for enabling TLS encryption between the pipeline component and external connecting clients.
ca_filePath to the Certificate Authority (CA) file used to validate connecting clients' TLS certificates.
crt_file [required]Path to the TLS server certificate file used to used to identify the pipeline component to connecting clients.
key_filePath to the private key file associated with the TLS server certificate.
key_pass_keyName of the environment variable or secret that holds the passphrase for the private key file.
verify_certificateWhen true, requires client connections to present a valid certificate, enabling mutual TLS authentication.
type [required]The source type. The value should always be logstash.
logstashlogstash<type=rsyslog>The rsyslog source listens for logs over TCP or UDP from an rsyslog server using the syslog protocol.
Supported pipeline types: logs
address_keyName of the environment variable or secret that holds the listen address for the syslog receiver.
id [required]The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the input to downstream components).
mode [required]Protocol used by the syslog source to receive messages.
tcp, udptlsConfiguration for enabling TLS encryption between the pipeline component and external connecting clients.
ca_filePath to the Certificate Authority (CA) file used to validate connecting clients' TLS certificates.
crt_file [required]Path to the TLS server certificate file used to used to identify the pipeline component to connecting clients.
key_filePath to the private key file associated with the TLS server certificate.
key_pass_keyName of the environment variable or secret that holds the passphrase for the private key file.
verify_certificateWhen true, requires client connections to present a valid certificate, enabling mutual TLS authentication.
type [required]The source type. The value should always be rsyslog.
rsyslogrsyslog<type=socket>The socket source ingests logs over TCP or UDP.
Supported pipeline types: logs
address_keyName of the environment variable or secret that holds the listen address for the socket.
framing [required]Framing method configuration for the socket source.
<method=newline_delimited>Byte frames which are delimited by a newline character.
method [required]Byte frames which are delimited by a newline character.
newline_delimited<method=bytes>Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments).
method [required]Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments).
bytes<method=character_delimited>Byte frames which are delimited by a chosen character.
delimiter [required]A single ASCII character used to delimit events.
method [required]Byte frames which are delimited by a chosen character.
character_delimited<method=octet_counting>Byte frames according to the octet counting format as per RFC6587.
method [required]Byte frames according to the octet counting format as per RFC6587.
octet_counting<method=chunked_gelf>Byte frames which are chunked GELF messages.
method [required]Byte frames which are chunked GELF messages.
chunked_gelfid [required]The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the input to downstream components).
mode [required]Protocol used to receive logs.
tcp, udptlsTLS configuration. Relevant only when mode is tcp.
ca_filePath to the Certificate Authority (CA) file used to validate connecting clients' TLS certificates.
crt_file [required]Path to the TLS server certificate file used to used to identify the pipeline component to connecting clients.
key_filePath to the private key file associated with the TLS server certificate.
key_pass_keyName of the environment variable or secret that holds the passphrase for the private key file.
verify_certificateWhen true, requires client connections to present a valid certificate, enabling mutual TLS authentication.
type [required]The source type. The value should always be socket.
socketsocket<type=splunk_hec>The splunk_hec source implements the Splunk HTTP Event Collector (HEC) API.
Supported pipeline types: logs
address_keyName of the environment variable or secret that holds the listen address for the HEC API.
id [required]The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the input to downstream components).
store_hec_tokenWhen true, the Splunk HEC token from the incoming request is stored in the event metadata.
This allows downstream components to forward the token to other Splunk HEC destinations.
tlsConfiguration for enabling TLS encryption between the pipeline component and external connecting clients.
ca_filePath to the Certificate Authority (CA) file used to validate connecting clients' TLS certificates.
crt_file [required]Path to the TLS server certificate file used to used to identify the pipeline component to connecting clients.
key_filePath to the private key file associated with the TLS server certificate.
key_pass_keyName of the environment variable or secret that holds the passphrase for the private key file.
verify_certificateWhen true, requires client connections to present a valid certificate, enabling mutual TLS authentication.
type [required]The source type. Always splunk_hec.
splunk_hecsplunk_hecvalid_tokensA list of tokens that are accepted for authenticating incoming HEC requests. When set, the source rejects any request whose HEC token does not match an enabled entry in this list.
enabledIndicates whether this token is currently accepted. Disabled tokens are rejected without being removed from the configuration.
truefield_to_addAn optional metadata field that is attached to every event authenticated by the
associated token. Both key and value must match ^[A-Za-z0-9_]+$.
key [required]The metadata field name to add to incoming events.
value [required]The metadata field value to add to incoming events.
token_key [required]Name of the environment variable or secret that holds the expected HEC token value.
<type=splunk_tcp>The splunk_tcp source receives logs from a Splunk Universal Forwarder over TCP.
TLS is supported for secure transmission.
Supported pipeline types: logs
address_keyName of the environment variable or secret that holds the listen address for the Splunk TCP receiver.
id [required]The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the input to downstream components).
max_connection_duration_secsMaximum duration, in seconds, that a connection can remain open before it is closed. When unset, connections can remain open indefinitely.
tlsConfiguration for enabling TLS encryption between the pipeline component and external connecting clients.
ca_filePath to the Certificate Authority (CA) file used to validate connecting clients' TLS certificates.
crt_file [required]Path to the TLS server certificate file used to used to identify the pipeline component to connecting clients.
key_filePath to the private key file associated with the TLS server certificate.
key_pass_keyName of the environment variable or secret that holds the passphrase for the private key file.
verify_certificateWhen true, requires client connections to present a valid certificate, enabling mutual TLS authentication.
type [required]The source type. Always splunk_tcp.
splunk_tcpsplunk_tcp<type=sumo_logic>The sumo_logic source receives logs from Sumo Logic collectors.
Supported pipeline types: logs
address_keyName of the environment variable or secret that holds the listen address for the Sumo Logic receiver.
id [required]The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the input to downstream components).
type [required]The source type. The value should always be sumo_logic.
sumo_logicsumo_logic<type=syslog_ng>The syslog_ng source listens for logs over TCP or UDP from a syslog-ng server using the syslog protocol.
Supported pipeline types: logs
address_keyName of the environment variable or secret that holds the listen address for the syslog-ng receiver.
id [required]The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the input to downstream components).
mode [required]Protocol used by the syslog source to receive messages.
tcp, udptlsConfiguration for enabling TLS encryption between the pipeline component and external connecting clients.
ca_filePath to the Certificate Authority (CA) file used to validate connecting clients' TLS certificates.
crt_file [required]Path to the TLS server certificate file used to used to identify the pipeline component to connecting clients.
key_filePath to the private key file associated with the TLS server certificate.
key_pass_keyName of the environment variable or secret that holds the passphrase for the private key file.
verify_certificateWhen true, requires client connections to present a valid certificate, enabling mutual TLS authentication.
type [required]The source type. The value should always be syslog_ng.
syslog_ngsyslog_ng<type=websocket>The websocket source ingests logs from a WebSocket server using the ws:// or wss:// protocol.
Supported pipeline types: logs.
auth_strategy [required]Authentication strategy for the WebSocket source connection.
none, basic, bearer, customcustom_keyName of the environment variable or secret that holds the custom authorization header value. Used when auth_strategy is custom.
decoding [required]The decoding format used to interpret incoming logs.
bytes, gelf, json, syslogid [required]The unique identifier for this component.
password_keyName of the environment variable or secret that holds the password. Used when auth_strategy is basic.
tlsTLS configuration for the WebSocket source. Use enabled for standard wss:// connections, or with_client_cert to present a client certificate for mutual TLS.
<mode=enabled>TLS configuration that enables encryption without a client certificate. Use this for standard wss:// connections that do not require mutual TLS.
mode [required]TLS mode. Must be enabled.
enabled<mode=with_client_cert>TLS configuration that enables encryption and presents a client certificate for mutual TLS authentication.
ca_filePath to the Certificate Authority (CA) file used to validate the remote server's TLS certificate.
crt_file [required]Path to the TLS client certificate file used to identify this source to the remote server.
key_filePath to the private key file associated with the client certificate.
key_pass_keyName of the environment variable or secret that holds the passphrase for the private key file.
mode [required]TLS mode. Must be with_client_cert.
with_client_certtoken_keyName of the environment variable or secret that holds the bearer token. Used when auth_strategy is bearer.
type [required]The source type. The value should always be websocket.
websocketwebsocketuri_keyName of the environment variable or secret that holds the WebSocket server URI (ws:// or wss://).
username_keyName of the environment variable or secret that holds the username. Used when auth_strategy is basic.
<type=prometheus_remote_write>The prometheus_remote_write source ingests metrics pushed over the Prometheus Remote Write protocol.
Supported pipeline types: metrics
address_keyName of the environment variable or secret that holds the listen address for the Prometheus Remote Write endpoint.
auth_strategy [required]HTTP authentication method.
none, plainid [required]The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the input to downstream components).
password_keyName of the environment variable or secret that holds the password (used when auth_strategy is plain).
path [required]The HTTP path on which the source listens for incoming Prometheus Remote Write requests.
tlsConfiguration for enabling TLS encryption between the pipeline component and external connecting clients.
ca_filePath to the Certificate Authority (CA) file used to validate connecting clients' TLS certificates.
crt_file [required]Path to the TLS server certificate file used to used to identify the pipeline component to connecting clients.
key_filePath to the private key file associated with the TLS server certificate.
key_pass_keyName of the environment variable or secret that holds the passphrase for the private key file.
verify_certificateWhen true, requires client connections to present a valid certificate, enabling mutual TLS authentication.
type [required]The source type. The value should always be prometheus_remote_write.
prometheus_remote_writeprometheus_remote_writeusername_keyName of the environment variable or secret that holds the username (used when auth_strategy is plain).
valid_tokensA list of tokens that are accepted for authenticating incoming requests. When set, the source rejects any request whose token does not match an enabled entry in this list.
enabledIndicates whether this token is currently accepted. Disabled tokens are rejected without being removed from the configuration.
truepath_to_tokenSpecifies where the worker extracts the token from in the incoming HTTP request.
This can be either a built-in location (path or address) or an HTTP header object.
Object 1Built-in token location on the incoming HTTP request.
path, addressObject 2Extract the token from a specific HTTP request header.
header [required]The name of the HTTP header that carries the token.
token_key [required]Name of the environment variable or secret that holds the expected token value.
use_legacy_search_syntaxSet to true to continue using the legacy search syntax while migrating filter queries. After migrating all queries to the new syntax, set to false.
The legacy syntax is deprecated and will eventually be removed.
Requires Observability Pipelines Worker 2.11 or later.
Only applies to logs pipelines. This field is ignored for metrics and traces pipelines.
See Upgrade Your Filter Queries to the New Search Syntax for more information.
name [required]Name of the pipeline.
type [required]The resource type identifier. For pipeline resources, this should always be set to pipelines.
pipelinesCreate a new pipeline returns "OK" response
{
"data": {
"attributes": {
"config": {
"destinations": [
{
"id": "datadog-logs-destination",
"inputs": [
"my-processor-group"
],
"type": "datadog_logs"
}
],
"processor_groups": [
{
"enabled": true,
"id": "my-processor-group",
"include": "service:my-service",
"inputs": [
"datadog-agent-source"
],
"processors": [
{
"enabled": true,
"id": "filter-processor",
"include": "status:error",
"type": "filter"
}
]
}
],
"sources": [
{
"id": "datadog-agent-source",
"type": "datadog_agent"
}
]
},
"name": "Main Observability Pipeline"
},
"type": "pipelines"
}
}Create a pipeline with dedupe processor with cache returns "OK" response
{
"data": {
"attributes": {
"config": {
"destinations": [
{
"id": "datadog-logs-destination",
"inputs": [
"my-processor-group"
],
"type": "datadog_logs"
}
],
"processor_groups": [
{
"enabled": true,
"id": "my-processor-group",
"include": "service:my-service",
"inputs": [
"datadog-agent-source"
],
"processors": [
{
"enabled": true,
"id": "dedupe-processor",
"include": "service:my-service",
"type": "dedupe",
"fields": [
"message"
],
"mode": "match",
"cache": {
"num_events": 5000
}
}
]
}
],
"sources": [
{
"id": "datadog-agent-source",
"type": "datadog_agent"
}
]
},
"name": "Pipeline with Dedupe Cache"
},
"type": "pipelines"
}
}Create a pipeline with dedupe processor without cache returns "OK" response
{
"data": {
"attributes": {
"config": {
"destinations": [
{
"id": "datadog-logs-destination",
"inputs": [
"my-processor-group"
],
"type": "datadog_logs"
}
],
"processor_groups": [
{
"enabled": true,
"id": "my-processor-group",
"include": "service:my-service",
"inputs": [
"datadog-agent-source"
],
"processors": [
{
"enabled": true,
"id": "dedupe-processor",
"include": "service:my-service",
"type": "dedupe",
"fields": [
"message"
],
"mode": "match"
}
]
}
],
"sources": [
{
"id": "datadog-agent-source",
"type": "datadog_agent"
}
]
},
"name": "Pipeline with Dedupe No Cache"
},
"type": "pipelines"
}
}Réponse
OK
Top-level schema representing a pipeline.
data [required]Contains the pipeline’s ID, type, and configuration attributes.
attributes [required]Defines the pipeline’s name and its components (sources, processors, and destinations).
config [required]Specifies the pipeline's configuration, including its sources, processors, and destinations.
destinations [required]A list of destination components where processed logs are sent.
<type=elasticsearch>The elasticsearch destination writes logs or metrics to an Elasticsearch cluster.
Supported pipeline types: logs, metrics
api_versionThe Elasticsearch API version to use. Set to auto to auto-detect.
auto, v6, v7, v8authAuthentication settings for the Elasticsearch destination.
When strategy is basic, use username_key and password_key to reference credentials stored in environment variables or secrets.
password_keyName of the environment variable or secret that holds the Elasticsearch password (used when strategy is basic).
strategy [required]The authentication strategy to use.
basic, awsusername_keyName of the environment variable or secret that holds the Elasticsearch username (used when strategy is basic).
bufferConfiguration for buffer settings on destination components.
Object 1Options for configuring a disk buffer.
max_size [required]Maximum size of the disk buffer.
typeThe type of the buffer that will be configured, a disk buffer.
diskdiskwhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 2Options for configuring a memory buffer by byte size.
max_size [required]Maximum size of the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 3Options for configuring a memory buffer by queue length.
max_events [required]Maximum events for the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestbulk_indexThe name of the index to write events to in Elasticsearch.
compressionCompression configuration for the Elasticsearch destination.
algorithm [required]The compression algorithm applied when sending data to Elasticsearch.
none, gzip, zlib, zstd, snappylevelThe compression level. Only applicable for gzip, zlib, and zstd algorithms.
data_streamConfiguration options for writing to Elasticsearch Data Streams instead of a fixed index.
auto_routingWhen true, automatically routes events to the appropriate data stream based on the event content.
datasetThe data stream dataset. This groups events by their source or application.
dtypeThe data stream type. This determines how events are categorized within the data stream.
namespaceThe data stream namespace. This separates events into different environments or domains.
sync_fieldsWhen true, synchronizes data stream fields with the Elasticsearch index mapping.
endpoint_url_keyName of the environment variable or secret that holds the Elasticsearch endpoint URL.
id [required]The unique identifier for this component.
id_keyThe name of the field used as the document ID in Elasticsearch.
inputs [required]A list of component IDs whose output is used as the input for this component.
pipelineThe name of an Elasticsearch ingest pipeline to apply to events before indexing.
request_retry_partialWhen true, retries failed partial bulk requests when some events in a batch fail while others succeed.
tlsConfiguration for enabling TLS encryption between the pipeline component and external services.
ca_filePath to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_filePath to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
key_pass_keyName of the environment variable or secret that holds the passphrase for the private key file.
type [required]The destination type. The value should always be elasticsearch.
elasticsearchelasticsearch<type=http_client>The http_client destination sends data to an HTTP endpoint.
Supported pipeline types: logs, metrics
auth_strategyHTTP authentication strategy.
none, basic, bearer, custombufferConfiguration for buffer settings on destination components.
Object 1Options for configuring a disk buffer.
max_size [required]Maximum size of the disk buffer.
typeThe type of the buffer that will be configured, a disk buffer.
diskdiskwhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 2Options for configuring a memory buffer by byte size.
max_size [required]Maximum size of the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 3Options for configuring a memory buffer by queue length.
max_events [required]Maximum events for the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestcompressionCompression configuration for HTTP requests.
algorithm [required]Compression algorithm.
gzipcustom_keyName of the environment variable or secret that holds a custom header value (used with custom auth strategies).
encoding [required]Encoding format for log events.
jsonid [required]The unique identifier for this component.
inputs [required]A list of component IDs whose output is used as the input for this component.
password_keyName of the environment variable or secret that holds the password (used when auth_strategy is basic).
tlsConfiguration for enabling TLS encryption between the pipeline component and external services.
ca_filePath to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_filePath to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
key_pass_keyName of the environment variable or secret that holds the passphrase for the private key file.
server_nameServer name to use for Server Name Indication (SNI) and to verify against the certificate presented by the remote host. Use this when the address you connect to doesn't match the certificate's Common Name or Subject Alternative Name.
token_keyName of the environment variable or secret that holds the bearer token (used when auth_strategy is bearer).
type [required]The destination type. The value should always be http_client.
http_clienthttp_clienturi_keyName of the environment variable or secret that holds the HTTP endpoint URI.
username_keyName of the environment variable or secret that holds the username (used when auth_strategy is basic).
<type=amazon_opensearch>The amazon_opensearch destination writes logs to Amazon OpenSearch.
Supported pipeline types: logs
authAuthentication settings for the Amazon OpenSearch destination.
The strategy field determines whether basic or AWS-based authentication is used.
When strategy is basic, use username_key and password_key to reference credentials stored in environment variables or secrets.
assume_roleThe ARN of the role to assume (used with aws strategy).
aws_regionAWS region
external_idExternal ID for the assumed role (used with aws strategy).
password_keyName of the environment variable or secret that holds the Amazon OpenSearch password (used when strategy is basic).
session_nameSession name for the assumed role (used with aws strategy).
strategy [required]The authentication strategy to use.
basic, awsusername_keyName of the environment variable or secret that holds the Amazon OpenSearch username (used when strategy is basic).
bufferConfiguration for buffer settings on destination components.
Object 1Options for configuring a disk buffer.
max_size [required]Maximum size of the disk buffer.
typeThe type of the buffer that will be configured, a disk buffer.
diskdiskwhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 2Options for configuring a memory buffer by byte size.
max_size [required]Maximum size of the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 3Options for configuring a memory buffer by queue length.
max_events [required]Maximum events for the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestbulk_indexThe index to write logs to.
data_streamConfiguration options for writing to OpenSearch Data Streams instead of a fixed index.
datasetThe data stream dataset for your logs. This groups logs by their source or application.
dtypeThe data stream type for your logs. This determines how logs are categorized within the data stream.
namespaceThe data stream namespace for your logs. This separates logs into different environments or domains.
endpoint_url_keyName of the environment variable or secret that holds the Amazon OpenSearch endpoint URL.
id [required]The unique identifier for this component.
inputs [required]A list of component IDs whose output is used as the input for this component.
type [required]The destination type. The value should always be amazon_opensearch.
amazon_opensearchamazon_opensearch<type=amazon_s3>The amazon_s3 destination sends your logs in Datadog-rehydratable format to an Amazon S3 bucket for archiving.
Supported pipeline types: logs
authAWS authentication credentials used for accessing AWS services such as S3. If omitted, the system’s default credentials are used (for example, the IAM role and environment variables).
assume_roleThe Amazon Resource Name (ARN) of the role to assume.
external_idA unique identifier for cross-account role assumption.
session_nameA session identifier used for logging and tracing the assumed role session.
bucket [required]S3 bucket name.
bufferConfiguration for buffer settings on destination components.
Object 1Options for configuring a disk buffer.
max_size [required]Maximum size of the disk buffer.
typeThe type of the buffer that will be configured, a disk buffer.
diskdiskwhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 2Options for configuring a memory buffer by byte size.
max_size [required]Maximum size of the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 3Options for configuring a memory buffer by queue length.
max_events [required]Maximum events for the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestcompressionCompression configuration for archived logs. When omitted, logs are compressed with gzip for backward compatibility.
<algorithm=zstd>Zstd compression.
algorithm [required]The compression type. Always zstd.
zstdzstdlevel [required]Zstd compression level. Valid values range from 1 to 21.
<algorithm=gzip>Gzip compression.
algorithm [required]The compression type. Always gzip.
gzipgziplevel [required]Gzip compression level. Valid values range from 1 to 9.
id [required]Unique identifier for the destination component.
inputs [required]A list of component IDs whose output is used as the input for this component.
key_prefixOptional prefix for object keys.
region [required]AWS region of the S3 bucket.
server_side_encryptionServer-side encryption type for Amazon S3.
aws:kms, AES256ssekms_key_idThe AWS KMS key ID used for SSE-KMS encryption.
Only applies when server_side_encryption is set to aws:kms.
storage_class [required]S3 storage class.
STANDARD, REDUCED_REDUNDANCY, INTELLIGENT_TIERING, STANDARD_IA, EXPRESS_ONEZONE, ONEZONE_IA, GLACIER, GLACIER_IR, DEEP_ARCHIVEtlsConfiguration for enabling TLS encryption between the pipeline component and external services.
ca_filePath to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_filePath to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
key_pass_keyName of the environment variable or secret that holds the passphrase for the private key file.
type [required]The destination type. Always amazon_s3.
amazon_s3amazon_s3<type=amazon_s3_generic>The amazon_s3_generic destination sends your logs to an Amazon S3 bucket.
Supported pipeline types: logs
authAWS authentication credentials used for accessing AWS services such as S3. If omitted, the system’s default credentials are used (for example, the IAM role and environment variables).
assume_roleThe Amazon Resource Name (ARN) of the role to assume.
external_idA unique identifier for cross-account role assumption.
session_nameA session identifier used for logging and tracing the assumed role session.
batch_settingsEvent batching settings
batch_sizeMaximum batch size in bytes.
timeout_secsMaximum number of seconds to wait before flushing the batch.
bucket [required]S3 bucket name.
bufferConfiguration for buffer settings on destination components.
Object 1Options for configuring a disk buffer.
max_size [required]Maximum size of the disk buffer.
typeThe type of the buffer that will be configured, a disk buffer.
diskdiskwhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 2Options for configuring a memory buffer by byte size.
max_size [required]Maximum size of the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 3Options for configuring a memory buffer by queue length.
max_events [required]Maximum events for the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestcompression [required]Compression algorithm applied to encoded logs.
<algorithm=zstd>Zstd compression.
algorithm [required]The compression type. Always zstd.
zstdzstdlevel [required]Zstd compression level.
<algorithm=gzip>Gzip compression.
algorithm [required]The compression type. Always gzip.
gzipgziplevel [required]Gzip compression level.
<algorithm=snappy>Snappy compression.
algorithm [required]The compression type. Always snappy.
snappysnappyencoding [required]Encoding format for the destination.
<type=json>JSON encoding.
type [required]The encoding type. Always json.
jsonjson<type=parquet>Parquet encoding.
type [required]The encoding type. Always parquet.
parquetparquetid [required]Unique identifier for the destination component.
inputs [required]A list of component IDs whose output is used as the input for this component.
key_prefixOptional prefix for object keys.
region [required]AWS region of the S3 bucket.
server_side_encryptionServer-side encryption type for Amazon S3.
aws:kms, AES256ssekms_key_idThe AWS KMS key ID used for SSE-KMS encryption.
Only applies when server_side_encryption is set to aws:kms.
storage_class [required]S3 storage class.
STANDARD, REDUCED_REDUNDANCY, INTELLIGENT_TIERING, STANDARD_IA, EXPRESS_ONEZONE, ONEZONE_IA, GLACIER, GLACIER_IR, DEEP_ARCHIVEtype [required]The destination type. Always amazon_s3_generic.
amazon_s3_genericamazon_s3_generic<type=amazon_security_lake>The amazon_security_lake destination sends your logs to Amazon Security Lake.
Supported pipeline types: logs
authAWS authentication credentials used for accessing AWS services such as S3. If omitted, the system’s default credentials are used (for example, the IAM role and environment variables).
assume_roleThe Amazon Resource Name (ARN) of the role to assume.
external_idA unique identifier for cross-account role assumption.
session_nameA session identifier used for logging and tracing the assumed role session.
bucket [required]Name of the Amazon S3 bucket in Security Lake (3-63 characters).
bufferConfiguration for buffer settings on destination components.
Object 1Options for configuring a disk buffer.
max_size [required]Maximum size of the disk buffer.
typeThe type of the buffer that will be configured, a disk buffer.
diskdiskwhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 2Options for configuring a memory buffer by byte size.
max_size [required]Maximum size of the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 3Options for configuring a memory buffer by queue length.
max_events [required]Maximum events for the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestcustom_source_name [required]Custom source name for the logs in Security Lake.
id [required]Unique identifier for the destination component.
inputs [required]A list of component IDs whose output is used as the input for this component.
region [required]AWS region of the S3 bucket.
tlsConfiguration for enabling TLS encryption between the pipeline component and external services.
ca_filePath to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_filePath to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
key_pass_keyName of the environment variable or secret that holds the passphrase for the private key file.
type [required]The destination type. Always amazon_security_lake.
amazon_security_lakeamazon_security_lake<type=azure_storage>The azure_storage destination forwards logs to an Azure Blob Storage container.
Supported pipeline types: logs
blob_prefixOptional prefix for blobs written to the container.
bufferConfiguration for buffer settings on destination components.
Object 1Options for configuring a disk buffer.
max_size [required]Maximum size of the disk buffer.
typeThe type of the buffer that will be configured, a disk buffer.
diskdiskwhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 2Options for configuring a memory buffer by byte size.
max_size [required]Maximum size of the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 3Options for configuring a memory buffer by queue length.
max_events [required]Maximum events for the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestcompressionCompression configuration for archived logs. When omitted, logs are compressed with gzip for backward compatibility.
<algorithm=zstd>Zstd compression.
algorithm [required]The compression type. Always zstd.
zstdzstdlevel [required]Zstd compression level. Valid values range from 1 to 21.
<algorithm=gzip>Gzip compression.
algorithm [required]The compression type. Always gzip.
gzipgziplevel [required]Gzip compression level. Valid values range from 1 to 9.
connection_string_keyName of the environment variable or secret that holds the Azure Storage connection string.
container_name [required]The name of the Azure Blob Storage container to store logs in.
id [required]The unique identifier for this component.
inputs [required]A list of component IDs whose output is used as the input for this component.
type [required]The destination type. The value should always be azure_storage.
azure_storageazure_storage<type=azure_data_explorer>The azure_data_explorer destination sends log events to an Azure Data Explorer table.
Supported pipeline types: logs
auth [required]Authentication configuration for Azure Data Explorer. The azure_credential_kind field selects the credential type.
<azure_credential_kind=azure_cli>Authenticate using the Azure CLI credentials available in the environment.
azure_credential_kind [required]The Azure credential kind. The value should always be azure_cli.
azure_cliazure_cli<azure_credential_kind=client_secret_credential>Authenticate using a Microsoft Entra application client secret.
azure_client_id [required]The Microsoft Entra application (client) ID.
azure_client_secret_key [required]Name of the environment variable or secret that holds the Microsoft Entra application client secret.
azure_credential_kind [required]The Azure credential kind. The value should always be client_secret_credential.
client_secret_credentialclient_secret_credentialazure_tenant_id [required]The Microsoft Entra tenant ID.
<azure_credential_kind=client_certificate_credential>Authenticate using a Microsoft Entra application client certificate.
azure_client_id [required]The Microsoft Entra application (client) ID.
azure_credential_kind [required]The Azure credential kind. The value should always be client_certificate_credential.
client_certificate_credentialclient_certificate_credentialazure_tenant_id [required]The Microsoft Entra tenant ID.
certificate_password_keyName of the environment variable or secret that holds the password for the client certificate.
certificate_path [required]Path to the .pfx client certificate file on the Worker.
<azure_credential_kind=managed_identity>Authenticate using an Azure managed identity.
azure_credential_kind [required]The Azure credential kind. The value should always be managed_identity.
managed_identitymanaged_identityuser_assigned_managed_identity_idThe ID of the user-assigned managed identity. If omitted, the system-assigned managed identity is used.
user_assigned_managed_identity_id_typeThe type of the user-assigned managed identity ID.
client_id, object_id, resource_id<azure_credential_kind=managed_identity_client_assertion>Authenticate using a managed identity as a client assertion for a Microsoft Entra application.
azure_credential_kind [required]The Azure credential kind. The value should always be managed_identity_client_assertion.
managed_identity_client_assertionmanaged_identity_client_assertionclient_assertion_client_id [required]The client ID of the Microsoft Entra application that trusts the managed identity.
client_assertion_tenant_id [required]The tenant ID of the Microsoft Entra application that trusts the managed identity.
user_assigned_managed_identity_idThe ID of the user-assigned managed identity. If omitted, the system-assigned managed identity is used.
user_assigned_managed_identity_id_typeThe type of the user-assigned managed identity ID.
client_id, object_id, resource_id<azure_credential_kind=workload_identity>Authenticate using Azure Workload Identity (for example, on Kubernetes).
azure_credential_kind [required]The Azure credential kind. The value should always be workload_identity.
workload_identityworkload_identityclient_idThe client ID of the Microsoft Entra application. If omitted, it is read from the environment.
tenant_idThe Microsoft Entra tenant ID. If omitted, it is read from the environment.
token_file_pathPath to the federated token file. If omitted, it is read from the environment.
batchEvent batching settings for Azure Data Explorer ingestion.
max_bytesMaximum batch size in bytes.
max_eventsMaximum number of events per batch before it is flushed.
timeout_secsMaximum number of seconds to wait before flushing a partial batch.
bufferConfiguration for buffer settings on destination components.
Object 1Options for configuring a disk buffer.
max_size [required]Maximum size of the disk buffer.
typeThe type of the buffer that will be configured, a disk buffer.
diskdiskwhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 2Options for configuring a memory buffer by byte size.
max_size [required]Maximum size of the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 3Options for configuring a memory buffer by queue length.
max_events [required]Maximum events for the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestcompressionGzip compression.
algorithm [required]The compression type. Always gzip.
gzipgziplevel [required]Gzip compression level. Valid values range from 1 to 9.
database [required]The name of the Azure Data Explorer database to ingest into. Supports template syntax.
id [required]The unique identifier for this component.
ingestion_endpoint_keyName of the environment variable or secret that holds the Azure Data Explorer ingestion endpoint URL.
Defaults to DESTINATION_AZURE_DATA_EXPLORER_INGESTION_ENDPOINT (prefixed with DD_OP_ at runtime).
inputs [required]A list of component IDs whose output is used as the input for this component.
mapping_referenceThe name of a pre-created ingestion mapping on the table used to map incoming events to columns. Supports template syntax.
table [required]The name of the Azure Data Explorer table to ingest into. Supports template syntax.
token_scopeThe OAuth scope requested when acquiring an access token for Azure Data Explorer.
Defaults to https://kusto.kusto.windows.net/.default.
type [required]The destination type. The value should always be azure_data_explorer.
azure_data_explorerazure_data_explorer<type=clickhouse>The clickhouse destination sends log events to a ClickHouse database table over HTTP.
Supported pipeline types: logs.
authHTTP Basic Authentication credentials for the ClickHouse destination.
When strategy is basic, provide username_key and password_key that reference environment variables or secrets containing the credentials.
password_keyName of the environment variable or secret that contains the ClickHouse password.
strategy [required]The authentication strategy for ClickHouse HTTP requests. Only basic is supported.
basicusername_keyName of the environment variable or secret that contains the ClickHouse username.
batchBatching configuration for ClickHouse inserts.
max_eventsMaximum number of events per batch before it is flushed.
timeout_secsMaximum number of seconds to wait before flushing a partial batch.
batch_encodingBatch encoding configuration for the ClickHouse destination.
Required when format is arrow_stream. The codec field must be set to arrow_stream.
allow_nullable_fieldsWhen true, null values are allowed for non-nullable fields in the ClickHouse schema.
When false (default), missing values for non-nullable columns cause encoding errors.
codec [required]The codec used for batch encoding. Only arrow_stream is supported.
arrow_streambufferConfiguration for buffer settings on destination components.
Object 1Options for configuring a disk buffer.
max_size [required]Maximum size of the disk buffer.
typeThe type of the buffer that will be configured, a disk buffer.
diskdiskwhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 2Options for configuring a memory buffer by byte size.
max_size [required]Maximum size of the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 3Options for configuring a memory buffer by queue length.
max_events [required]Maximum events for the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestcompressionCompression setting for outbound HTTP requests to ClickHouse.
Can be specified as a shorthand string ("gzip" or "none") or as an object
with an algorithm field and an optional level (gzip only, 1–9).
Object 1The compression algorithm applied to outbound HTTP requests.
gzip, noneObject 2Structured compression configuration for the ClickHouse destination.
Use algorithm to specify the compression type and level (optional, gzip only) to control compression strength.
algorithm [required]The compression algorithm applied to outbound HTTP requests.
gzip, nonelevelCompression level (1–9). Only applicable when algorithm is gzip.
databaseOptional ClickHouse database name. If omitted, the user's default database on the ClickHouse server is used.
date_time_best_effortWhen true, enables flexible DateTime parsing on the ClickHouse server side.
endpoint_url_keyName of the environment variable or secret that contains the ClickHouse HTTP endpoint URL.
Defaults to DESTINATION_CLICKHOUSE_ENDPOINT_URL (prefixed with DD_OP_ at runtime).
formatInsert format for events sent to ClickHouse.
json_each_row: Maps event fields to columns by name (ClickHouseJSONEachRow).json_as_object: Inserts each event into a singleObject('json')/JSONcolumn (ClickHouseJSONAsObject).json_as_string: Inserts each event into a singleString-typed column as raw JSON (ClickHouseJSONAsString).arrow_stream: Batches events using Apache Arrow IPC streaming format. Requiresbatch_encoding.
json_each_row, json_as_object, json_as_string, arrow_streamid [required]The unique identifier for this component.
inputs [required]A list of component IDs whose output is used as the input for this component.
skip_unknown_fieldsWhen true, fields not present in the target table schema are dropped instead of causing insert errors.
When unset, the ClickHouse server's own input_format_skip_unknown_fields setting applies.
table [required]Target ClickHouse table name. Events are inserted into this table.
tlsConfiguration for enabling TLS encryption between the pipeline component and external services.
ca_filePath to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_filePath to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
key_pass_keyName of the environment variable or secret that holds the passphrase for the private key file.
type [required]The destination type. The value must be clickhouse.
clickhouseclickhouse<type=cloud_prem>The cloud_prem destination sends logs to Datadog CloudPrem.
Supported pipeline types: logs
bufferConfiguration for buffer settings on destination components.
Object 1Options for configuring a disk buffer.
max_size [required]Maximum size of the disk buffer.
typeThe type of the buffer that will be configured, a disk buffer.
diskdiskwhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 2Options for configuring a memory buffer by byte size.
max_size [required]Maximum size of the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 3Options for configuring a memory buffer by queue length.
max_events [required]Maximum events for the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestendpoint_url_keyName of the environment variable or secret that holds the CloudPrem endpoint URL.
id [required]The unique identifier for this component.
inputs [required]A list of component IDs whose output is used as the input for this component.
tlsConfiguration for TLS encryption.
ca_filePath to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_filePath to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
key_pass_keyName of the environment variable or secret that holds the passphrase for the private key file.
server_nameServer name to use for Server Name Indication (SNI) and to verify against the certificate presented by the remote host. Use this when the address you connect to doesn't match the certificate's Common Name or Subject Alternative Name.
type [required]The destination type. The value should always be cloud_prem.
cloud_premcloud_prem<type=crowdstrike_next_gen_siem>The crowdstrike_next_gen_siem destination forwards logs to CrowdStrike Next Gen SIEM.
Supported pipeline types: logs
bufferConfiguration for buffer settings on destination components.
Object 1Options for configuring a disk buffer.
max_size [required]Maximum size of the disk buffer.
typeThe type of the buffer that will be configured, a disk buffer.
diskdiskwhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 2Options for configuring a memory buffer by byte size.
max_size [required]Maximum size of the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 3Options for configuring a memory buffer by queue length.
max_events [required]Maximum events for the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestcompressionCompression configuration for log events.
algorithm [required]Compression algorithm for log events.
gzip, zliblevelCompression level.
encoding [required]Encoding format for log events.
json, raw_messageendpoint_url_keyName of the environment variable or secret that holds the CrowdStrike endpoint URL.
id [required]The unique identifier for this component.
inputs [required]A list of component IDs whose output is used as the input for this component.
tlsConfiguration for enabling TLS encryption between the pipeline component and external services.
ca_filePath to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_filePath to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
key_pass_keyName of the environment variable or secret that holds the passphrase for the private key file.
token_keyName of the environment variable or secret that holds the CrowdStrike API token.
type [required]The destination type. The value should always be crowdstrike_next_gen_siem.
crowdstrike_next_gen_siemcrowdstrike_next_gen_siem<type=datadog_logs>The datadog_logs destination forwards logs to Datadog Log Management.
Supported pipeline types: logs
bufferConfiguration for buffer settings on destination components.
Object 1Options for configuring a disk buffer.
max_size [required]Maximum size of the disk buffer.
typeThe type of the buffer that will be configured, a disk buffer.
diskdiskwhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 2Options for configuring a memory buffer by byte size.
max_size [required]Maximum size of the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 3Options for configuring a memory buffer by queue length.
max_events [required]Maximum events for the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestid [required]The unique identifier for this component.
inputs [required]A list of component IDs whose output is used as the input for this component.
routesA list of routing rules that forward matching logs to Datadog using dedicated API keys.
api_key_keyName of the environment variable or secret that stores the Datadog API key used by this route.
includeA Datadog search query that determines which logs are forwarded using this route.
route_idUnique identifier for this route within the destination.
siteDatadog site where matching logs are sent (for example, us1).
type [required]The destination type. The value should always be datadog_logs.
datadog_logsdatadog_logs<type=google_chronicle>The google_chronicle destination sends logs to Google Chronicle.
Supported pipeline types: logs
authGoogle Cloud credentials used to authenticate with Google Cloud Storage.
credentials_file [required]Path to the Google Cloud service account key file.
bufferConfiguration for buffer settings on destination components.
Object 1Options for configuring a disk buffer.
max_size [required]Maximum size of the disk buffer.
typeThe type of the buffer that will be configured, a disk buffer.
diskdiskwhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 2Options for configuring a memory buffer by byte size.
max_size [required]Maximum size of the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 3Options for configuring a memory buffer by queue length.
max_events [required]Maximum events for the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestcustomer_id [required]The Google Chronicle customer ID.
encodingThe encoding format for the logs sent to Chronicle.
json, raw_messageendpoint_url_keyName of the environment variable or secret that holds the Google Chronicle endpoint URL.
id [required]The unique identifier for this component.
inputs [required]A list of component IDs whose output is used as the input for this component.
log_typeThe log type metadata associated with the Chronicle destination.
type [required]The destination type. The value should always be google_chronicle.
google_chroniclegoogle_chronicle<type=google_cloud_storage>The google_cloud_storage destination stores logs in a Google Cloud Storage (GCS) bucket.
It requires a bucket name, Google Cloud authentication, and metadata fields.
Supported pipeline types: logs
aclAccess control list setting for objects written to the bucket.
private, project-private, public-read, authenticated-read, bucket-owner-read, bucket-owner-full-controlauthGoogle Cloud credentials used to authenticate with Google Cloud Storage.
credentials_file [required]Path to the Google Cloud service account key file.
bucket [required]Name of the GCS bucket.
bufferConfiguration for buffer settings on destination components.
Object 1Options for configuring a disk buffer.
max_size [required]Maximum size of the disk buffer.
typeThe type of the buffer that will be configured, a disk buffer.
diskdiskwhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 2Options for configuring a memory buffer by byte size.
max_size [required]Maximum size of the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 3Options for configuring a memory buffer by queue length.
max_events [required]Maximum events for the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestcompressionCompression configuration for archived logs. When omitted, logs are compressed with gzip for backward compatibility.
<algorithm=zstd>Zstd compression.
algorithm [required]The compression type. Always zstd.
zstdzstdlevel [required]Zstd compression level. Valid values range from 1 to 21.
<algorithm=gzip>Gzip compression.
algorithm [required]The compression type. Always gzip.
gzipgziplevel [required]Gzip compression level. Valid values range from 1 to 9.
id [required]Unique identifier for the destination component.
inputs [required]A list of component IDs whose output is used as the input for this component.
key_prefixOptional prefix for object keys within the GCS bucket.
metadataCustom metadata to attach to each object uploaded to the GCS bucket.
name [required]The metadata key.
value [required]The metadata value.
storage_class [required]Storage class used for objects stored in GCS.
STANDARD, NEARLINE, COLDLINE, ARCHIVEtype [required]The destination type. Always google_cloud_storage.
google_cloud_storagegoogle_cloud_storage<type=google_pubsub>The google_pubsub destination publishes logs to a Google Cloud Pub/Sub topic.
Supported pipeline types: logs
authGoogle Cloud credentials used to authenticate with Google Cloud Storage.
credentials_file [required]Path to the Google Cloud service account key file.
bufferConfiguration for buffer settings on destination components.
Object 1Options for configuring a disk buffer.
max_size [required]Maximum size of the disk buffer.
typeThe type of the buffer that will be configured, a disk buffer.
diskdiskwhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 2Options for configuring a memory buffer by byte size.
max_size [required]Maximum size of the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 3Options for configuring a memory buffer by queue length.
max_events [required]Maximum events for the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestencoding [required]Encoding format for log events.
json, raw_messageendpoint_url_keyName of the environment variable or secret that holds the Google Cloud Pub/Sub endpoint URL.
id [required]The unique identifier for this component.
inputs [required]A list of component IDs whose output is used as the input for this component.
project [required]The Google Cloud project ID that owns the Pub/Sub topic.
tlsConfiguration for enabling TLS encryption between the pipeline component and external services.
ca_filePath to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_filePath to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
key_pass_keyName of the environment variable or secret that holds the passphrase for the private key file.
topic [required]The Pub/Sub topic name to publish logs to.
type [required]The destination type. The value should always be google_pubsub.
google_pubsubgoogle_pubsub<type=kafka>The kafka destination sends logs to Apache Kafka topics.
Supported pipeline types: logs
bootstrap_servers_keyName of the environment variable or secret that holds the Kafka bootstrap servers list.
bufferConfiguration for buffer settings on destination components.
Object 1Options for configuring a disk buffer.
max_size [required]Maximum size of the disk buffer.
typeThe type of the buffer that will be configured, a disk buffer.
diskdiskwhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 2Options for configuring a memory buffer by byte size.
max_size [required]Maximum size of the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 3Options for configuring a memory buffer by queue length.
max_events [required]Maximum events for the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestcompressionCompression codec for Kafka messages.
none, gzip, snappy, lz4, zstdencoding [required]Encoding format for log events.
json, raw_messageheaders_keyThe field name to use for Kafka message headers.
id [required]The unique identifier for this component.
inputs [required]A list of component IDs whose output is used as the input for this component.
key_fieldThe field name to use as the Kafka message key.
librdkafka_optionsOptional list of advanced Kafka producer configuration options, defined as key-value pairs.
name [required]The name of the librdkafka configuration option to set.
value [required]The value assigned to the specified librdkafka configuration option.
message_timeout_msMaximum time in milliseconds to wait for message delivery confirmation.
rate_limit_duration_secsDuration in seconds for the rate limit window.
rate_limit_numMaximum number of messages allowed per rate limit duration.
saslSpecifies the SASL mechanism for authenticating with a Kafka cluster.
mechanismSASL mechanism used for Kafka authentication.
PLAIN, SCRAM-SHA-256, SCRAM-SHA-512password_keyName of the environment variable or secret that holds the SASL password.
username_keyName of the environment variable or secret that holds the SASL username.
socket_timeout_msSocket timeout in milliseconds for network requests.
tlsConfiguration for enabling TLS encryption between the pipeline component and external services.
ca_filePath to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_filePath to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
key_pass_keyName of the environment variable or secret that holds the passphrase for the private key file.
topic [required]The Kafka topic name to publish logs to.
type [required]The destination type. The value should always be kafka.
kafkakafka<type=microsoft_sentinel>The microsoft_sentinel destination forwards logs to Microsoft Sentinel.
Supported pipeline types: logs
bufferConfiguration for buffer settings on destination components.
Object 1Options for configuring a disk buffer.
max_size [required]Maximum size of the disk buffer.
typeThe type of the buffer that will be configured, a disk buffer.
diskdiskwhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 2Options for configuring a memory buffer by byte size.
max_size [required]Maximum size of the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 3Options for configuring a memory buffer by queue length.
max_events [required]Maximum events for the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestclient_id [required]Azure AD client ID used for authentication.
client_secret_keyName of the environment variable or secret that holds the Azure AD client secret.
dce_uri_keyName of the environment variable or secret that holds the Data Collection Endpoint (DCE) URI.
dcr_immutable_id [required]The immutable ID of the Data Collection Rule (DCR).
id [required]The unique identifier for this component.
inputs [required]A list of component IDs whose output is used as the input for this component.
table [required]The name of the Log Analytics table where logs are sent.
tenant_id [required]Azure AD tenant ID.
type [required]The destination type. The value should always be microsoft_sentinel.
microsoft_sentinelmicrosoft_sentinel<type=new_relic>The new_relic destination sends logs to the New Relic platform.
Supported pipeline types: logs
account_id_keyName of the environment variable or secret that holds the New Relic account ID.
bufferConfiguration for buffer settings on destination components.
Object 1Options for configuring a disk buffer.
max_size [required]Maximum size of the disk buffer.
typeThe type of the buffer that will be configured, a disk buffer.
diskdiskwhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 2Options for configuring a memory buffer by byte size.
max_size [required]Maximum size of the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 3Options for configuring a memory buffer by queue length.
max_events [required]Maximum events for the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestid [required]The unique identifier for this component.
inputs [required]A list of component IDs whose output is used as the input for this component.
license_key_keyName of the environment variable or secret that holds the New Relic license key.
region [required]The New Relic region.
us, eutype [required]The destination type. The value should always be new_relic.
new_relicnew_relic<type=opensearch>The opensearch destination writes logs to an OpenSearch cluster.
Supported pipeline types: logs
authAuthentication settings for the Elasticsearch destination.
When strategy is basic, use username_key and password_key to reference credentials stored in environment variables or secrets.
password_keyName of the environment variable or secret that holds the Elasticsearch password (used when strategy is basic).
strategy [required]The authentication strategy to use.
basic, awsusername_keyName of the environment variable or secret that holds the Elasticsearch username (used when strategy is basic).
bufferConfiguration for buffer settings on destination components.
Object 1Options for configuring a disk buffer.
max_size [required]Maximum size of the disk buffer.
typeThe type of the buffer that will be configured, a disk buffer.
diskdiskwhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 2Options for configuring a memory buffer by byte size.
max_size [required]Maximum size of the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 3Options for configuring a memory buffer by queue length.
max_events [required]Maximum events for the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestbulk_indexThe index to write logs to.
data_streamConfiguration options for writing to OpenSearch Data Streams instead of a fixed index.
datasetThe data stream dataset for your logs. This groups logs by their source or application.
dtypeThe data stream type for your logs. This determines how logs are categorized within the data stream.
namespaceThe data stream namespace for your logs. This separates logs into different environments or domains.
endpoint_url_keyName of the environment variable or secret that holds the OpenSearch endpoint URL.
id [required]The unique identifier for this component.
inputs [required]A list of component IDs whose output is used as the input for this component.
type [required]The destination type. The value should always be opensearch.
opensearchopensearch<type=rsyslog>The rsyslog destination forwards logs to an external rsyslog server over TCP or UDP using the syslog protocol.
Supported pipeline types: logs
bufferConfiguration for buffer settings on destination components.
Object 1Options for configuring a disk buffer.
max_size [required]Maximum size of the disk buffer.
typeThe type of the buffer that will be configured, a disk buffer.
diskdiskwhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 2Options for configuring a memory buffer by byte size.
max_size [required]Maximum size of the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 3Options for configuring a memory buffer by queue length.
max_events [required]Maximum events for the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestendpoint_url_keyName of the environment variable or secret that holds the syslog server endpoint URL.
id [required]The unique identifier for this component.
inputs [required]A list of component IDs whose output is used as the input for this component.
keepaliveOptional socket keepalive duration in milliseconds.
tlsConfiguration for enabling TLS encryption between the pipeline component and external services.
ca_filePath to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_filePath to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
key_pass_keyName of the environment variable or secret that holds the passphrase for the private key file.
type [required]The destination type. The value should always be rsyslog.
rsyslogrsyslog<type=sentinel_one>The sentinel_one destination sends logs to SentinelOne.
Supported pipeline types: logs
bufferConfiguration for buffer settings on destination components.
Object 1Options for configuring a disk buffer.
max_size [required]Maximum size of the disk buffer.
typeThe type of the buffer that will be configured, a disk buffer.
diskdiskwhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 2Options for configuring a memory buffer by byte size.
max_size [required]Maximum size of the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 3Options for configuring a memory buffer by queue length.
max_events [required]Maximum events for the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestid [required]The unique identifier for this component.
inputs [required]A list of component IDs whose output is used as the input for this component.
region [required]The SentinelOne region to send logs to.
us, eu, ca, data_set_ustoken_keyName of the environment variable or secret that holds the SentinelOne API token.
type [required]The destination type. The value should always be sentinel_one.
sentinel_onesentinel_one<type=socket>The socket destination sends logs over TCP or UDP to a remote server.
Supported pipeline types: logs
address_keyName of the environment variable or secret that holds the socket address (host:port).
bufferConfiguration for buffer settings on destination components.
Object 1Options for configuring a disk buffer.
max_size [required]Maximum size of the disk buffer.
typeThe type of the buffer that will be configured, a disk buffer.
diskdiskwhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 2Options for configuring a memory buffer by byte size.
max_size [required]Maximum size of the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 3Options for configuring a memory buffer by queue length.
max_events [required]Maximum events for the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestencoding [required]Encoding format for log events.
json, raw_messageframing [required]Framing method configuration.
<method=newline_delimited>Each log event is delimited by a newline character.
method [required]The definition of ObservabilityPipelineSocketDestinationFramingNewlineDelimitedMethod object.
newline_delimited<method=bytes>Event data is not delimited at all.
method [required]The definition of ObservabilityPipelineSocketDestinationFramingBytesMethod object.
bytes<method=character_delimited>Each log event is separated using the specified delimiter character.
delimiter [required]A single ASCII character used as a delimiter.
method [required]The definition of ObservabilityPipelineSocketDestinationFramingCharacterDelimitedMethod object.
character_delimitedid [required]The unique identifier for this component.
inputs [required]A list of component IDs whose output is used as the input for this component.
mode [required]Protocol used to send logs.
tcp, udptlsTLS configuration. Relevant only when mode is tcp.
ca_filePath to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_filePath to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
key_pass_keyName of the environment variable or secret that holds the passphrase for the private key file.
server_nameServer name to use for Server Name Indication (SNI) and to verify against the certificate presented by the remote host. Use this when the address you connect to doesn't match the certificate's Common Name or Subject Alternative Name.
type [required]The destination type. The value should always be socket.
socketsocket<type=splunk_hec>The splunk_hec destination forwards logs to Splunk using the HTTP Event Collector (HEC).
Supported pipeline types: logs
auto_extract_timestampIf true, Splunk tries to extract timestamps from incoming log events.
If false, Splunk assigns the time the event was received.
Only applies when endpoint_target is event; cannot be true when endpoint_target is raw.
bufferConfiguration for buffer settings on destination components.
Object 1Options for configuring a disk buffer.
max_size [required]Maximum size of the disk buffer.
typeThe type of the buffer that will be configured, a disk buffer.
diskdiskwhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 2Options for configuring a memory buffer by byte size.
max_size [required]Maximum size of the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 3Options for configuring a memory buffer by queue length.
max_events [required]Maximum events for the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestencodingEncoding format for log events.
json, raw_messageendpoint_targetThe Splunk HEC endpoint to send events to. Use event to send structured events to the /event endpoint, or raw to send the raw message to the /raw endpoint.
event, rawendpoint_url_keyName of the environment variable or secret that holds the Splunk HEC endpoint URL.
id [required]The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the input to downstream components).
indexOptional name of the Splunk index where logs are written.
indexed_fieldsList of log field names to send as indexed fields to Splunk HEC. Available only when encoding is json.
inputs [required]A list of component IDs whose output is used as the input for this component.
sourcetypeThe Splunk sourcetype to assign to log events.
token_keyName of the environment variable or secret that holds the Splunk HEC token.
token_strategyControls how the Splunk HEC token is supplied. Use custom to provide a token with token_key, or from_source to forward the token received from an upstream Splunk HEC source.
custom, from_sourcetype [required]The destination type. Always splunk_hec.
splunk_hecsplunk_hec<type=sumo_logic>The sumo_logic destination forwards logs to Sumo Logic.
Supported pipeline types: logs
bufferConfiguration for buffer settings on destination components.
Object 1Options for configuring a disk buffer.
max_size [required]Maximum size of the disk buffer.
typeThe type of the buffer that will be configured, a disk buffer.
diskdiskwhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 2Options for configuring a memory buffer by byte size.
max_size [required]Maximum size of the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 3Options for configuring a memory buffer by queue length.
max_events [required]Maximum events for the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestencodingThe output encoding format.
json, raw_message, logfmtendpoint_url_keyName of the environment variable or secret that holds the Sumo Logic HTTP endpoint URL.
header_custom_fieldsA list of custom headers to include in the request to Sumo Logic.
name [required]The header field name.
value [required]The header field value.
header_host_nameOptional override for the host name header.
header_source_categoryOptional override for the source category header.
header_source_nameOptional override for the source name header.
id [required]The unique identifier for this component.
inputs [required]A list of component IDs whose output is used as the input for this component.
type [required]The destination type. The value should always be sumo_logic.
sumo_logicsumo_logic<type=syslog_ng>The syslog_ng destination forwards logs to an external syslog-ng server over TCP or UDP using the syslog protocol.
Supported pipeline types: logs
bufferConfiguration for buffer settings on destination components.
Object 1Options for configuring a disk buffer.
max_size [required]Maximum size of the disk buffer.
typeThe type of the buffer that will be configured, a disk buffer.
diskdiskwhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 2Options for configuring a memory buffer by byte size.
max_size [required]Maximum size of the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 3Options for configuring a memory buffer by queue length.
max_events [required]Maximum events for the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestendpoint_url_keyName of the environment variable or secret that holds the syslog-ng server endpoint URL.
id [required]The unique identifier for this component.
inputs [required]A list of component IDs whose output is used as the input for this component.
keepaliveOptional socket keepalive duration in milliseconds.
tlsConfiguration for enabling TLS encryption between the pipeline component and external services.
ca_filePath to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_filePath to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
key_pass_keyName of the environment variable or secret that holds the passphrase for the private key file.
server_nameServer name to use for Server Name Indication (SNI) and to verify against the certificate presented by the remote host. Use this when the address you connect to doesn't match the certificate's Common Name or Subject Alternative Name.
type [required]The destination type. The value should always be syslog_ng.
syslog_ngsyslog_ng<type=databricks_zerobus>The databricks_zerobus destination sends logs to Databricks using the Zerobus ingestion API, streaming data directly into your Databricks Lakehouse.
Supported pipeline types: Logs, rehydration
auth [required]OAuth credentials for authenticating with the Databricks Zerobus ingestion API.
client_id [required]Your service principal application ID (UUID).
client_secret_keyName of the environment variable or secret that holds the OAuth client secret used to authenticate with the Databricks ingestion endpoint.
bufferConfiguration for buffer settings on destination components.
Object 1Options for configuring a disk buffer.
max_size [required]Maximum size of the disk buffer.
typeThe type of the buffer that will be configured, a disk buffer.
diskdiskwhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 2Options for configuring a memory buffer by byte size.
max_size [required]Maximum size of the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 3Options for configuring a memory buffer by queue length.
max_events [required]Maximum events for the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestid [required]The unique identifier for this component.
ingestion_endpoint_keyName of the environment variable or the secret identifier that references the Databricks Zerobus ingestion endpoint, which is used to stream data directly into your Databricks Lakehouse.
inputs [required]A list of component IDs whose output is used as the input for this component.
table_name [required]The fully qualified name of your target Databricks table. Make sure this table already exists in your Databricks workspace before deploying.
type [required]The destination type. The value must be databricks_zerobus.
databricks_zerobusdatabricks_zerobusunity_catalog_endpoint_keyName of the environment variable or the secret identifier that references your Databricks workspace URL, which is used to communicate with the Unity Catalog API.
<type=datadog_metrics>The datadog_metrics destination forwards metrics to Datadog.
Supported pipeline types: metrics
id [required]The unique identifier for this component.
inputs [required]A list of component IDs whose output is used as the input for this component.
type [required]The destination type. The value should always be datadog_metrics.
datadog_metricsdatadog_metrics<type=prometheus_remote_write>The prometheus_remote_write destination forwards metrics to an endpoint that supports the Prometheus Remote Write protocol.
Supported pipeline types: metrics
auth_strategyThe authentication strategy to use for outgoing Prometheus Remote Write requests.
none, basic, bearerbufferConfiguration for buffer settings on destination components.
Object 1Options for configuring a disk buffer.
max_size [required]Maximum size of the disk buffer.
typeThe type of the buffer that will be configured, a disk buffer.
diskdiskwhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 2Options for configuring a memory buffer by byte size.
max_size [required]Maximum size of the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 3Options for configuring a memory buffer by queue length.
max_events [required]Maximum events for the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestdefault_namespaceThe default namespace to add as a prefix to metric names that do not already have one.
endpoint_url_keyName of the environment variable or secret that holds the Prometheus Remote Write endpoint URL.
Defaults to DESTINATION_PROMETHEUS_REMOTE_WRITE_ENDPOINT_URL (prefixed with DD_OP_ at runtime).
id [required]The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the input to downstream components).
inputs [required]A list of component IDs whose output is used as the input for this component.
password_keyName of the environment variable or secret that holds the password (used when auth_strategy is basic).
tenant_idThe tenant ID to include with outgoing requests. Used by multi-tenant Prometheus Remote Write receivers.
tlsConfiguration for enabling TLS encryption between the pipeline component and external services.
ca_filePath to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_filePath to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
key_pass_keyName of the environment variable or secret that holds the passphrase for the private key file.
server_nameServer name to use for Server Name Indication (SNI) and to verify against the certificate presented by the remote host. Use this when the address you connect to doesn't match the certificate's Common Name or Subject Alternative Name.
token_keyName of the environment variable or secret that holds the bearer token (used when auth_strategy is bearer).
type [required]The destination type. The value should always be prometheus_remote_write.
prometheus_remote_writeprometheus_remote_writeusername_keyName of the environment variable or secret that holds the username (used when auth_strategy is basic).
<type=splunk_hec_metrics>The splunk_hec_metrics destination forwards metrics to Splunk using the HTTP Event Collector (HEC).
Supported pipeline types: metrics
bufferConfiguration for buffer settings on destination components.
Object 1Options for configuring a disk buffer.
max_size [required]Maximum size of the disk buffer.
typeThe type of the buffer that will be configured, a disk buffer.
diskdiskwhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 2Options for configuring a memory buffer by byte size.
max_size [required]Maximum size of the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 3Options for configuring a memory buffer by queue length.
max_events [required]Maximum events for the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestcompressionCompression algorithm applied when sending metrics to Splunk HEC.
nonenone, gzipdefault_namespaceOptional default namespace for metrics sent to Splunk HEC.
endpoint_url_keyName of the environment variable or secret that holds the Splunk HEC endpoint URL.
id [required]The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the input to downstream components).
indexOptional name of the Splunk index where metrics are written.
inputs [required]A list of component IDs whose output is used as the input for this component.
sourceThe Splunk source field value for metric events.
sourcetypeThe Splunk sourcetype to assign to metric events.
tlsConfiguration for enabling TLS encryption between the pipeline component and external services.
ca_filePath to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_filePath to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
key_pass_keyName of the environment variable or secret that holds the passphrase for the private key file.
token_keyName of the environment variable or secret that holds the Splunk HEC token.
type [required]The destination type. Always splunk_hec_metrics.
splunk_hec_metricssplunk_hec_metrics<type=opentelemetry>The opentelemetry destination forwards metrics or traces using the OpenTelemetry Protocol (OTLP) over HTTP.
Supported pipeline types: metrics, traces
bufferConfiguration for buffer settings on destination components.
Object 1Options for configuring a disk buffer.
max_size [required]Maximum size of the disk buffer.
typeThe type of the buffer that will be configured, a disk buffer.
diskdiskwhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 2Options for configuring a memory buffer by byte size.
max_size [required]Maximum size of the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 3Options for configuring a memory buffer by queue length.
max_events [required]Maximum events for the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newesthttp_client_uri_keyEnvironment variable name containing the URI of the OTLP HTTP endpoint to send metrics or traces to.
DESTINATION_OTEL_HTTP_CLIENT_URIid [required]The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the input to downstream components).
inputs [required]A list of component IDs whose output is used as the input for this component.
tlsConfiguration for enabling TLS encryption between the pipeline component and external services.
ca_filePath to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_filePath to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
key_pass_keyName of the environment variable or secret that holds the passphrase for the private key file.
type [required]The destination type. Always opentelemetry.
opentelemetryopentelemetry<type=datadog_traces>The datadog_traces destination forwards traces to Datadog.
Supported pipeline types: traces
bufferConfiguration for buffer settings on destination components.
Object 1Options for configuring a disk buffer.
max_size [required]Maximum size of the disk buffer.
typeThe type of the buffer that will be configured, a disk buffer.
diskdiskwhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 2Options for configuring a memory buffer by byte size.
max_size [required]Maximum size of the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestObject 3Options for configuring a memory buffer by queue length.
max_events [required]Maximum events for the memory buffer.
typeThe type of the buffer that will be configured, a memory buffer.
memorymemorywhen_fullBehavior when the buffer is full (block and stop accepting new events, or drop new events)
blockblock, drop_newestid [required]The unique identifier for this component.
inputs [required]A list of component IDs whose output is used as the input for this component.
type [required]The destination type. The value should always be datadog_traces.
datadog_tracesdatadog_tracespipeline_typeThe type of data being ingested. Defaults to logs if not specified.
logslogs, metrics, tracesprocessor_groupsA list of processor groups that transform or enrich log data.
display_nameThe display name for a component.
enabled [required]Whether this processor group is enabled.
id [required]The unique identifier for the processor group.
include [required]Conditional expression for when this processor group should execute.
inputs [required]A list of IDs for components whose output is used as the input for this processor group.
processors [required]Processors applied sequentially within this group. Events flow through each processor in order.
<type=filter>The filter processor allows conditional processing of logs, metrics, or traces based on a Datadog search query. Events that match the include query are passed through; others are discarded.
Supported pipeline types: logs, metrics, traces
display_nameThe display name for a component.
enabled [required]Indicates whether the processor is enabled.
id [required]The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the input to downstream components).
include [required]A Datadog search query used to determine which events should pass through the filter. Events that match this query continue to downstream components; others are dropped.
type [required]The processor type. The value should always be filter.
filterfilter<type=add_env_vars>The add_env_vars processor adds environment variable values to log events.
Supported pipeline types: logs
display_nameThe display name for a component.
enabled [required]Indicates whether the processor is enabled.
id [required]The unique identifier for this component. Used to reference this processor in the pipeline.
include [required]A Datadog search query used to determine which logs this processor targets.
type [required]The processor type. The value should always be add_env_vars.
add_env_varsadd_env_varsvariables [required]A list of environment variable mappings to apply to log fields.
field [required]The target field in the log event.
name [required]The name of the environment variable to read.
<type=add_fields>The add_fields processor adds static key-value fields to logs.
Supported pipeline types: logs
display_nameThe display name for a component.
enabled [required]Indicates whether the processor is enabled.
fields [required]A list of static fields (key-value pairs) that is added to each log event processed by this component.
name [required]The field name.
value [required]The field value.
id [required]The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the input to downstream components).
include [required]A Datadog search query used to determine which logs this processor targets.
type [required]The processor type. The value should always be add_fields.
add_fieldsadd_fields<type=add_hostname>The add_hostname processor adds the hostname to log events.
Supported pipeline types: logs
display_nameThe display name for a component.
enabled [required]Indicates whether the processor is enabled.
id [required]The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the input to downstream components).
include [required]A Datadog search query used to determine which logs this processor targets.
type [required]The processor type. The value should always be add_hostname.
add_hostnameadd_hostname<type=datadog_tags>The datadog_tags processor includes or excludes specific Datadog tags in your logs.
Supported pipeline types: logs
action [required]The action to take on tags with matching keys.
include, excludedisplay_nameThe display name for a component.
enabled [required]Indicates whether the processor is enabled.
id [required]The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the input to downstream components).
include [required]A Datadog search query used to determine which logs this processor targets.
keys [required]A list of tag keys.
mode [required]The processing mode.
filtertype [required]The processor type. The value should always be datadog_tags.
datadog_tagsdatadog_tags<type=dedupe>The dedupe processor removes duplicate fields in log events.
Supported pipeline types: logs
cacheConfiguration for the cache used to detect duplicates.
num_events [required]The number of events to cache for duplicate detection.
display_nameThe display name for a component.
enabled [required]Indicates whether the processor is enabled.
fields [required]A list of log field paths to check for duplicates.
id [required]The unique identifier for this processor.
include [required]A Datadog search query used to determine which logs this processor targets.
mode [required]The deduplication mode to apply to the fields.
match, ignoretype [required]The processor type. The value should always be dedupe.
dedupededupe<type=enrichment_table>The enrichment_table processor enriches logs using a static CSV file, GeoIP database, or reference table. Exactly one of file, geoip, or reference_table must be configured.
Supported pipeline types: logs
display_nameThe display name for a component.
enabled [required]Indicates whether the processor is enabled.
fileDefines a static enrichment table loaded from a CSV file.
encoding [required]File encoding format.
delimiter [required]The encoding delimiter.
includes_headers [required]The encoding includes_headers.
type [required]Specifies the encoding format (e.g., CSV) used for enrichment tables.
csvkey [required]Key fields used to look up enrichment values.
column [required]The items column.
comparison [required]Defines how to compare key fields for enrichment table lookups.
equalsfield [required]Specifies the source of the key value used for enrichment table lookups.
Can be a plain field path string or an object specifying event, vrl, or secret.
Object 1A plain field path in the log event, used as the lookup key.
Object 2Looks up a value from a field path in the log event.
event [required]The path to the field in the log event to use as the lookup key.
Object 3Evaluates a VRL expression to produce the lookup key.
vrl [required]A VRL expression that returns the value to use as the lookup key.
Object 4Looks up a value stored as a pipeline secret.
secret [required]The name of the secret containing the lookup key value.
path [required]Path to the CSV file.
schema [required]Schema defining column names and their types.
column [required]The items column.
type [required]Declares allowed data types for enrichment table columns.
string, boolean, integer, float, date, timestampgeoipUses a GeoIP database to enrich logs based on an IP field.
key_field [required]Path to the IP field in the log.
locale [required]Locale used to resolve geographical names.
path [required]Path to the GeoIP database file.
id [required]The unique identifier for this processor.
include [required]A Datadog search query used to determine which logs this processor targets.
reference_tableUses a Datadog reference table to enrich logs.
app_key_keyName of the environment variable or secret that holds the Datadog application key used to access the reference table.
columnsList of column names to include from the reference table. If not provided, all columns are included.
key_field [required]Path to the field in the log event to match against the reference table.
table_id [required]The unique identifier of the reference table.
target [required]Path where enrichment results should be stored in the log.
type [required]The processor type. The value should always be enrichment_table.
enrichment_tableenrichment_table<type=generate_datadog_metrics>DEPRECATED The generate_datadog_metrics processor creates custom metrics from logs and sends them to Datadog.
Metrics can be counters, gauges, or distributions and optionally grouped by log fields.
Supported pipeline types: logs
Deprecated: This processor is deprecated. Use the generate_metrics processor instead.
display_nameThe display name for a component.
enabled [required]Indicates whether the processor is enabled.
id [required]The unique identifier for this component. Used to reference this component in other parts of the pipeline.
includeA Datadog search query used to determine which logs this processor targets.
metricsConfiguration for generating individual metrics.
group_byOptional fields used to group the metric series.
include [required]Datadog filter query to match events for metric generation.
metric_type [required]Type of metric to create.
count, gauge, distributionname [required]Name of the custom metric to be created.
value [required]Specifies how the value of the generated metric is computed.
<strategy=increment_by_one>Strategy that increments a generated metric by one for each matching event.
strategy [required]Increments the metric by 1 for each matching event.
increment_by_one<strategy=increment_by_field>Strategy that increments a generated metric based on the value of an event field.
field [required]Name of the field containing the numeric value to increment the metric by.
strategy [required]Uses a numeric field in the log event as the metric increment.
increment_by_fieldtype [required]The processor type. Always generate_datadog_metrics.
generate_datadog_metricsgenerate_datadog_metrics<type=ocsf_mapper>The ocsf_mapper processor transforms logs into the OCSF schema using a predefined mapping configuration.
Supported pipeline types: logs
display_nameThe display name for a component.
enabled [required]Indicates whether the processor is enabled.
id [required]The unique identifier for this component. Used to reference this component in other parts of the pipeline.
include [required]A Datadog search query used to determine which logs this processor targets.
keep_unmatchedWhether to keep an event that does not match any of the mapping filters.
mappings [required]A list of mapping rules to convert events to the OCSF format.
include [required]A Datadog search query used to select the logs that this mapping should apply to.
mapping [required]Defines a single mapping rule for transforming logs into the OCSF schema.
Object 1Predefined library mappings for common log formats.
CloudTrail Account Change, GCP Cloud Audit CreateBucket, GCP Cloud Audit CreateSink, GCP Cloud Audit SetIamPolicy, GCP Cloud Audit UpdateSink, Github Audit Log API Activity, Google Workspace Admin Audit addPrivilege, Microsoft 365 Defender Incident, Microsoft 365 Defender UserLoggedIn, Okta System Log AuthenticationShow 1 more
Palo Alto Networks Firewall TrafficObject 2Custom OCSF mapping configuration for transforming logs.
mapping [required]A list of field mapping rules for transforming log fields to OCSF schema fields.
defaultThe default value to use if the source field is missing or empty.
dest [required]The destination OCSF field path.
lookupLookup table configuration for mapping source values to destination values.
sourceThe source field path from the log event.
sourcesMultiple source field paths for combined mapping.
valueA static value to use for the destination field.
metadata [required]Metadata for the custom OCSF mapping.
class [required]The OCSF event class name.
profilesA list of OCSF profiles to apply.
version [required]The OCSF schema version.
version [required]The version of the custom mapping configuration.
type [required]The processor type. The value should always be ocsf_mapper.
ocsf_mapperocsf_mapper<type=parse_grok>The parse_grok processor extracts structured fields from unstructured log messages using Grok patterns.
Supported pipeline types: logs
disable_library_rulesIf set to true, disables the default Grok rules provided by Datadog.
falsedisplay_nameThe display name for a component.
enabled [required]Indicates whether the processor is enabled.
fieldThe log field to parse with the Grok rules.
messageid [required]A unique identifier for this processor.
include [required]A Datadog search query used to determine which logs this processor targets.
rules [required]The list of Grok parsing rules selected by either source field or include query.
Object 1A Grok parsing rule used in the parse_grok processor. Each rule defines how to extract structured fields
from a specific log field using Grok patterns.
match_rules [required]A list of Grok parsing rules that define how to extract fields from the source field. Each rule must contain a name and a valid Grok pattern.
name [required]The name of the rule.
rule [required]The definition of the Grok rule.
source [required]The value of the source field in log events to be processed by the Grok rules.
support_rulesA list of Grok helper rules that can be referenced by the parsing rules.
name [required]The name of the Grok helper rule.
rule [required]The definition of the Grok helper rule.
Object 2A Grok parsing rule selected using the include query. Each rule defines how to extract structured fields
from logs matching a Datadog search query.
include [required]A Datadog search query used to determine which logs this Grok rule targets.
match_rules [required]A list of Grok parsing rules that define how to extract fields from matching logs. Each rule must contain a name and a valid Grok pattern.
name [required]The name of the rule.
rule [required]The definition of the Grok rule.
support_rulesA list of Grok helper rules that can be referenced by the parsing rules.
name [required]The name of the Grok helper rule.
rule [required]The definition of the Grok helper rule.
type [required]The processor type. The value should always be parse_grok.
parse_grokparse_grok<type=parse_json>The parse_json processor extracts JSON from a specified field and flattens it into the event. This is useful when logs contain embedded JSON as a string.
Supported pipeline types: logs
display_nameThe display name for a component.
enabled [required]Indicates whether the processor is enabled.
field [required]The name of the log field that contains a JSON string.
id [required]A unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
include [required]A Datadog search query used to determine which logs this processor targets.
type [required]The processor type. The value should always be parse_json.
parse_jsonparse_json<type=parse_xml>The parse_xml processor parses XML from a specified field and extracts it into the event.
Supported pipeline types: logs
always_use_text_keyWhether to always use a text key for element content.
attr_prefixThe prefix to use for XML attributes in the parsed output.
display_nameThe display name for a component.
enabled [required]Indicates whether the processor is enabled.
field [required]The name of the log field that contains an XML string.
id [required]The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the input to downstream components).
include [required]A Datadog search query used to determine which logs this processor targets.
include_attrWhether to include XML attributes in the parsed output.
parse_boolWhether to parse boolean values from strings.
parse_nullWhether to parse null values.
parse_numberWhether to parse numeric values from strings.
text_keyThe key name to use for text content within XML elements. Must be at least 1 character if specified.
type [required]The processor type. The value should always be parse_xml.
parse_xmlparse_xml<type=quota>The quota processor measures logging traffic for logs that match a specified filter. When the configured daily quota is met, the processor can drop or alert.
Supported pipeline types: logs
display_nameThe display name for a component.
drop_eventsIf set to true, logs that match the quota filter and are sent after the quota is exceeded are dropped. Logs that do not match the filter continue through the pipeline. Note: You can set either drop_events or overflow_action, but not both.
enabled [required]Indicates whether the processor is enabled.
id [required]The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the input to downstream components).
ignore_when_missing_partitionsIf true, the processor skips quota checks when partition fields are missing from the logs.
include [required]A Datadog search query used to determine which logs this processor targets.
limit [required]The maximum amount of data or number of events allowed before the quota is enforced. Can be specified in bytes or events.
enforce [required]Unit for quota enforcement in bytes for data size or events for count.
bytes, eventslimit [required]The limit for quota enforcement.
name [required]Name of the quota.
overflow_actionThe action to take when the quota or bucket limit is exceeded. Options:
drop: Drop the event.no_action: Let the event pass through.overflow_routing: Route to an overflow destination.
When overflow_routing is used, there must be a destination whose inputs reference this processor with the <processor-id>.overflow_events suffix to route the overflowing events. Only the following destination types support overflow inputs: amazon_s3_generic, amazon_s3, google_cloud_storage, and azure_storage.
drop, no_action, overflow_routingoverridesA list of alternate quota rules that apply to specific sets of events, identified by matching field values. Each override can define a custom limit.
fields [required]A list of field matchers used to apply a specific override. If an event matches all listed key-value pairs, the corresponding override limit is enforced.
name [required]The field name.
value [required]The field value.
limit [required]The maximum amount of data or number of events allowed before the quota is enforced. Can be specified in bytes or events.
enforce [required]Unit for quota enforcement in bytes for data size or events for count.
bytes, eventslimit [required]The limit for quota enforcement.
partition_fieldsA list of fields used to segment log traffic for quota enforcement. Quotas are tracked independently by unique combinations of these field values.
too_many_buckets_actionThe action to take when the quota or bucket limit is exceeded. Options:
drop: Drop the event.no_action: Let the event pass through.overflow_routing: Route to an overflow destination.
When overflow_routing is used, there must be a destination whose inputs reference this processor with the <processor-id>.overflow_events suffix to route the overflowing events. Only the following destination types support overflow inputs: amazon_s3_generic, amazon_s3, google_cloud_storage, and azure_storage.
drop, no_action, overflow_routingtype [required]The processor type. The value should always be quota.
quotaquota<type=reduce>The reduce processor aggregates and merges logs based on matching keys and merge strategies.
Supported pipeline types: logs
display_nameThe display name for a component.
enabled [required]Indicates whether the processor is enabled.
group_by [required]A list of fields used to group log events for merging.
id [required]The unique identifier for this processor.
include [required]A Datadog search query used to determine which logs this processor targets.
merge_strategies [required]List of merge strategies defining how values from grouped events should be combined.
path [required]The field path in the log event.
strategy [required]The merge strategy to apply.
discard, retain, sum, max, min, array, concat, concat_newline, concat_raw, shortest_arrayShow 2 more
longest_array, flat_uniquetype [required]The processor type. The value should always be reduce.
reducereduce<type=remove_fields>The remove_fields processor deletes specified fields from logs.
Supported pipeline types: logs
display_nameThe display name for a component.
enabled [required]Indicates whether the processor is enabled.
fields [required]A list of field names to be removed from each log event.
id [required]The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the input to downstream components).
include [required]A Datadog search query used to determine which logs this processor targets.
type [required]The processor type. The value should always be remove_fields.
remove_fieldsremove_fields<type=rename_fields>The rename_fields processor changes field names.
Supported pipeline types: logs
display_nameThe display name for a component.
enabled [required]Indicates whether the processor is enabled.
fields [required]A list of rename rules specifying which fields to rename in the event, what to rename them to, and whether to preserve the original fields.
destination [required]The field name to assign the renamed value to.
preserve_source [required]Indicates whether the original field, that is received from the source, should be kept (true) or removed (false) after renaming.
source [required]The original field name in the log event that should be renamed.
id [required]A unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
include [required]A Datadog search query used to determine which logs this processor targets.
type [required]The processor type. The value should always be rename_fields.
rename_fieldsrename_fields<type=split_array>The split_array processor splits array fields into separate events based on configured rules.
Supported pipeline types: logs
arrays [required]A list of array split configurations.
field [required]The path to the array field to split.
include [required]A Datadog search query used to determine which logs this array split operation targets.
display_nameThe display name for a component.
enabled [required]Indicates whether the processor is enabled.
id [required]The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the input to downstream components).
include [required]A Datadog search query used to determine which logs this processor targets. For split_array, this should typically be *.
type [required]The processor type. The value should always be split_array.
split_arraysplit_array<type=throttle>The throttle processor limits the number of events that pass through over a given time window.
Supported pipeline types: logs
display_nameThe display name for a component.
enabled [required]Indicates whether the processor is enabled.
group_byOptional list of fields used to group events before the threshold has been reached.
id [required]The unique identifier for this processor.
include [required]A Datadog search query used to determine which logs this processor targets.
threshold [required]the number of events allowed in a given time window. Events sent after the threshold has been reached, are dropped.
type [required]The processor type. The value should always be throttle.
throttlethrottlewindow [required]The time window in seconds over which the threshold applies.
<type=custom_processor>The custom_processor processor transforms events using Vector Remap Language (VRL) scripts with advanced filtering capabilities.
Supported pipeline types: logs, traces
display_nameThe display name for a component.
enabled [required]Indicates whether the processor is enabled.
id [required]The unique identifier for this processor.
include [required]A Datadog search query used to determine which events this processor targets. This field should always be set to * for the custom_processor processor.
*remaps [required]Array of VRL remap rules.
drop_on_error [required]Whether to drop events that caused errors during processing.
enabledWhether this remap rule is enabled.
include [required]A Datadog search query used to filter events for this specific remap rule.
name [required]A descriptive name for this remap rule.
source [required]The VRL script source code that defines the processing logic.
type [required]The processor type. The value should always be custom_processor.
custom_processorcustom_processor<type=generate_metrics>The generate_metrics processor creates custom metrics from logs or traces.
Metrics can be counters, gauges, or distributions and optionally grouped by event fields.
There must be a destination whose inputs reference this processor with the <processor-id>.metrics suffix to route the generated metrics. All destination types normally supported for metrics pipelines are also supported as metrics destinations in logs and traces pipelines.
Supported pipeline types: logs, traces
display_nameThe display name for a component.
enabled [required]Indicates whether the processor is enabled.
enabled_metric_aggregation[Recommended] Indicates whether generated metrics are automatically aggregated and enriched with the worker_uuid tag before being sent to the metric destination.
trueid [required]The unique identifier for this component. Used to reference this component in other parts of the pipeline.
includeA Datadog search query used to determine which events this processor targets.
metricsConfiguration for generating individual metrics.
group_byOptional fields used to group the metric series.
include [required]Datadog filter query to match events for metric generation.
metric_type [required]Type of metric to create.
count, gauge, distributionname [required]Name of the custom metric to be created.
value [required]Specifies how the value of the generated metric is computed.
<strategy=increment_by_one>Strategy that increments a generated metric by one for each matching event.
strategy [required]Increments the metric by 1 for each matching event.
increment_by_one<strategy=increment_by_field>Strategy that increments a generated metric based on the value of an event field.
field [required]Name of the field containing the numeric value to increment the metric by.
strategy [required]Uses a numeric field in the log event as the metric increment.
increment_by_fieldtype [required]The processor type. Always generate_metrics.
generate_metricsgenerate_metrics<type=sample>The sample processor allows probabilistic sampling of logs or traces at a fixed rate.
Supported pipeline types: logs, traces
display_nameThe display name for a component.
enabled [required]Indicates whether the processor is enabled.
group_byOptional list of fields to group events by. Each group is sampled independently.
id [required]The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the input to downstream components).
include [required]A Datadog search query used to determine which events this processor targets.
percentage [required]The percentage of events to sample.
type [required]The processor type. The value should always be sample.
samplesample<type=sensitive_data_scanner>The sensitive_data_scanner processor detects and optionally redacts sensitive data in log or trace events.
Supported pipeline types: logs, traces
display_nameThe display name for a component.
enabled [required]Indicates whether the processor is enabled.
id [required]The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the input to downstream components).
include [required]A Datadog search query used to determine which events this processor targets.
rules [required]A list of rules for identifying and acting on sensitive data patterns.
keyword_optionsConfiguration for keywords used to reinforce sensitive data pattern detection.
keywords [required]A list of keywords to match near the sensitive pattern.
proximity [required]Maximum number of tokens between a keyword and a sensitive value match.
name [required]A name identifying the rule.
on_match [required]Defines what action to take when sensitive data is matched.
<action=redact>Configuration for completely redacting matched sensitive data.
action [required]Action type that completely replaces the matched sensitive data with a fixed replacement string to remove all visibility.
redactoptions [required]Configuration for fully redacting sensitive data.
replace [required]The string used to replace matched sensitive data (for example, "***" or "[REDACTED]").
<action=hash>Configuration for hashing matched sensitive values.
action [required]Action type that replaces the matched sensitive data with a hashed representation, preserving structure while securing content.
hashoptionsOptional settings for the hash action. When omitted or empty, matched sensitive data is replaced with a deterministic hashed value that preserves structure for analytics while protecting the original content. Reserved for future hash configuration (for example, algorithm or salt).
<action=partial_redact>Configuration for partially redacting matched sensitive data.
action [required]Action type that redacts part of the sensitive data while preserving a configurable number of characters, typically used for masking purposes (e.g., show last 4 digits of a credit card).
partial_redactoptions [required]Controls how partial redaction is applied, including character count and direction.
characters [required]Number of characters to leave visible from the start or end of the matched value; the rest are redacted.
direction [required]Indicates whether to redact characters from the first or last part of the matched value.
first, lastpattern [required]Pattern detection configuration for identifying sensitive data using either a custom regex or a library reference.
<type=custom>Defines a custom regex-based pattern for identifying sensitive data in logs.
options [required]Options for defining a custom regex pattern.
descriptionHuman-readable description providing context about a sensitive data scanner rule
rule [required]A regular expression used to detect sensitive values. Must be a valid regex.
type [required]Indicates a custom regular expression is used for matching.
custom<type=library>Specifies a pattern from Datadog’s sensitive data detection library to match known sensitive data types.
options [required]Options for selecting a predefined library pattern and enabling keyword support.
descriptionHuman-readable description providing context about a sensitive data scanner rule
id [required]Identifier for a predefined pattern from the sensitive data scanner pattern library.
use_recommended_keywordsWhether to augment the pattern with recommended keywords (optional).
type [required]Indicates that a predefined library pattern is used.
libraryscope [required]Determines which parts of the log the pattern-matching rule should be applied to.
<target=include>Includes only specific fields for sensitive data scanning.
options [required]Fields to which the scope rule applies.
fields [required]List of log attribute names (field paths) to which the scope applies. Only these fields are included in or excluded from pattern matching.
target [required]Applies the rule only to included fields.
include<target=exclude>Excludes specific fields from sensitive data scanning.
options [required]Fields to which the scope rule applies.
fields [required]List of log attribute names (field paths) to which the scope applies. Only these fields are included in or excluded from pattern matching.
target [required]Excludes specific fields from processing.
exclude<target=all>Applies scanning across all available fields.
target [required]Applies the rule to all fields.
alltagsTags assigned to this rule for filtering and classification.
type [required]The processor type. The value should always be sensitive_data_scanner.
sensitive_data_scannersensitive_data_scanner<type=add_metric_tags>The add_metric_tags processor adds static tags to metrics.
Supported pipeline types: metrics
display_nameThe display name for a component.
enabled [required]Indicates whether the processor is enabled.
id [required]The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the input to downstream components).
include [required]A Datadog search query used to determine which metrics this processor targets.
tags [required]A list of static tags (key-value pairs) added to each metric processed by this component.
name [required]The field name.
value [required]The field value.
type [required]The processor type. The value must be add_metric_tags.
add_metric_tagsadd_metric_tags<type=aggregate>The aggregate processor combines metrics that share the same name and tags into a single metric over a configurable interval.
Supported pipeline types: metrics
aggregation_timingConfigures how metrics are assigned to aggregation windows. When omitted, metrics are grouped using system time.
allowed_lateness_secsGrace period, in seconds, for late-arriving metrics when using event time. Defaults to 10 seconds when omitted.
type [required]Determines whether metrics are assigned to aggregation windows based on when they are processed or their timestamps.
system_time, event_timedisplay_nameThe display name for a component.
enabled [required]Indicates whether the processor is enabled.
id [required]The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the input to downstream components).
include [required]A Datadog search query used to determine which metrics this processor targets.
interval_secs [required]The interval, in seconds, over which metrics are aggregated.
mode [required]The aggregation mode applied to metrics that share the same name and tags within the interval.
auto, sum, latest, count, max, min, meantype [required]The processor type. The value must be aggregate.
aggregateaggregateObject 25The enrichment_table processor enriches metrics with tags from a static CSV file or a Datadog reference table.
It looks up a row using the metric name or a metric tag value. It then adds each column of the matching row as a
metric tag, overwriting any existing tag with the same key. Exactly one of file or reference_table must be
configured.
Supported pipeline types: metrics
Object 1An enrichment_table processor that enriches metrics using a static CSV file.
display_nameThe display name for a component.
enabled [required]Indicates whether the processor is enabled.
file [required]Defines a static enrichment table loaded from a CSV file for metric enrichment.
encoding [required]File encoding format.
delimiter [required]The encoding delimiter.
includes_headers [required]The encoding includes_headers.
type [required]Specifies the encoding format (e.g., CSV) used for enrichment tables.
csvkey [required]Defines how to map a metric lookup value to a CSV column during enrichment table lookups.
column [required]The CSV column name or index to match against the lookup value.
source [required]Specifies the source of the key value used for metric enrichment table lookups. The lookup key can be either the metric name or a metric tag.
<type=metric_name>Uses the metric name as the lookup key for enrichment table matching.
<type=tag>Uses a metric tag as the lookup key for enrichment table matching.
path [required]Path to the CSV file.
id [required]The unique identifier for this component. Used in other parts of the pipeline to reference this component
(for example, as the input to downstream components).
include [required]A Datadog search query used to determine which metrics this processor targets.
type [required]The processor type. The value should always be enrichment_table.
enrichment_tableenrichment_tableObject 2An enrichment_table processor that enriches metrics using a Datadog reference table.
display_nameThe display name for a component.
enabled [required]Indicates whether the processor is enabled.
id [required]The unique identifier for this component. Used in other parts of the pipeline to reference this component
(for example, as the input to downstream components).
include [required]A Datadog search query used to determine which metrics this processor targets.
reference_table [required]Uses a Datadog reference table to enrich metrics.
app_key_keyThe name of the environment variable or secret that holds the Datadog application key used to access the reference table.
columnsA list of column names to include from the reference table. If not provided, all columns are included.
key [required]Defines the metric lookup value used as the reference-table row ID.
source [required]Specifies the source of the key value used for metric enrichment table lookups. The lookup key can be either the metric name or a metric tag.
<type=metric_name>Uses the metric name as the lookup key for enrichment table matching.
<type=tag>Uses a metric tag as the lookup key for enrichment table matching.
table_id [required]The unique identifier of the reference table.
type [required]The processor type. The value should always be enrichment_table.
enrichment_tableenrichment_table<type=metric_tags>The metric_tags processor filters metrics based on their tags using Datadog tag key patterns.
Supported pipeline types: metrics
display_nameThe display name for a component.
enabled [required]Indicates whether the processor is enabled.
id [required]The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the input to downstream components).
include [required]A Datadog search query that determines which metrics the processor targets.
rules [required]A list of rules for filtering metric tags.
action [required]The action to take on tags with matching keys.
include, excludeinclude [required]A Datadog search query used to determine which metrics this rule targets.
keys [required]A list of tag keys to include or exclude.
mode [required]The processing mode for tag filtering.
filtertype [required]The processor type. The value should always be metric_tags.
metric_tagsmetric_tags<type=rename_metric_tags>The rename_metric_tags processor changes the keys of tags on metrics.
Supported pipeline types: metrics
display_nameThe display name for a component.
enabled [required]Indicates whether the processor is enabled.
id [required]The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the input to downstream components).
include [required]A Datadog search query used to determine which metrics this processor targets.
tags [required]A list of rename rules specifying which tag keys to rename on each metric.
rename_to [required]The new tag key to assign in place of the original.
tag [required]The original tag key on the metric event.
type [required]The processor type. The value must be rename_metric_tags.
rename_metric_tagsrename_metric_tags<type=tag_cardinality_limit>The tag_cardinality_limit processor caps the number of distinct tag value combinations on metrics, dropping tags or events once the limit is exceeded.
Supported pipeline types: metrics
display_nameThe display name for a component.
enabled [required]Indicates whether the processor is enabled.
id [required]The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the input to downstream components).
include [required]A Datadog search query used to determine which metrics this processor targets.
limit_exceeded_action [required]The action to take when the cardinality limit is exceeded.
drop_tag, drop_eventper_metric_limitsA list of per-metric cardinality overrides that take precedence over the default value_limit.
limit_exceeded_actionThe action to take when the cardinality limit is exceeded.
drop_tag, drop_eventmetric_name [required]The name of the metric this override applies to.
override_type [required]How the override is applied. limit_override enforces a custom limit; excluded omits the metric or tag from cardinality tracking.
limit_override, excludedper_tag_limitsA list of per-tag cardinality overrides that apply within this metric. Must be omitted when override_type is excluded.
override_type [required]How the override is applied. limit_override enforces a custom limit; excluded omits the metric or tag from cardinality tracking.
limit_override, excludedtag_key [required]The tag key this override applies to.
value_limitThe maximum number of distinct values allowed for this tag. Required when override_type is limit_override. Must be omitted when override_type is excluded.
value_limitThe maximum number of distinct tag value combinations allowed for this metric. Required when override_type is limit_override. Must be omitted when override_type is excluded.
tracking_mode [required]Controls whether the processor uses exact or probabilistic tag tracking.
mode [required]The cardinality tracking algorithm to use.
exact_fingerprint, probabilistictype [required]The processor type. The value must be tag_cardinality_limit.
tag_cardinality_limittag_cardinality_limitvalue_limit [required]The default maximum number of distinct tag value combinations allowed per metric.
processorsDEPRECATED A list of processor groups that transform or enrich log data.
Deprecated: This field is deprecated, you should now use the processor_groups field.
display_nameThe display name for a component.
enabled [required]Whether this processor group is enabled.
id [required]The unique identifier for the processor group.
include [required]Conditional expression for when this processor group should execute.
inputs [required]A list of IDs for components whose output is used as the input for this processor group.
processors [required]Processors applied sequentially within this group. Events flow through each processor in order.
<type=filter>The filter processor allows conditional processing of logs, metrics, or traces based on a Datadog search query. Events that match the include query are passed through; others are discarded.
Supported pipeline types: logs, metrics, traces
display_nameThe display name for a component.
enabled [required]Indicates whether the processor is enabled.
id [required]The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the input to downstream components).
include [required]A Datadog search query used to determine which events should pass through the filter. Events that match this query continue to downstream components; others are dropped.
type [required]The processor type. The value should always be filter.
filterfilter<type=add_env_vars>The add_env_vars processor adds environment variable values to log events.
Supported pipeline types: logs
display_nameThe display name for a component.
enabled [required]Indicates whether the processor is enabled.
id [required]The unique identifier for this component. Used to reference this processor in the pipeline.
include [required]A Datadog search query used to determine which logs this processor targets.
type [required]The processor type. The value should always be add_env_vars.
add_env_varsadd_env_varsvariables [required]A list of environment variable mappings to apply to log fields.
field [required]The target field in the log event.
name [required]The name of the environment variable to read.
<type=add_fields>The add_fields processor adds static key-value fields to logs.
Supported pipeline types: logs
display_nameThe display name for a component.
enabled [required]Indicates whether the processor is enabled.
fields [required]A list of static fields (key-value pairs) that is added to each log event processed by this component.
name [required]The field name.
value [required]The field value.
id [required]The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the input to downstream components).
include [required]A Datadog search query used to determine which logs this processor targets.
type [required]The processor type. The value should always be add_fields.
add_fieldsadd_fields<type=add_hostname>The add_hostname processor adds the hostname to log events.
Supported pipeline types: logs
display_nameThe display name for a component.
enabled [required]Indicates whether the processor is enabled.
id [required]The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the input to downstream components).
include [required]A Datadog search query used to determine which logs this processor targets.
type [required]The processor type. The value should always be add_hostname.
add_hostnameadd_hostname<type=datadog_tags>The datadog_tags processor includes or excludes specific Datadog tags in your logs.
Supported pipeline types: logs
action [required]The action to take on tags with matching keys.
include, excludedisplay_nameThe display name for a component.
enabled [required]Indicates whether the processor is enabled.
id [required]The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the input to downstream components).
include [required]A Datadog search query used to determine which logs this processor targets.
keys [required]A list of tag keys.
mode [required]The processing mode.
filtertype [required]The processor type. The value should always be datadog_tags.
datadog_tagsdatadog_tags<type=dedupe>The dedupe processor removes duplicate fields in log events.
Supported pipeline types: logs
cacheConfiguration for the cache used to detect duplicates.
num_events [required]The number of events to cache for duplicate detection.
display_nameThe display name for a component.
enabled [required]Indicates whether the processor is enabled.
fields [required]A list of log field paths to check for duplicates.
id [required]The unique identifier for this processor.
include [required]A Datadog search query used to determine which logs this processor targets.
mode [required]The deduplication mode to apply to the fields.
match, ignoretype [required]The processor type. The value should always be dedupe.
dedupededupe<type=enrichment_table>The enrichment_table processor enriches logs using a static CSV file, GeoIP database, or reference table. Exactly one of file, geoip, or reference_table must be configured.
Supported pipeline types: logs
display_nameThe display name for a component.
enabled [required]Indicates whether the processor is enabled.
fileDefines a static enrichment table loaded from a CSV file.
encoding [required]File encoding format.
delimiter [required]The encoding delimiter.
includes_headers [required]The encoding includes_headers.
type [required]Specifies the encoding format (e.g., CSV) used for enrichment tables.
csvkey [required]Key fields used to look up enrichment values.
column [required]The items column.
comparison [required]Defines how to compare key fields for enrichment table lookups.
equalsfield [required]Specifies the source of the key value used for enrichment table lookups.
Can be a plain field path string or an object specifying event, vrl, or secret.
Object 1A plain field path in the log event, used as the lookup key.
Object 2Looks up a value from a field path in the log event.
event [required]The path to the field in the log event to use as the lookup key.
Object 3Evaluates a VRL expression to produce the lookup key.
vrl [required]A VRL expression that returns the value to use as the lookup key.
Object 4Looks up a value stored as a pipeline secret.
secret [required]The name of the secret containing the lookup key value.
path [required]Path to the CSV file.
schema [required]Schema defining column names and their types.
column [required]The items column.
type [required]Declares allowed data types for enrichment table columns.
string, boolean, integer, float, date, timestampgeoipUses a GeoIP database to enrich logs based on an IP field.
key_field [required]Path to the IP field in the log.
locale [required]Locale used to resolve geographical names.
path [required]Path to the GeoIP database file.
id [required]The unique identifier for this processor.
include [required]A Datadog search query used to determine which logs this processor targets.
reference_tableUses a Datadog reference table to enrich logs.
app_key_keyName of the environment variable or secret that holds the Datadog application key used to access the reference table.
columnsList of column names to include from the reference table. If not provided, all columns are included.
key_field [required]Path to the field in the log event to match against the reference table.
table_id [required]The unique identifier of the reference table.
target [required]Path where enrichment results should be stored in the log.
type [required]The processor type. The value should always be enrichment_table.
enrichment_tableenrichment_table<type=generate_datadog_metrics>DEPRECATED The generate_datadog_metrics processor creates custom metrics from logs and sends them to Datadog.
Metrics can be counters, gauges, or distributions and optionally grouped by log fields.
Supported pipeline types: logs
Deprecated: This processor is deprecated. Use the generate_metrics processor instead.
display_nameThe display name for a component.
enabled [required]Indicates whether the processor is enabled.
id [required]The unique identifier for this component. Used to reference this component in other parts of the pipeline.
includeA Datadog search query used to determine which logs this processor targets.
metricsConfiguration for generating individual metrics.
group_byOptional fields used to group the metric series.
include [required]Datadog filter query to match events for metric generation.
metric_type [required]Type of metric to create.
count, gauge, distributionname [required]Name of the custom metric to be created.
value [required]Specifies how the value of the generated metric is computed.
<strategy=increment_by_one>Strategy that increments a generated metric by one for each matching event.
strategy [required]Increments the metric by 1 for each matching event.
increment_by_one<strategy=increment_by_field>Strategy that increments a generated metric based on the value of an event field.
field [required]Name of the field containing the numeric value to increment the metric by.
strategy [required]Uses a numeric field in the log event as the metric increment.
increment_by_fieldtype [required]The processor type. Always generate_datadog_metrics.
generate_datadog_metricsgenerate_datadog_metrics<type=ocsf_mapper>The ocsf_mapper processor transforms logs into the OCSF schema using a predefined mapping configuration.
Supported pipeline types: logs
display_nameThe display name for a component.
enabled [required]Indicates whether the processor is enabled.
id [required]The unique identifier for this component. Used to reference this component in other parts of the pipeline.
include [required]A Datadog search query used to determine which logs this processor targets.
keep_unmatchedWhether to keep an event that does not match any of the mapping filters.
mappings [required]A list of mapping rules to convert events to the OCSF format.
include [required]A Datadog search query used to select the logs that this mapping should apply to.
mapping [required]Defines a single mapping rule for transforming logs into the OCSF schema.
Object 1Predefined library mappings for common log formats.
CloudTrail Account Change, GCP Cloud Audit CreateBucket, GCP Cloud Audit CreateSink, GCP Cloud Audit SetIamPolicy, GCP Cloud Audit UpdateSink, Github Audit Log API Activity, Google Workspace Admin Audit addPrivilege, Microsoft 365 Defender Incident, Microsoft 365 Defender UserLoggedIn, Okta System Log AuthenticationShow 1 more
Palo Alto Networks Firewall TrafficObject 2Custom OCSF mapping configuration for transforming logs.
mapping [required]A list of field mapping rules for transforming log fields to OCSF schema fields.
defaultThe default value to use if the source field is missing or empty.
dest [required]The destination OCSF field path.
lookupLookup table configuration for mapping source values to destination values.
sourceThe source field path from the log event.
sourcesMultiple source field paths for combined mapping.
valueA static value to use for the destination field.
metadata [required]Metadata for the custom OCSF mapping.
class [required]The OCSF event class name.
profilesA list of OCSF profiles to apply.
version [required]The OCSF schema version.
version [required]The version of the custom mapping configuration.
type [required]The processor type. The value should always be ocsf_mapper.
ocsf_mapperocsf_mapper<type=parse_grok>The parse_grok processor extracts structured fields from unstructured log messages using Grok patterns.
Supported pipeline types: logs
disable_library_rulesIf set to true, disables the default Grok rules provided by Datadog.
falsedisplay_nameThe display name for a component.
enabled [required]Indicates whether the processor is enabled.
fieldThe log field to parse with the Grok rules.
messageid [required]A unique identifier for this processor.
include [required]A Datadog search query used to determine which logs this processor targets.
rules [required]The list of Grok parsing rules selected by either source field or include query.
Object 1A Grok parsing rule used in the parse_grok processor. Each rule defines how to extract structured fields
from a specific log field using Grok patterns.
match_rules [required]A list of Grok parsing rules that define how to extract fields from the source field. Each rule must contain a name and a valid Grok pattern.
name [required]The name of the rule.
rule [required]The definition of the Grok rule.
source [required]The value of the source field in log events to be processed by the Grok rules.
support_rulesA list of Grok helper rules that can be referenced by the parsing rules.
name [required]The name of the Grok helper rule.
rule [required]The definition of the Grok helper rule.
Object 2A Grok parsing rule selected using the include query. Each rule defines how to extract structured fields
from logs matching a Datadog search query.
include [required]A Datadog search query used to determine which logs this Grok rule targets.
match_rules [required]A list of Grok parsing rules that define how to extract fields from matching logs. Each rule must contain a name and a valid Grok pattern.
name [required]The name of the rule.
rule [required]The definition of the Grok rule.
support_rulesA list of Grok helper rules that can be referenced by the parsing rules.
name [required]The name of the Grok helper rule.
rule [required]The definition of the Grok helper rule.
type [required]The processor type. The value should always be parse_grok.
parse_grokparse_grok<type=parse_json>The parse_json processor extracts JSON from a specified field and flattens it into the event. This is useful when logs contain embedded JSON as a string.
Supported pipeline types: logs
display_nameThe display name for a component.
enabled [required]Indicates whether the processor is enabled.
field [required]The name of the log field that contains a JSON string.
id [required]A unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
include [required]A Datadog search query used to determine which logs this processor targets.
type [required]The processor type. The value should always be parse_json.
parse_jsonparse_json<type=parse_xml>The parse_xml processor parses XML from a specified field and extracts it into the event.
Supported pipeline types: logs
always_use_text_keyWhether to always use a text key for element content.
attr_prefixThe prefix to use for XML attributes in the parsed output.
display_nameThe display name for a component.
enabled [required]Indicates whether the processor is enabled.
field [required]The name of the log field that contains an XML string.
id [required]The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the input to downstream components).
include [required]A Datadog search query used to determine which logs this processor targets.
include_attrWhether to include XML attributes in the parsed output.
parse_boolWhether to parse boolean values from strings.
parse_nullWhether to parse null values.
parse_numberWhether to parse numeric values from strings.
text_keyThe key name to use for text content within XML elements. Must be at least 1 character if specified.
type [required]The processor type. The value should always be parse_xml.
parse_xmlparse_xml<type=quota>The quota processor measures logging traffic for logs that match a specified filter. When the configured daily quota is met, the processor can drop or alert.
Supported pipeline types: logs
display_nameThe display name for a component.
drop_eventsIf set to true, logs that match the quota filter and are sent after the quota is exceeded are dropped. Logs that do not match the filter continue through the pipeline. Note: You can set either drop_events or overflow_action, but not both.
enabled [required]Indicates whether the processor is enabled.
id [required]The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the input to downstream components).
ignore_when_missing_partitionsIf true, the processor skips quota checks when partition fields are missing from the logs.
include [required]A Datadog search query used to determine which logs this processor targets.
limit [required]The maximum amount of data or number of events allowed before the quota is enforced. Can be specified in bytes or events.
enforce [required]Unit for quota enforcement in bytes for data size or events for count.
bytes, eventslimit [required]The limit for quota enforcement.
name [required]Name of the quota.
overflow_actionThe action to take when the quota or bucket limit is exceeded. Options:
drop: Drop the event.no_action: Let the event pass through.overflow_routing: Route to an overflow destination.
When overflow_routing is used, there must be a destination whose inputs reference this processor with the <processor-id>.overflow_events suffix to route the overflowing events. Only the following destination types support overflow inputs: amazon_s3_generic, amazon_s3, google_cloud_storage, and azure_storage.
drop, no_action, overflow_routingoverridesA list of alternate quota rules that apply to specific sets of events, identified by matching field values. Each override can define a custom limit.
fields [required]A list of field matchers used to apply a specific override. If an event matches all listed key-value pairs, the corresponding override limit is enforced.
name [required]The field name.
value [required]The field value.
limit [required]The maximum amount of data or number of events allowed before the quota is enforced. Can be specified in bytes or events.
enforce [required]Unit for quota enforcement in bytes for data size or events for count.
bytes, eventslimit [required]The limit for quota enforcement.
partition_fieldsA list of fields used to segment log traffic for quota enforcement. Quotas are tracked independently by unique combinations of these field values.
too_many_buckets_actionThe action to take when the quota or bucket limit is exceeded. Options:
drop: Drop the event.no_action: Let the event pass through.overflow_routing: Route to an overflow destination.
When overflow_routing is used, there must be a destination whose inputs reference this processor with the <processor-id>.overflow_events suffix to route the overflowing events. Only the following destination types support overflow inputs: amazon_s3_generic, amazon_s3, google_cloud_storage, and azure_storage.
drop, no_action, overflow_routingtype [required]The processor type. The value should always be quota.
quotaquota<type=reduce>The reduce processor aggregates and merges logs based on matching keys and merge strategies.
Supported pipeline types: logs
display_nameThe display name for a component.
enabled [required]Indicates whether the processor is enabled.
group_by [required]A list of fields used to group log events for merging.
id [required]The unique identifier for this processor.
include [required]A Datadog search query used to determine which logs this processor targets.
merge_strategies [required]List of merge strategies defining how values from grouped events should be combined.
path [required]The field path in the log event.
strategy [required]The merge strategy to apply.
discard, retain, sum, max, min, array, concat, concat_newline, concat_raw, shortest_arrayShow 2 more
longest_array, flat_uniquetype [required]The processor type. The value should always be reduce.
reducereduce<type=remove_fields>The remove_fields processor deletes specified fields from logs.
Supported pipeline types: logs
display_nameThe display name for a component.
enabled [required]Indicates whether the processor is enabled.
fields [required]A list of field names to be removed from each log event.
id [required]The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the input to downstream components).
include [required]A Datadog search query used to determine which logs this processor targets.
type [required]The processor type. The value should always be remove_fields.
remove_fieldsremove_fields<type=rename_fields>The rename_fields processor changes field names.
Supported pipeline types: logs
display_nameThe display name for a component.
enabled [required]Indicates whether the processor is enabled.
fields [required]A list of rename rules specifying which fields to rename in the event, what to rename them to, and whether to preserve the original fields.
destination [required]The field name to assign the renamed value to.
preserve_source [required]Indicates whether the original field, that is received from the source, should be kept (true) or removed (false) after renaming.
source [required]The original field name in the log event that should be renamed.
id [required]A unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components).
include [required]A Datadog search query used to determine which logs this processor targets.
type [required]The processor type. The value should always be rename_fields.
rename_fieldsrename_fields<type=split_array>The split_array processor splits array fields into separate events based on configured rules.
Supported pipeline types: logs
arrays [required]A list of array split configurations.
field [required]The path to the array field to split.
include [required]A Datadog search query used to determine which logs this array split operation targets.
display_nameThe display name for a component.
enabled [required]Indicates whether the processor is enabled.
id [required]The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the input to downstream components).
include [required]A Datadog search query used to determine which logs this processor targets. For split_array, this should typically be *.
type [required]The processor type. The value should always be split_array.
split_arraysplit_array<type=throttle>The throttle processor limits the number of events that pass through over a given time window.
Supported pipeline types: logs
display_nameThe display name for a component.
enabled [required]Indicates whether the processor is enabled.
group_byOptional list of fields used to group events before the threshold has been reached.
id [required]The unique identifier for this processor.
include [required]A Datadog search query used to determine which logs this processor targets.
threshold [required]the number of events allowed in a given time window. Events sent after the threshold has been reached, are dropped.
type [required]The processor type. The value should always be throttle.
throttlethrottlewindow [required]The time window in seconds over which the threshold applies.
<type=custom_processor>The custom_processor processor transforms events using Vector Remap Language (VRL) scripts with advanced filtering capabilities.
Supported pipeline types: logs, traces
display_nameThe display name for a component.
enabled [required]Indicates whether the processor is enabled.
id [required]The unique identifier for this processor.
include [required]A Datadog search query used to determine which events this processor targets. This field should always be set to * for the custom_processor processor.
*remaps [required]Array of VRL remap rules.
drop_on_error [required]Whether to drop events that caused errors during processing.
enabledWhether this remap rule is enabled.
include [required]A Datadog search query used to filter events for this specific remap rule.
name [required]A descriptive name for this remap rule.
source [required]The VRL script source code that defines the processing logic.
type [required]The processor type. The value should always be custom_processor.
custom_processorcustom_processor<type=generate_metrics>The generate_metrics processor creates custom metrics from logs or traces.
Metrics can be counters, gauges, or distributions and optionally grouped by event fields.
There must be a destination whose inputs reference this processor with the <processor-id>.metrics suffix to route the generated metrics. All destination types normally supported for metrics pipelines are also supported as metrics destinations in logs and traces pipelines.
Supported pipeline types: logs, traces
display_nameThe display name for a component.
enabled [required]Indicates whether the processor is enabled.
enabled_metric_aggregation[Recommended] Indicates whether generated metrics are automatically aggregated and enriched with the worker_uuid tag before being sent to the metric destination.
trueid [required]The unique identifier for this component. Used to reference this component in other parts of the pipeline.
includeA Datadog search query used to determine which events this processor targets.
metricsConfiguration for generating individual metrics.
group_byOptional fields used to group the metric series.
include [required]Datadog filter query to match events for metric generation.
metric_type [required]Type of metric to create.
count, gauge, distributionname [required]Name of the custom metric to be created.
value [required]Specifies how the value of the generated metric is computed.
<strategy=increment_by_one>Strategy that increments a generated metric by one for each matching event.
strategy [required]Increments the metric by 1 for each matching event.
increment_by_one<strategy=increment_by_field>Strategy that increments a generated metric based on the value of an event field.
field [required]Name of the field containing the numeric value to increment the metric by.
strategy [required]Uses a numeric field in the log event as the metric increment.
increment_by_fieldtype [required]The processor type. Always generate_metrics.
generate_metricsgenerate_metrics<type=sample>The sample processor allows probabilistic sampling of logs or traces at a fixed rate.
Supported pipeline types: logs, traces
display_nameThe display name for a component.
enabled [required]Indicates whether the processor is enabled.
group_byOptional list of fields to group events by. Each group is sampled independently.
id [required]The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the input to downstream components).
include [required]A Datadog search query used to determine which events this processor targets.
percentage [required]The percentage of events to sample.
type [required]The processor type. The value should always be sample.
samplesample<type=sensitive_data_scanner>The sensitive_data_scanner processor detects and optionally redacts sensitive data in log or trace events.
Supported pipeline types: logs, traces
display_nameThe display name for a component.
enabled [required]Indicates whether the processor is enabled.
id [required]The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the input to downstream components).
include [required]A Datadog search query used to determine which events this processor targets.
rules [required]A list of rules for identifying and acting on sensitive data patterns.
keyword_optionsConfiguration for keywords used to reinforce sensitive data pattern detection.
keywords [required]A list of keywords to match near the sensitive pattern.
proximity [required]Maximum number of tokens between a keyword and a sensitive value match.
name [required]A name identifying the rule.
on_match [required]Defines what action to take when sensitive data is matched.
<action=redact>Configuration for completely redacting matched sensitive data.
action [required]Action type that completely replaces the matched sensitive data with a fixed replacement string to remove all visibility.
redactoptions [required]Configuration for fully redacting sensitive data.
replace [required]The string used to replace matched sensitive data (for example, "***" or "[REDACTED]").
<action=hash>Configuration for hashing matched sensitive values.
action [required]Action type that replaces the matched sensitive data with a hashed representation, preserving structure while securing content.
hashoptionsOptional settings for the hash action. When omitted or empty, matched sensitive data is replaced with a deterministic hashed value that preserves structure for analytics while protecting the original content. Reserved for future hash configuration (for example, algorithm or salt).
<action=partial_redact>Configuration for partially redacting matched sensitive data.
action [required]Action type that redacts part of the sensitive data while preserving a configurable number of characters, typically used for masking purposes (e.g., show last 4 digits of a credit card).
partial_redactoptions [required]Controls how partial redaction is applied, including character count and direction.
characters [required]Number of characters to leave visible from the start or end of the matched value; the rest are redacted.
direction [required]Indicates whether to redact characters from the first or last part of the matched value.
first, lastpattern [required]Pattern detection configuration for identifying sensitive data using either a custom regex or a library reference.
<type=custom>Defines a custom regex-based pattern for identifying sensitive data in logs.
options [required]Options for defining a custom regex pattern.
descriptionHuman-readable description providing context about a sensitive data scanner rule
rule [required]A regular expression used to detect sensitive values. Must be a valid regex.
type [required]Indicates a custom regular expression is used for matching.
custom<type=library>Specifies a pattern from Datadog’s sensitive data detection library to match known sensitive data types.
options [required]Options for selecting a predefined library pattern and enabling keyword support.
descriptionHuman-readable description providing context about a sensitive data scanner rule
id [required]Identifier for a predefined pattern from the sensitive data scanner pattern library.
use_recommended_keywordsWhether to augment the pattern with recommended keywords (optional).
type [required]Indicates that a predefined library pattern is used.
libraryscope [required]Determines which parts of the log the pattern-matching rule should be applied to.
<target=include>Includes only specific fields for sensitive data scanning.
options [required]Fields to which the scope rule applies.
fields [required]List of log attribute names (field paths) to which the scope applies. Only these fields are included in or excluded from pattern matching.
target [required]Applies the rule only to included fields.
include<target=exclude>Excludes specific fields from sensitive data scanning.
options [required]Fields to which the scope rule applies.
fields [required]List of log attribute names (field paths) to which the scope applies. Only these fields are included in or excluded from pattern matching.
target [required]Excludes specific fields from processing.
exclude<target=all>Applies scanning across all available fields.
target [required]Applies the rule to all fields.
alltagsTags assigned to this rule for filtering and classification.
type [required]The processor type. The value should always be sensitive_data_scanner.
sensitive_data_scannersensitive_data_scanner<type=add_metric_tags>The add_metric_tags processor adds static tags to metrics.
Supported pipeline types: metrics
display_nameThe display name for a component.
enabled [required]Indicates whether the processor is enabled.
id [required]The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the input to downstream components).
include [required]A Datadog search query used to determine which metrics this processor targets.
tags [required]A list of static tags (key-value pairs) added to each metric processed by this component.
name [required]The field name.
value [required]The field value.
type [required]The processor type. The value must be add_metric_tags.
add_metric_tagsadd_metric_tags<type=aggregate>The aggregate processor combines metrics that share the same name and tags into a single metric over a configurable interval.
Supported pipeline types: metrics
aggregation_timingConfigures how metrics are assigned to aggregation windows. When omitted, metrics are grouped using system time.
allowed_lateness_secsGrace period, in seconds, for late-arriving metrics when using event time. Defaults to 10 seconds when omitted.
type [required]Determines whether metrics are assigned to aggregation windows based on when they are processed or their timestamps.
system_time, event_timedisplay_nameThe display name for a component.
enabled [required]Indicates whether the processor is enabled.
id [required]The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the input to downstream components).
include [required]A Datadog search query used to determine which metrics this processor targets.
interval_secs [required]The interval, in seconds, over which metrics are aggregated.
mode [required]The aggregation mode applied to metrics that share the same name and tags within the interval.
auto, sum, latest, count, max, min, meantype [required]The processor type. The value must be aggregate.
aggregateaggregateObject 25The enrichment_table processor enriches metrics with tags from a static CSV file or a Datadog reference table.
It looks up a row using the metric name or a metric tag value. It then adds each column of the matching row as a
metric tag, overwriting any existing tag with the same key. Exactly one of file or reference_table must be
configured.
Supported pipeline types: metrics
Object 1An enrichment_table processor that enriches metrics using a static CSV file.
display_nameThe display name for a component.
enabled [required]Indicates whether the processor is enabled.
file [required]Defines a static enrichment table loaded from a CSV file for metric enrichment.
encoding [required]File encoding format.
delimiter [required]The encoding delimiter.
includes_headers [required]The encoding includes_headers.
type [required]Specifies the encoding format (e.g., CSV) used for enrichment tables.
csvkey [required]Defines how to map a metric lookup value to a CSV column during enrichment table lookups.
column [required]The CSV column name or index to match against the lookup value.
source [required]Specifies the source of the key value used for metric enrichment table lookups. The lookup key can be either the metric name or a metric tag.
<type=metric_name>Uses the metric name as the lookup key for enrichment table matching.
<type=tag>Uses a metric tag as the lookup key for enrichment table matching.
path [required]Path to the CSV file.
id [required]The unique identifier for this component. Used in other parts of the pipeline to reference this component
(for example, as the input to downstream components).
include [required]A Datadog search query used to determine which metrics this processor targets.
type [required]The processor type. The value should always be enrichment_table.
enrichment_tableenrichment_tableObject 2An enrichment_table processor that enriches metrics using a Datadog reference table.
display_nameThe display name for a component.
enabled [required]Indicates whether the processor is enabled.
id [required]The unique identifier for this component. Used in other parts of the pipeline to reference this component
(for example, as the input to downstream components).
include [required]A Datadog search query used to determine which metrics this processor targets.
reference_table [required]Uses a Datadog reference table to enrich metrics.
app_key_keyThe name of the environment variable or secret that holds the Datadog application key used to access the reference table.
columnsA list of column names to include from the reference table. If not provided, all columns are included.
key [required]Defines the metric lookup value used as the reference-table row ID.
source [required]Specifies the source of the key value used for metric enrichment table lookups. The lookup key can be either the metric name or a metric tag.
<type=metric_name>Uses the metric name as the lookup key for enrichment table matching.
<type=tag>Uses a metric tag as the lookup key for enrichment table matching.
table_id [required]The unique identifier of the reference table.
type [required]The processor type. The value should always be enrichment_table.
enrichment_tableenrichment_table<type=metric_tags>The metric_tags processor filters metrics based on their tags using Datadog tag key patterns.
Supported pipeline types: metrics
display_nameThe display name for a component.
enabled [required]Indicates whether the processor is enabled.
id [required]The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the input to downstream components).
include [required]A Datadog search query that determines which metrics the processor targets.
rules [required]A list of rules for filtering metric tags.
action [required]The action to take on tags with matching keys.
include, excludeinclude [required]A Datadog search query used to determine which metrics this rule targets.
keys [required]A list of tag keys to include or exclude.
mode [required]The processing mode for tag filtering.
filtertype [required]The processor type. The value should always be metric_tags.
metric_tagsmetric_tags<type=rename_metric_tags>The rename_metric_tags processor changes the keys of tags on metrics.
Supported pipeline types: metrics
display_nameThe display name for a component.
enabled [required]Indicates whether the processor is enabled.
id [required]The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the input to downstream components).
include [required]A Datadog search query used to determine which metrics this processor targets.
tags [required]A list of rename rules specifying which tag keys to rename on each metric.
rename_to [required]The new tag key to assign in place of the original.
tag [required]The original tag key on the metric event.
type [required]The processor type. The value must be rename_metric_tags.
rename_metric_tagsrename_metric_tags<type=tag_cardinality_limit>The tag_cardinality_limit processor caps the number of distinct tag value combinations on metrics, dropping tags or events once the limit is exceeded.
Supported pipeline types: metrics
display_nameThe display name for a component.
enabled [required]Indicates whether the processor is enabled.
id [required]The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the input to downstream components).
include [required]A Datadog search query used to determine which metrics this processor targets.
limit_exceeded_action [required]The action to take when the cardinality limit is exceeded.
drop_tag, drop_eventper_metric_limitsA list of per-metric cardinality overrides that take precedence over the default value_limit.
limit_exceeded_actionThe action to take when the cardinality limit is exceeded.
drop_tag, drop_eventmetric_name [required]The name of the metric this override applies to.
override_type [required]How the override is applied. limit_override enforces a custom limit; excluded omits the metric or tag from cardinality tracking.
limit_override, excludedper_tag_limitsA list of per-tag cardinality overrides that apply within this metric. Must be omitted when override_type is excluded.
override_type [required]How the override is applied. limit_override enforces a custom limit; excluded omits the metric or tag from cardinality tracking.
limit_override, excludedtag_key [required]The tag key this override applies to.
value_limitThe maximum number of distinct values allowed for this tag. Required when override_type is limit_override. Must be omitted when override_type is excluded.
value_limitThe maximum number of distinct tag value combinations allowed for this metric. Required when override_type is limit_override. Must be omitted when override_type is excluded.
tracking_mode [required]Controls whether the processor uses exact or probabilistic tag tracking.
mode [required]The cardinality tracking algorithm to use.
exact_fingerprint, probabilistictype [required]The processor type. The value must be tag_cardinality_limit.
tag_cardinality_limittag_cardinality_limitvalue_limit [required]The default maximum number of distinct tag value combinations allowed per metric.
sources [required]A list of configured data sources for the pipeline.
<type=datadog_agent>The datadog_agent source collects logs, metrics, or traces from the Datadog Agent.
Supported pipeline types: logs, metrics, traces
address_keyName of the environment variable or secret that holds the listen address for the Datadog Agent source.
id [required]The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the input to downstream components).
tlsConfiguration for enabling TLS encryption between the pipeline component and external services.
ca_filePath to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_filePath to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
key_pass_keyName of the environment variable or secret that holds the passphrase for the private key file.
type [required]The source type. The value should always be datadog_agent.
datadog_agentdatadog_agent<type=opentelemetry>The opentelemetry source receives telemetry data using the OpenTelemetry Protocol (OTLP) over gRPC and HTTP.
Supported pipeline types: logs, metrics, traces
grpc_address_keyEnvironment variable name containing the gRPC server address for receiving OTLP data. Must be a valid environment variable name (alphanumeric characters and underscores only).
http_address_keyEnvironment variable name containing the HTTP server address for receiving OTLP data. Must be a valid environment variable name (alphanumeric characters and underscores only).
id [required]The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the input to downstream components).
tlsConfiguration for enabling TLS encryption between the pipeline component and external connecting clients.
ca_filePath to the Certificate Authority (CA) file used to validate connecting clients' TLS certificates.
crt_file [required]Path to the TLS server certificate file used to used to identify the pipeline component to connecting clients.
key_filePath to the private key file associated with the TLS server certificate.
key_pass_keyName of the environment variable or secret that holds the passphrase for the private key file.
verify_certificateWhen true, requires client connections to present a valid certificate, enabling mutual TLS authentication.
type [required]The source type. The value should always be opentelemetry.
opentelemetryopentelemetry<type=amazon_data_firehose>The amazon_data_firehose source ingests logs from AWS Data Firehose.
Supported pipeline types: logs
address_keyName of the environment variable or secret that holds the Firehose delivery stream address.
authAWS authentication credentials used for accessing AWS services such as S3. If omitted, the system’s default credentials are used (for example, the IAM role and environment variables).
assume_roleThe Amazon Resource Name (ARN) of the role to assume.
external_idA unique identifier for cross-account role assumption.
session_nameA session identifier used for logging and tracing the assumed role session.
id [required]The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the input to downstream components).
tlsConfiguration for enabling TLS encryption between the pipeline component and external services.
ca_filePath to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_filePath to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
key_pass_keyName of the environment variable or secret that holds the passphrase for the private key file.
type [required]The source type. The value should always be amazon_data_firehose.
amazon_data_firehoseamazon_data_firehose<type=amazon_s3>The amazon_s3 source ingests logs from an Amazon S3 bucket.
It supports AWS authentication, TLS encryption, and configurable compression.
Supported pipeline types: logs
authAWS authentication credentials used for accessing AWS services such as S3. If omitted, the system’s default credentials are used (for example, the IAM role and environment variables).
assume_roleThe Amazon Resource Name (ARN) of the role to assume.
external_idA unique identifier for cross-account role assumption.
session_nameA session identifier used for logging and tracing the assumed role session.
compressionCompression format for objects retrieved from the S3 bucket. Use auto to detect compression from the object's Content-Encoding header or file extension.
auto, none, gzip, zstdid [required]The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the input to downstream components).
region [required]AWS region where the S3 bucket resides.
tlsConfiguration for enabling TLS encryption between the pipeline component and external services.
ca_filePath to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_filePath to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
key_pass_keyName of the environment variable or secret that holds the passphrase for the private key file.
type [required]The source type. Always amazon_s3.
amazon_s3amazon_s3url_keyName of the environment variable or secret that holds the S3 bucket URL.
<type=fluent_bit>The fluent_bit source ingests logs from Fluent Bit.
Supported pipeline types: logs
address_keyName of the environment variable or secret that holds the listen address for the Fluent Bit receiver.
id [required]The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the input to downstream components).
tlsConfiguration for enabling TLS encryption between the pipeline component and external connecting clients.
ca_filePath to the Certificate Authority (CA) file used to validate connecting clients' TLS certificates.
crt_file [required]Path to the TLS server certificate file used to used to identify the pipeline component to connecting clients.
key_filePath to the private key file associated with the TLS server certificate.
key_pass_keyName of the environment variable or secret that holds the passphrase for the private key file.
verify_certificateWhen true, requires client connections to present a valid certificate, enabling mutual TLS authentication.
type [required]The source type. The value should always be fluent_bit.
fluent_bitfluent_bit<type=fluentd>The fluentd source ingests logs from a Fluentd-compatible service.
Supported pipeline types: logs
address_keyName of the environment variable or secret that holds the listen address for the Fluent receiver.
id [required]The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the input to downstream components).
tlsConfiguration for enabling TLS encryption between the pipeline component and external connecting clients.
ca_filePath to the Certificate Authority (CA) file used to validate connecting clients' TLS certificates.
crt_file [required]Path to the TLS server certificate file used to used to identify the pipeline component to connecting clients.
key_filePath to the private key file associated with the TLS server certificate.
key_pass_keyName of the environment variable or secret that holds the passphrase for the private key file.
verify_certificateWhen true, requires client connections to present a valid certificate, enabling mutual TLS authentication.
type [required]The source type. The value should always be `fluentd.
fluentdfluentd<type=google_pubsub>The google_pubsub source ingests logs from a Google Cloud Pub/Sub subscription.
Supported pipeline types: logs
authGoogle Cloud credentials used to authenticate with Google Cloud Storage.
credentials_file [required]Path to the Google Cloud service account key file.
decoding [required]The decoding format used to interpret incoming logs.
bytes, gelf, json, syslogid [required]The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the input to downstream components).
project [required]The Google Cloud project ID that owns the Pub/Sub subscription.
subscription [required]The Pub/Sub subscription name from which messages are consumed.
tlsConfiguration for enabling TLS encryption between the pipeline component and external services.
ca_filePath to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_filePath to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
key_pass_keyName of the environment variable or secret that holds the passphrase for the private key file.
type [required]The source type. The value should always be google_pubsub.
google_pubsubgoogle_pubsub<type=http_client>The http_client source scrapes logs from HTTP endpoints at regular intervals.
Supported pipeline types: logs
auth_strategyOptional authentication strategy for HTTP requests.
none, basic, bearer, customcustom_keyName of the environment variable or secret that holds a custom header value (used with custom auth strategies).
decoding [required]The decoding format used to interpret incoming logs.
bytes, gelf, json, syslogendpoint_url_keyName of the environment variable or secret that holds the HTTP endpoint URL to scrape.
id [required]The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the input to downstream components).
password_keyName of the environment variable or secret that holds the password (used when auth_strategy is basic).
scrape_interval_secsThe interval (in seconds) between HTTP scrape requests.
scrape_timeout_secsThe timeout (in seconds) for each scrape request.
tlsConfiguration for enabling TLS encryption between the pipeline component and external services.
ca_filePath to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_filePath to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
key_pass_keyName of the environment variable or secret that holds the passphrase for the private key file.
server_nameServer name to use for Server Name Indication (SNI) and to verify against the certificate presented by the remote host. Use this when the address you connect to doesn't match the certificate's Common Name or Subject Alternative Name.
token_keyName of the environment variable or secret that holds the bearer token (used when auth_strategy is bearer).
type [required]The source type. The value should always be http_client.
http_clienthttp_clientusername_keyName of the environment variable or secret that holds the username (used when auth_strategy is basic).
<type=http_server>The http_server source collects logs over HTTP POST from external services.
Supported pipeline types: logs
address_keyName of the environment variable or secret that holds the listen address for the HTTP server.
auth_strategy [required]HTTP authentication method.
none, plaincustom_keyName of the environment variable or secret that holds a custom header value (used with custom auth strategies).
decoding [required]The decoding format used to interpret incoming logs.
bytes, gelf, json, syslogid [required]Unique ID for the HTTP server source.
password_keyName of the environment variable or secret that holds the password (used when auth_strategy is plain).
tlsConfiguration for enabling TLS encryption between the pipeline component and external connecting clients.
ca_filePath to the Certificate Authority (CA) file used to validate connecting clients' TLS certificates.
crt_file [required]Path to the TLS server certificate file used to used to identify the pipeline component to connecting clients.
key_filePath to the private key file associated with the TLS server certificate.
key_pass_keyName of the environment variable or secret that holds the passphrase for the private key file.
verify_certificateWhen true, requires client connections to present a valid certificate, enabling mutual TLS authentication.
type [required]The source type. The value should always be http_server.
http_serverhttp_serverusername_keyName of the environment variable or secret that holds the username (used when auth_strategy is plain).
valid_tokensA list of tokens that are accepted for authenticating incoming HTTP requests. When set,
the source rejects any request whose token does not match an enabled entry in this list.
Cannot be combined with the plain auth strategy.
enabledIndicates whether this token is currently accepted. Disabled tokens are rejected without being removed from the configuration.
truefield_to_addAn optional metadata field that is attached to every event authenticated by the
associated token. Both key and value must match ^[A-Za-z0-9_]+$.
key [required]The metadata field name to add to incoming events.
value [required]The metadata field value to add to incoming events.
path_to_tokenSpecifies where the worker extracts the token from in the incoming HTTP request.
This can be either a built-in location (path or address) or an HTTP header object.
Object 1Built-in token location on the incoming HTTP request.
path, addressObject 2Extract the token from a specific HTTP request header.
header [required]The name of the HTTP header that carries the token.
token_key [required]Name of the environment variable or secret that holds the expected token value.
<type=kafka>The kafka source ingests data from Apache Kafka topics.
Supported pipeline types: logs
bootstrap_servers_keyName of the environment variable or secret that holds the Kafka bootstrap servers list.
group_id [required]Consumer group ID used by the Kafka client.
id [required]The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the input to downstream components).
librdkafka_optionsOptional list of advanced Kafka client configuration options, defined as key-value pairs.
name [required]The name of the librdkafka configuration option to set.
value [required]The value assigned to the specified librdkafka configuration option.
saslSpecifies the SASL mechanism for authenticating with a Kafka cluster.
mechanismSASL mechanism used for Kafka authentication.
PLAIN, SCRAM-SHA-256, SCRAM-SHA-512password_keyName of the environment variable or secret that holds the SASL password.
username_keyName of the environment variable or secret that holds the SASL username.
tlsConfiguration for enabling TLS encryption between the pipeline component and external services.
ca_filePath to the Certificate Authority (CA) file used to validate the server’s TLS certificate.
crt_file [required]Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services.
key_filePath to the private key file associated with the TLS client certificate. Used for mutual TLS authentication.
key_pass_keyName of the environment variable or secret that holds the passphrase for the private key file.
topics [required]A list of Kafka topic names to subscribe to. The source ingests messages from each topic specified.
type [required]The source type. The value should always be kafka.
kafkakafka<type=logstash>The logstash source ingests logs from a Logstash forwarder.
Supported pipeline types: logs
address_keyName of the environment variable or secret that holds the listen address for the Logstash receiver.
id [required]The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the input to downstream components).
tlsConfiguration for enabling TLS encryption between the pipeline component and external connecting clients.
ca_filePath to the Certificate Authority (CA) file used to validate connecting clients' TLS certificates.
crt_file [required]Path to the TLS server certificate file used to used to identify the pipeline component to connecting clients.
key_filePath to the private key file associated with the TLS server certificate.
key_pass_keyName of the environment variable or secret that holds the passphrase for the private key file.
verify_certificateWhen true, requires client connections to present a valid certificate, enabling mutual TLS authentication.
type [required]The source type. The value should always be logstash.
logstashlogstash<type=rsyslog>The rsyslog source listens for logs over TCP or UDP from an rsyslog server using the syslog protocol.
Supported pipeline types: logs
address_keyName of the environment variable or secret that holds the listen address for the syslog receiver.
id [required]The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the input to downstream components).
mode [required]Protocol used by the syslog source to receive messages.
tcp, udptlsConfiguration for enabling TLS encryption between the pipeline component and external connecting clients.
ca_filePath to the Certificate Authority (CA) file used to validate connecting clients' TLS certificates.
crt_file [required]Path to the TLS server certificate file used to used to identify the pipeline component to connecting clients.
key_filePath to the private key file associated with the TLS server certificate.
key_pass_keyName of the environment variable or secret that holds the passphrase for the private key file.
verify_certificateWhen true, requires client connections to present a valid certificate, enabling mutual TLS authentication.
type [required]The source type. The value should always be rsyslog.
rsyslogrsyslog<type=socket>The socket source ingests logs over TCP or UDP.
Supported pipeline types: logs
address_keyName of the environment variable or secret that holds the listen address for the socket.
framing [required]Framing method configuration for the socket source.
<method=newline_delimited>Byte frames which are delimited by a newline character.
method [required]Byte frames which are delimited by a newline character.
newline_delimited<method=bytes>Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments).
method [required]Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments).
bytes<method=character_delimited>Byte frames which are delimited by a chosen character.
delimiter [required]A single ASCII character used to delimit events.
method [required]Byte frames which are delimited by a chosen character.
character_delimited<method=octet_counting>Byte frames according to the octet counting format as per RFC6587.
method [required]Byte frames according to the octet counting format as per RFC6587.
octet_counting<method=chunked_gelf>Byte frames which are chunked GELF messages.
method [required]Byte frames which are chunked GELF messages.
chunked_gelfid [required]The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the input to downstream components).
mode [required]Protocol used to receive logs.
tcp, udptlsTLS configuration. Relevant only when mode is tcp.
ca_filePath to the Certificate Authority (CA) file used to validate connecting clients' TLS certificates.
crt_file [required]Path to the TLS server certificate file used to used to identify the pipeline component to connecting clients.
key_filePath to the private key file associated with the TLS server certificate.
key_pass_keyName of the environment variable or secret that holds the passphrase for the private key file.
verify_certificateWhen true, requires client connections to present a valid certificate, enabling mutual TLS authentication.
type [required]The source type. The value should always be socket.
socketsocket<type=splunk_hec>The splunk_hec source implements the Splunk HTTP Event Collector (HEC) API.
Supported pipeline types: logs
address_keyName of the environment variable or secret that holds the listen address for the HEC API.
id [required]The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the input to downstream components).
store_hec_tokenWhen true, the Splunk HEC token from the incoming request is stored in the event metadata.
This allows downstream components to forward the token to other Splunk HEC destinations.
tlsConfiguration for enabling TLS encryption between the pipeline component and external connecting clients.
ca_filePath to the Certificate Authority (CA) file used to validate connecting clients' TLS certificates.
crt_file [required]Path to the TLS server certificate file used to used to identify the pipeline component to connecting clients.
key_filePath to the private key file associated with the TLS server certificate.
key_pass_keyName of the environment variable or secret that holds the passphrase for the private key file.
verify_certificateWhen true, requires client connections to present a valid certificate, enabling mutual TLS authentication.
type [required]The source type. Always splunk_hec.
splunk_hecsplunk_hecvalid_tokensA list of tokens that are accepted for authenticating incoming HEC requests. When set, the source rejects any request whose HEC token does not match an enabled entry in this list.
enabledIndicates whether this token is currently accepted. Disabled tokens are rejected without being removed from the configuration.
truefield_to_addAn optional metadata field that is attached to every event authenticated by the
associated token. Both key and value must match ^[A-Za-z0-9_]+$.
key [required]The metadata field name to add to incoming events.
value [required]The metadata field value to add to incoming events.
token_key [required]Name of the environment variable or secret that holds the expected HEC token value.
<type=splunk_tcp>The splunk_tcp source receives logs from a Splunk Universal Forwarder over TCP.
TLS is supported for secure transmission.
Supported pipeline types: logs
address_keyName of the environment variable or secret that holds the listen address for the Splunk TCP receiver.
id [required]The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the input to downstream components).
max_connection_duration_secsMaximum duration, in seconds, that a connection can remain open before it is closed. When unset, connections can remain open indefinitely.
tlsConfiguration for enabling TLS encryption between the pipeline component and external connecting clients.
ca_filePath to the Certificate Authority (CA) file used to validate connecting clients' TLS certificates.
crt_file [required]Path to the TLS server certificate file used to used to identify the pipeline component to connecting clients.
key_filePath to the private key file associated with the TLS server certificate.
key_pass_keyName of the environment variable or secret that holds the passphrase for the private key file.
verify_certificateWhen true, requires client connections to present a valid certificate, enabling mutual TLS authentication.
type [required]The source type. Always splunk_tcp.
splunk_tcpsplunk_tcp<type=sumo_logic>The sumo_logic source receives logs from Sumo Logic collectors.
Supported pipeline types: logs
address_keyName of the environment variable or secret that holds the listen address for the Sumo Logic receiver.
id [required]The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the input to downstream components).
type [required]The source type. The value should always be sumo_logic.
sumo_logicsumo_logic<type=syslog_ng>The syslog_ng source listens for logs over TCP or UDP from a syslog-ng server using the syslog protocol.
Supported pipeline types: logs
address_keyName of the environment variable or secret that holds the listen address for the syslog-ng receiver.
id [required]The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the input to downstream components).
mode [required]Protocol used by the syslog source to receive messages.
tcp, udptlsConfiguration for enabling TLS encryption between the pipeline component and external connecting clients.
ca_filePath to the Certificate Authority (CA) file used to validate connecting clients' TLS certificates.
crt_file [required]Path to the TLS server certificate file used to used to identify the pipeline component to connecting clients.
key_filePath to the private key file associated with the TLS server certificate.
key_pass_keyName of the environment variable or secret that holds the passphrase for the private key file.
verify_certificateWhen true, requires client connections to present a valid certificate, enabling mutual TLS authentication.
type [required]The source type. The value should always be syslog_ng.
syslog_ngsyslog_ng<type=websocket>The websocket source ingests logs from a WebSocket server using the ws:// or wss:// protocol.
Supported pipeline types: logs.
auth_strategy [required]Authentication strategy for the WebSocket source connection.
none, basic, bearer, customcustom_keyName of the environment variable or secret that holds the custom authorization header value. Used when auth_strategy is custom.
decoding [required]The decoding format used to interpret incoming logs.
bytes, gelf, json, syslogid [required]The unique identifier for this component.
password_keyName of the environment variable or secret that holds the password. Used when auth_strategy is basic.
tlsTLS configuration for the WebSocket source. Use enabled for standard wss:// connections, or with_client_cert to present a client certificate for mutual TLS.
<mode=enabled>TLS configuration that enables encryption without a client certificate. Use this for standard wss:// connections that do not require mutual TLS.
mode [required]TLS mode. Must be enabled.
enabled<mode=with_client_cert>TLS configuration that enables encryption and presents a client certificate for mutual TLS authentication.
ca_filePath to the Certificate Authority (CA) file used to validate the remote server's TLS certificate.
crt_file [required]Path to the TLS client certificate file used to identify this source to the remote server.
key_filePath to the private key file associated with the client certificate.
key_pass_keyName of the environment variable or secret that holds the passphrase for the private key file.
mode [required]TLS mode. Must be with_client_cert.
with_client_certtoken_keyName of the environment variable or secret that holds the bearer token. Used when auth_strategy is bearer.
type [required]The source type. The value should always be websocket.
websocketwebsocketuri_keyName of the environment variable or secret that holds the WebSocket server URI (ws:// or wss://).
username_keyName of the environment variable or secret that holds the username. Used when auth_strategy is basic.
<type=prometheus_remote_write>The prometheus_remote_write source ingests metrics pushed over the Prometheus Remote Write protocol.
Supported pipeline types: metrics
address_keyName of the environment variable or secret that holds the listen address for the Prometheus Remote Write endpoint.
auth_strategy [required]HTTP authentication method.
none, plainid [required]The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the input to downstream components).
password_keyName of the environment variable or secret that holds the password (used when auth_strategy is plain).
path [required]The HTTP path on which the source listens for incoming Prometheus Remote Write requests.
tlsConfiguration for enabling TLS encryption between the pipeline component and external connecting clients.
ca_filePath to the Certificate Authority (CA) file used to validate connecting clients' TLS certificates.
crt_file [required]Path to the TLS server certificate file used to used to identify the pipeline component to connecting clients.
key_filePath to the private key file associated with the TLS server certificate.
key_pass_keyName of the environment variable or secret that holds the passphrase for the private key file.
verify_certificateWhen true, requires client connections to present a valid certificate, enabling mutual TLS authentication.
type [required]The source type. The value should always be prometheus_remote_write.
prometheus_remote_writeprometheus_remote_writeusername_keyName of the environment variable or secret that holds the username (used when auth_strategy is plain).
valid_tokensA list of tokens that are accepted for authenticating incoming requests. When set, the source rejects any request whose token does not match an enabled entry in this list.
enabledIndicates whether this token is currently accepted. Disabled tokens are rejected without being removed from the configuration.
truepath_to_tokenSpecifies where the worker extracts the token from in the incoming HTTP request.
This can be either a built-in location (path or address) or an HTTP header object.
Object 1Built-in token location on the incoming HTTP request.
path, addressObject 2Extract the token from a specific HTTP request header.
header [required]The name of the HTTP header that carries the token.
token_key [required]Name of the environment variable or secret that holds the expected token value.
use_legacy_search_syntaxSet to true to continue using the legacy search syntax while migrating filter queries. After migrating all queries to the new syntax, set to false.
The legacy syntax is deprecated and will eventually be removed.
Requires Observability Pipelines Worker 2.11 or later.
Only applies to logs pipelines. This field is ignored for metrics and traces pipelines.
See Upgrade Your Filter Queries to the New Search Syntax for more information.
name [required]Name of the pipeline.
id [required]Unique identifier for the pipeline.
type [required]The resource type identifier. For pipeline resources, this should always be set to pipelines.
pipelines{
"data": {
"attributes": {
"config": {
"destinations": [
{
"api_version": "auto",
"auth": {
"password_key": "ELASTICSEARCH_PASSWORD",
"strategy": "aws",
"username_key": "ELASTICSEARCH_USERNAME"
},
"buffer": "undefined",
"bulk_index": "logs-index",
"compression": {
"algorithm": "gzip",
"level": 6
},
"data_stream": {
"auto_routing": false,
"dataset": "string",
"dtype": "string",
"namespace": "string",
"sync_fields": false
},
"endpoint_url_key": "ELASTICSEARCH_ENDPOINT_URL",
"id": "elasticsearch-destination",
"id_key": "id",
"inputs": [
"filter-processor"
],
"pipeline": "my-pipeline",
"request_retry_partial": false,
"tls": {
"ca_file": "string",
"crt_file": "/path/to/cert.crt",
"key_file": "string",
"key_pass_key": "TLS_KEY_PASSPHRASE"
},
"type": "elasticsearch"
}
],
"pipeline_type": "logs",
"processor_groups": [
{
"display_name": "my component",
"enabled": true,
"id": "grouped-processors",
"include": "service:my-service",
"inputs": [
"datadog-agent-source"
],
"processors": [
[]
]
}
],
"processors": [
{
"display_name": "my component",
"enabled": true,
"id": "grouped-processors",
"include": "service:my-service",
"inputs": [
"datadog-agent-source"
],
"processors": [
[]
]
}
],
"sources": [
{
"address_key": "DATADOG_AGENT_ADDRESS",
"id": "datadog-agent-source",
"tls": {
"ca_file": "string",
"crt_file": "/path/to/cert.crt",
"key_file": "string",
"key_pass_key": "TLS_KEY_PASSPHRASE"
},
"type": "datadog_agent"
}
],
"use_legacy_search_syntax": false
},
"name": "Main Observability Pipeline"
},
"id": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
"type": "pipelines"
}
}Bad Request
API error response.
errors [required]A list of errors.
{
"errors": [
"Bad Request"
]
}Not Authorized
API error response.
errors [required]A list of errors.
{
"errors": [
"Bad Request"
]
}Conflict
API error response.
errors [required]A list of errors.
{
"errors": [
"Bad Request"
]
}Too many requests
API error response.
errors [required]A list of errors.
{
"errors": [
"Bad Request"
]
}Exemple de code
Create a new pipeline returns "OK" response
# Set your Datadog site and credentials
export DD_SITE="datadoghq.com"
# Use a Personal Access Token or Service Access Token
export DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>"
curl -X POST "https://api.${DD_SITE}/api/v2/obs-pipelines/pipelines" \
-H "Accept: application/json" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer ${DD_BEARER_TOKEN}" \
-d @- << EOF
{
"data": {
"attributes": {
"config": {
"destinations": [
{
"id": "datadog-logs-destination",
"inputs": [
"my-processor-group"
],
"type": "datadog_logs"
}
],
"processor_groups": [
{
"enabled": true,
"id": "my-processor-group",
"include": "service:my-service",
"inputs": [
"datadog-agent-source"
],
"processors": [
{
"enabled": true,
"id": "filter-processor",
"include": "status:error",
"type": "filter"
}
]
}
],
"sources": [
{
"id": "datadog-agent-source",
"type": "datadog_agent"
}
]
},
"name": "Main Observability Pipeline"
},
"type": "pipelines"
}
}
EOF# Set your Datadog site and credentials
export DD_SITE="us3.datadoghq.com"
# Use a Personal Access Token or Service Access Token
export DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>"
curl -X POST "https://api.${DD_SITE}/api/v2/obs-pipelines/pipelines" \
-H "Accept: application/json" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer ${DD_BEARER_TOKEN}" \
-d @- << EOF
{
"data": {
"attributes": {
"config": {
"destinations": [
{
"id": "datadog-logs-destination",
"inputs": [
"my-processor-group"
],
"type": "datadog_logs"
}
],
"processor_groups": [
{
"enabled": true,
"id": "my-processor-group",
"include": "service:my-service",
"inputs": [
"datadog-agent-source"
],
"processors": [
{
"enabled": true,
"id": "filter-processor",
"include": "status:error",
"type": "filter"
}
]
}
],
"sources": [
{
"id": "datadog-agent-source",
"type": "datadog_agent"
}
]
},
"name": "Main Observability Pipeline"
},
"type": "pipelines"
}
}
EOF# Set your Datadog site and credentials
export DD_SITE="us5.datadoghq.com"
# Use a Personal Access Token or Service Access Token
export DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>"
curl -X POST "https://api.${DD_SITE}/api/v2/obs-pipelines/pipelines" \
-H "Accept: application/json" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer ${DD_BEARER_TOKEN}" \
-d @- << EOF
{
"data": {
"attributes": {
"config": {
"destinations": [
{
"id": "datadog-logs-destination",
"inputs": [
"my-processor-group"
],
"type": "datadog_logs"
}
],
"processor_groups": [
{
"enabled": true,
"id": "my-processor-group",
"include": "service:my-service",
"inputs": [
"datadog-agent-source"
],
"processors": [
{
"enabled": true,
"id": "filter-processor",
"include": "status:error",
"type": "filter"
}
]
}
],
"sources": [
{
"id": "datadog-agent-source",
"type": "datadog_agent"
}
]
},
"name": "Main Observability Pipeline"
},
"type": "pipelines"
}
}
EOF# Set your Datadog site and credentials
export DD_SITE="datadoghq.eu"
# Use a Personal Access Token or Service Access Token
export DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>"
curl -X POST "https://api.${DD_SITE}/api/v2/obs-pipelines/pipelines" \
-H "Accept: application/json" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer ${DD_BEARER_TOKEN}" \
-d @- << EOF
{
"data": {
"attributes": {
"config": {
"destinations": [
{
"id": "datadog-logs-destination",
"inputs": [
"my-processor-group"
],
"type": "datadog_logs"
}
],
"processor_groups": [
{
"enabled": true,
"id": "my-processor-group",
"include": "service:my-service",
"inputs": [
"datadog-agent-source"
],
"processors": [
{
"enabled": true,
"id": "filter-processor",
"include": "status:error",
"type": "filter"
}
]
}
],
"sources": [
{
"id": "datadog-agent-source",
"type": "datadog_agent"
}
]
},
"name": "Main Observability Pipeline"
},
"type": "pipelines"
}
}
EOF# Set your Datadog site and credentials
export DD_SITE="ap1.datadoghq.com"
# Use a Personal Access Token or Service Access Token
export DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>"
curl -X POST "https://api.${DD_SITE}/api/v2/obs-pipelines/pipelines" \
-H "Accept: application/json" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer ${DD_BEARER_TOKEN}" \
-d @- << EOF
{
"data": {
"attributes": {
"config": {
"destinations": [
{
"id": "datadog-logs-destination",
"inputs": [
"my-processor-group"
],
"type": "datadog_logs"
}
],
"processor_groups": [
{
"enabled": true,
"id": "my-processor-group",
"include": "service:my-service",
"inputs": [
"datadog-agent-source"
],
"processors": [
{
"enabled": true,
"id": "filter-processor",
"include": "status:error",
"type": "filter"
}
]
}
],
"sources": [
{
"id": "datadog-agent-source",
"type": "datadog_agent"
}
]
},
"name": "Main Observability Pipeline"
},
"type": "pipelines"
}
}
EOF# Set your Datadog site and credentials
export DD_SITE="ap2.datadoghq.com"
# Use a Personal Access Token or Service Access Token
export DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>"
curl -X POST "https://api.${DD_SITE}/api/v2/obs-pipelines/pipelines" \
-H "Accept: application/json" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer ${DD_BEARER_TOKEN}" \
-d @- << EOF
{
"data": {
"attributes": {
"config": {
"destinations": [
{
"id": "datadog-logs-destination",
"inputs": [
"my-processor-group"
],
"type": "datadog_logs"
}
],
"processor_groups": [
{
"enabled": true,
"id": "my-processor-group",
"include": "service:my-service",
"inputs": [
"datadog-agent-source"
],
"processors": [
{
"enabled": true,
"id": "filter-processor",
"include": "status:error",
"type": "filter"
}
]
}
],
"sources": [
{
"id": "datadog-agent-source",
"type": "datadog_agent"
}
]
},
"name": "Main Observability Pipeline"
},
"type": "pipelines"
}
}
EOF# Set your Datadog site and credentials
export DD_SITE="uk1.datadoghq.com"
# Use a Personal Access Token or Service Access Token
export DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>"
curl -X POST "https://api.${DD_SITE}/api/v2/obs-pipelines/pipelines" \
-H "Accept: application/json" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer ${DD_BEARER_TOKEN}" \
-d @- << EOF
{
"data": {
"attributes": {
"config": {
"destinations": [
{
"id": "datadog-logs-destination",
"inputs": [
"my-processor-group"
],
"type": "datadog_logs"
}
],
"processor_groups": [
{
"enabled": true,
"id": "my-processor-group",
"include": "service:my-service",
"inputs": [
"datadog-agent-source"
],
"processors": [
{
"enabled": true,
"id": "filter-processor",
"include": "status:error",
"type": "filter"
}
]
}
],
"sources": [
{
"id": "datadog-agent-source",
"type": "datadog_agent"
}
]
},
"name": "Main Observability Pipeline"
},
"type": "pipelines"
}
}
EOF# Set your Datadog site and credentials
export DD_SITE="ddog-gov.com"
# Use a Personal Access Token or Service Access Token
export DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>"
curl -X POST "https://api.${DD_SITE}/api/v2/obs-pipelines/pipelines" \
-H "Accept: application/json" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer ${DD_BEARER_TOKEN}" \
-d @- << EOF
{
"data": {
"attributes": {
"config": {
"destinations": [
{
"id": "datadog-logs-destination",
"inputs": [
"my-processor-group"
],
"type": "datadog_logs"
}
],
"processor_groups": [
{
"enabled": true,
"id": "my-processor-group",
"include": "service:my-service",
"inputs": [
"datadog-agent-source"
],
"processors": [
{
"enabled": true,
"id": "filter-processor",
"include": "status:error",
"type": "filter"
}
]
}
],
"sources": [
{
"id": "datadog-agent-source",
"type": "datadog_agent"
}
]
},
"name": "Main Observability Pipeline"
},
"type": "pipelines"
}
}
EOF# Set your Datadog site and credentials
export DD_SITE="us2.ddog-gov.com"
# Use a Personal Access Token or Service Access Token
export DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>"
curl -X POST "https://api.${DD_SITE}/api/v2/obs-pipelines/pipelines" \
-H "Accept: application/json" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer ${DD_BEARER_TOKEN}" \
-d @- << EOF
{
"data": {
"attributes": {
"config": {
"destinations": [
{
"id": "datadog-logs-destination",
"inputs": [
"my-processor-group"
],
"type": "datadog_logs"
}
],
"processor_groups": [
{
"enabled": true,
"id": "my-processor-group",
"include": "service:my-service",
"inputs": [
"datadog-agent-source"
],
"processors": [
{
"enabled": true,
"id": "filter-processor",
"include": "status:error",
"type": "filter"
}
]
}
],
"sources": [
{
"id": "datadog-agent-source",
"type": "datadog_agent"
}
]
},
"name": "Main Observability Pipeline"
},
"type": "pipelines"
}
}
EOFCreate a pipeline with dedupe processor with cache returns "OK" response
# Set your Datadog site and credentials
export DD_SITE="datadoghq.com"
# Use a Personal Access Token or Service Access Token
export DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>"
curl -X POST "https://api.${DD_SITE}/api/v2/obs-pipelines/pipelines" \
-H "Accept: application/json" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer ${DD_BEARER_TOKEN}" \
-d @- << EOF
{
"data": {
"attributes": {
"config": {
"destinations": [
{
"id": "datadog-logs-destination",
"inputs": [
"my-processor-group"
],
"type": "datadog_logs"
}
],
"processor_groups": [
{
"enabled": true,
"id": "my-processor-group",
"include": "service:my-service",
"inputs": [
"datadog-agent-source"
],
"processors": [
{
"enabled": true,
"id": "dedupe-processor",
"include": "service:my-service",
"type": "dedupe",
"fields": [
"message"
],
"mode": "match",
"cache": {
"num_events": 5000
}
}
]
}
],
"sources": [
{
"id": "datadog-agent-source",
"type": "datadog_agent"
}
]
},
"name": "Pipeline with Dedupe Cache"
},
"type": "pipelines"
}
}
EOF# Set your Datadog site and credentials
export DD_SITE="us3.datadoghq.com"
# Use a Personal Access Token or Service Access Token
export DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>"
curl -X POST "https://api.${DD_SITE}/api/v2/obs-pipelines/pipelines" \
-H "Accept: application/json" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer ${DD_BEARER_TOKEN}" \
-d @- << EOF
{
"data": {
"attributes": {
"config": {
"destinations": [
{
"id": "datadog-logs-destination",
"inputs": [
"my-processor-group"
],
"type": "datadog_logs"
}
],
"processor_groups": [
{
"enabled": true,
"id": "my-processor-group",
"include": "service:my-service",
"inputs": [
"datadog-agent-source"
],
"processors": [
{
"enabled": true,
"id": "dedupe-processor",
"include": "service:my-service",
"type": "dedupe",
"fields": [
"message"
],
"mode": "match",
"cache": {
"num_events": 5000
}
}
]
}
],
"sources": [
{
"id": "datadog-agent-source",
"type": "datadog_agent"
}
]
},
"name": "Pipeline with Dedupe Cache"
},
"type": "pipelines"
}
}
EOF# Set your Datadog site and credentials
export DD_SITE="us5.datadoghq.com"
# Use a Personal Access Token or Service Access Token
export DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>"
curl -X POST "https://api.${DD_SITE}/api/v2/obs-pipelines/pipelines" \
-H "Accept: application/json" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer ${DD_BEARER_TOKEN}" \
-d @- << EOF
{
"data": {
"attributes": {
"config": {
"destinations": [
{
"id": "datadog-logs-destination",
"inputs": [
"my-processor-group"
],
"type": "datadog_logs"
}
],
"processor_groups": [
{
"enabled": true,
"id": "my-processor-group",
"include": "service:my-service",
"inputs": [
"datadog-agent-source"
],
"processors": [
{
"enabled": true,
"id": "dedupe-processor",
"include": "service:my-service",
"type": "dedupe",
"fields": [
"message"
],
"mode": "match",
"cache": {
"num_events": 5000
}
}
]
}
],
"sources": [
{
"id": "datadog-agent-source",
"type": "datadog_agent"
}
]
},
"name": "Pipeline with Dedupe Cache"
},
"type": "pipelines"
}
}
EOF# Set your Datadog site and credentials
export DD_SITE="datadoghq.eu"
# Use a Personal Access Token or Service Access Token
export DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>"
curl -X POST "https://api.${DD_SITE}/api/v2/obs-pipelines/pipelines" \
-H "Accept: application/json" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer ${DD_BEARER_TOKEN}" \
-d @- << EOF
{
"data": {
"attributes": {
"config": {
"destinations": [
{
"id": "datadog-logs-destination",
"inputs": [
"my-processor-group"
],
"type": "datadog_logs"
}
],
"processor_groups": [
{
"enabled": true,
"id": "my-processor-group",
"include": "service:my-service",
"inputs": [
"datadog-agent-source"
],
"processors": [
{
"enabled": true,
"id": "dedupe-processor",
"include": "service:my-service",
"type": "dedupe",
"fields": [
"message"
],
"mode": "match",
"cache": {
"num_events": 5000
}
}
]
}
],
"sources": [
{
"id": "datadog-agent-source",
"type": "datadog_agent"
}
]
},
"name": "Pipeline with Dedupe Cache"
},
"type": "pipelines"
}
}
EOF# Set your Datadog site and credentials
export DD_SITE="ap1.datadoghq.com"
# Use a Personal Access Token or Service Access Token
export DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>"
curl -X POST "https://api.${DD_SITE}/api/v2/obs-pipelines/pipelines" \
-H "Accept: application/json" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer ${DD_BEARER_TOKEN}" \
-d @- << EOF
{
"data": {
"attributes": {
"config": {
"destinations": [
{
"id": "datadog-logs-destination",
"inputs": [
"my-processor-group"
],
"type": "datadog_logs"
}
],
"processor_groups": [
{
"enabled": true,
"id": "my-processor-group",
"include": "service:my-service",
"inputs": [
"datadog-agent-source"
],
"processors": [
{
"enabled": true,
"id": "dedupe-processor",
"include": "service:my-service",
"type": "dedupe",
"fields": [
"message"
],
"mode": "match",
"cache": {
"num_events": 5000
}
}
]
}
],
"sources": [
{
"id": "datadog-agent-source",
"type": "datadog_agent"
}
]
},
"name": "Pipeline with Dedupe Cache"
},
"type": "pipelines"
}
}
EOF# Set your Datadog site and credentials
export DD_SITE="ap2.datadoghq.com"
# Use a Personal Access Token or Service Access Token
export DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>"
curl -X POST "https://api.${DD_SITE}/api/v2/obs-pipelines/pipelines" \
-H "Accept: application/json" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer ${DD_BEARER_TOKEN}" \
-d @- << EOF
{
"data": {
"attributes": {
"config": {
"destinations": [
{
"id": "datadog-logs-destination",
"inputs": [
"my-processor-group"
],
"type": "datadog_logs"
}
],
"processor_groups": [
{
"enabled": true,
"id": "my-processor-group",
"include": "service:my-service",
"inputs": [
"datadog-agent-source"
],
"processors": [
{
"enabled": true,
"id": "dedupe-processor",
"include": "service:my-service",
"type": "dedupe",
"fields": [
"message"
],
"mode": "match",
"cache": {
"num_events": 5000
}
}
]
}
],
"sources": [
{
"id": "datadog-agent-source",
"type": "datadog_agent"
}
]
},
"name": "Pipeline with Dedupe Cache"
},
"type": "pipelines"
}
}
EOF# Set your Datadog site and credentials
export DD_SITE="uk1.datadoghq.com"
# Use a Personal Access Token or Service Access Token
export DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>"
curl -X POST "https://api.${DD_SITE}/api/v2/obs-pipelines/pipelines" \
-H "Accept: application/json" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer ${DD_BEARER_TOKEN}" \
-d @- << EOF
{
"data": {
"attributes": {
"config": {
"destinations": [
{
"id": "datadog-logs-destination",
"inputs": [
"my-processor-group"
],
"type": "datadog_logs"
}
],
"processor_groups": [
{
"enabled": true,
"id": "my-processor-group",
"include": "service:my-service",
"inputs": [
"datadog-agent-source"
],
"processors": [
{
"enabled": true,
"id": "dedupe-processor",
"include": "service:my-service",
"type": "dedupe",
"fields": [
"message"
],
"mode": "match",
"cache": {
"num_events": 5000
}
}
]
}
],
"sources": [
{
"id": "datadog-agent-source",
"type": "datadog_agent"
}
]
},
"name": "Pipeline with Dedupe Cache"
},
"type": "pipelines"
}
}
EOF# Set your Datadog site and credentials
export DD_SITE="ddog-gov.com"
# Use a Personal Access Token or Service Access Token
export DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>"
curl -X POST "https://api.${DD_SITE}/api/v2/obs-pipelines/pipelines" \
-H "Accept: application/json" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer ${DD_BEARER_TOKEN}" \
-d @- << EOF
{
"data": {
"attributes": {
"config": {
"destinations": [
{
"id": "datadog-logs-destination",
"inputs": [
"my-processor-group"
],
"type": "datadog_logs"
}
],
"processor_groups": [
{
"enabled": true,
"id": "my-processor-group",
"include": "service:my-service",
"inputs": [
"datadog-agent-source"
],
"processors": [
{
"enabled": true,
"id": "dedupe-processor",
"include": "service:my-service",
"type": "dedupe",
"fields": [
"message"
],
"mode": "match",
"cache": {
"num_events": 5000
}
}
]
}
],
"sources": [
{
"id": "datadog-agent-source",
"type": "datadog_agent"
}
]
},
"name": "Pipeline with Dedupe Cache"
},
"type": "pipelines"
}
}
EOF# Set your Datadog site and credentials
export DD_SITE="us2.ddog-gov.com"
# Use a Personal Access Token or Service Access Token
export DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>"
curl -X POST "https://api.${DD_SITE}/api/v2/obs-pipelines/pipelines" \
-H "Accept: application/json" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer ${DD_BEARER_TOKEN}" \
-d @- << EOF
{
"data": {
"attributes": {
"config": {
"destinations": [
{
"id": "datadog-logs-destination",
"inputs": [
"my-processor-group"
],
"type": "datadog_logs"
}
],
"processor_groups": [
{
"enabled": true,
"id": "my-processor-group",
"include": "service:my-service",
"inputs": [
"datadog-agent-source"
],
"processors": [
{
"enabled": true,
"id": "dedupe-processor",
"include": "service:my-service",
"type": "dedupe",
"fields": [
"message"
],
"mode": "match",
"cache": {
"num_events": 5000
}
}
]
}
],
"sources": [
{
"id": "datadog-agent-source",
"type": "datadog_agent"
}
]
},
"name": "Pipeline with Dedupe Cache"
},
"type": "pipelines"
}
}
EOFCreate a pipeline with dedupe processor without cache returns "OK" response
# Set your Datadog site and credentials
export DD_SITE="datadoghq.com"
# Use a Personal Access Token or Service Access Token
export DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>"
curl -X POST "https://api.${DD_SITE}/api/v2/obs-pipelines/pipelines" \
-H "Accept: application/json" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer ${DD_BEARER_TOKEN}" \
-d @- << EOF
{
"data": {
"attributes": {
"config": {
"destinations": [
{
"id": "datadog-logs-destination",
"inputs": [
"my-processor-group"
],
"type": "datadog_logs"
}
],
"processor_groups": [
{
"enabled": true,
"id": "my-processor-group",
"include": "service:my-service",
"inputs": [
"datadog-agent-source"
],
"processors": [
{
"enabled": true,
"id": "dedupe-processor",
"include": "service:my-service",
"type": "dedupe",
"fields": [
"message"
],
"mode": "match"
}
]
}
],
"sources": [
{
"id": "datadog-agent-source",
"type": "datadog_agent"
}
]
},
"name": "Pipeline with Dedupe No Cache"
},
"type": "pipelines"
}
}
EOF# Set your Datadog site and credentials
export DD_SITE="us3.datadoghq.com"
# Use a Personal Access Token or Service Access Token
export DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>"
curl -X POST "https://api.${DD_SITE}/api/v2/obs-pipelines/pipelines" \
-H "Accept: application/json" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer ${DD_BEARER_TOKEN}" \
-d @- << EOF
{
"data": {
"attributes": {
"config": {
"destinations": [
{
"id": "datadog-logs-destination",
"inputs": [
"my-processor-group"
],
"type": "datadog_logs"
}
],
"processor_groups": [
{
"enabled": true,
"id": "my-processor-group",
"include": "service:my-service",
"inputs": [
"datadog-agent-source"
],
"processors": [
{
"enabled": true,
"id": "dedupe-processor",
"include": "service:my-service",
"type": "dedupe",
"fields": [
"message"
],
"mode": "match"
}
]
}
],
"sources": [
{
"id": "datadog-agent-source",
"type": "datadog_agent"
}
]
},
"name": "Pipeline with Dedupe No Cache"
},
"type": "pipelines"
}
}
EOF# Set your Datadog site and credentials
export DD_SITE="us5.datadoghq.com"
# Use a Personal Access Token or Service Access Token
export DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>"
curl -X POST "https://api.${DD_SITE}/api/v2/obs-pipelines/pipelines" \
-H "Accept: application/json" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer ${DD_BEARER_TOKEN}" \
-d @- << EOF
{
"data": {
"attributes": {
"config": {
"destinations": [
{
"id": "datadog-logs-destination",
"inputs": [
"my-processor-group"
],
"type": "datadog_logs"
}
],
"processor_groups": [
{
"enabled": true,
"id": "my-processor-group",
"include": "service:my-service",
"inputs": [
"datadog-agent-source"
],
"processors": [
{
"enabled": true,
"id": "dedupe-processor",
"include": "service:my-service",
"type": "dedupe",
"fields": [
"message"
],
"mode": "match"
}
]
}
],
"sources": [
{
"id": "datadog-agent-source",
"type": "datadog_agent"
}
]
},
"name": "Pipeline with Dedupe No Cache"
},
"type": "pipelines"
}
}
EOF# Set your Datadog site and credentials
export DD_SITE="datadoghq.eu"
# Use a Personal Access Token or Service Access Token
export DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>"
curl -X POST "https://api.${DD_SITE}/api/v2/obs-pipelines/pipelines" \
-H "Accept: application/json" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer ${DD_BEARER_TOKEN}" \
-d @- << EOF
{
"data": {
"attributes": {
"config": {
"destinations": [
{
"id": "datadog-logs-destination",
"inputs": [
"my-processor-group"
],
"type": "datadog_logs"
}
],
"processor_groups": [
{
"enabled": true,
"id": "my-processor-group",
"include": "service:my-service",
"inputs": [
"datadog-agent-source"
],
"processors": [
{
"enabled": true,
"id": "dedupe-processor",
"include": "service:my-service",
"type": "dedupe",
"fields": [
"message"
],
"mode": "match"
}
]
}
],
"sources": [
{
"id": "datadog-agent-source",
"type": "datadog_agent"
}
]
},
"name": "Pipeline with Dedupe No Cache"
},
"type": "pipelines"
}
}
EOF# Set your Datadog site and credentials
export DD_SITE="ap1.datadoghq.com"
# Use a Personal Access Token or Service Access Token
export DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>"
curl -X POST "https://api.${DD_SITE}/api/v2/obs-pipelines/pipelines" \
-H "Accept: application/json" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer ${DD_BEARER_TOKEN}" \
-d @- << EOF
{
"data": {
"attributes": {
"config": {
"destinations": [
{
"id": "datadog-logs-destination",
"inputs": [
"my-processor-group"
],
"type": "datadog_logs"
}
],
"processor_groups": [
{
"enabled": true,
"id": "my-processor-group",
"include": "service:my-service",
"inputs": [
"datadog-agent-source"
],
"processors": [
{
"enabled": true,
"id": "dedupe-processor",
"include": "service:my-service",
"type": "dedupe",
"fields": [
"message"
],
"mode": "match"
}
]
}
],
"sources": [
{
"id": "datadog-agent-source",
"type": "datadog_agent"
}
]
},
"name": "Pipeline with Dedupe No Cache"
},
"type": "pipelines"
}
}
EOF# Set your Datadog site and credentials
export DD_SITE="ap2.datadoghq.com"
# Use a Personal Access Token or Service Access Token
export DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>"
curl -X POST "https://api.${DD_SITE}/api/v2/obs-pipelines/pipelines" \
-H "Accept: application/json" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer ${DD_BEARER_TOKEN}" \
-d @- << EOF
{
"data": {
"attributes": {
"config": {
"destinations": [
{
"id": "datadog-logs-destination",
"inputs": [
"my-processor-group"
],
"type": "datadog_logs"
}
],
"processor_groups": [
{
"enabled": true,
"id": "my-processor-group",
"include": "service:my-service",
"inputs": [
"datadog-agent-source"
],
"processors": [
{
"enabled": true,
"id": "dedupe-processor",
"include": "service:my-service",
"type": "dedupe",
"fields": [
"message"
],
"mode": "match"
}
]
}
],
"sources": [
{
"id": "datadog-agent-source",
"type": "datadog_agent"
}
]
},
"name": "Pipeline with Dedupe No Cache"
},
"type": "pipelines"
}
}
EOF# Set your Datadog site and credentials
export DD_SITE="uk1.datadoghq.com"
# Use a Personal Access Token or Service Access Token
export DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>"
curl -X POST "https://api.${DD_SITE}/api/v2/obs-pipelines/pipelines" \
-H "Accept: application/json" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer ${DD_BEARER_TOKEN}" \
-d @- << EOF
{
"data": {
"attributes": {
"config": {
"destinations": [
{
"id": "datadog-logs-destination",
"inputs": [
"my-processor-group"
],
"type": "datadog_logs"
}
],
"processor_groups": [
{
"enabled": true,
"id": "my-processor-group",
"include": "service:my-service",
"inputs": [
"datadog-agent-source"
],
"processors": [
{
"enabled": true,
"id": "dedupe-processor",
"include": "service:my-service",
"type": "dedupe",
"fields": [
"message"
],
"mode": "match"
}
]
}
],
"sources": [
{
"id": "datadog-agent-source",
"type": "datadog_agent"
}
]
},
"name": "Pipeline with Dedupe No Cache"
},
"type": "pipelines"
}
}
EOF# Set your Datadog site and credentials
export DD_SITE="ddog-gov.com"
# Use a Personal Access Token or Service Access Token
export DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>"
curl -X POST "https://api.${DD_SITE}/api/v2/obs-pipelines/pipelines" \
-H "Accept: application/json" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer ${DD_BEARER_TOKEN}" \
-d @- << EOF
{
"data": {
"attributes": {
"config": {
"destinations": [
{
"id": "datadog-logs-destination",
"inputs": [
"my-processor-group"
],
"type": "datadog_logs"
}
],
"processor_groups": [
{
"enabled": true,
"id": "my-processor-group",
"include": "service:my-service",
"inputs": [
"datadog-agent-source"
],
"processors": [
{
"enabled": true,
"id": "dedupe-processor",
"include": "service:my-service",
"type": "dedupe",
"fields": [
"message"
],
"mode": "match"
}
]
}
],
"sources": [
{
"id": "datadog-agent-source",
"type": "datadog_agent"
}
]
},
"name": "Pipeline with Dedupe No Cache"
},
"type": "pipelines"
}
}
EOF# Set your Datadog site and credentials
export DD_SITE="us2.ddog-gov.com"
# Use a Personal Access Token or Service Access Token
export DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>"
curl -X POST "https://api.${DD_SITE}/api/v2/obs-pipelines/pipelines" \
-H "Accept: application/json" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer ${DD_BEARER_TOKEN}" \
-d @- << EOF
{
"data": {
"attributes": {
"config": {
"destinations": [
{
"id": "datadog-logs-destination",
"inputs": [
"my-processor-group"
],
"type": "datadog_logs"
}
],
"processor_groups": [
{
"enabled": true,
"id": "my-processor-group",
"include": "service:my-service",
"inputs": [
"datadog-agent-source"
],
"processors": [
{
"enabled": true,
"id": "dedupe-processor",
"include": "service:my-service",
"type": "dedupe",
"fields": [
"message"
],
"mode": "match"
}
]
}
],
"sources": [
{
"id": "datadog-agent-source",
"type": "datadog_agent"
}
]
},
"name": "Pipeline with Dedupe No Cache"
},
"type": "pipelines"
}
}
EOFCreate a new pipeline returns "OK" response
// Create a new pipeline returns "OK" response
package main
import (
"context"
"encoding/json"
"fmt"
"os"
"github.com/DataDog/datadog-api-client-go/v2/api/datadog"
"github.com/DataDog/datadog-api-client-go/v2/api/datadogV2"
)
func main() {
body := datadogV2.ObservabilityPipelineSpec{
Data: datadogV2.ObservabilityPipelineSpecData{
Attributes: datadogV2.ObservabilityPipelineDataAttributes{
Config: datadogV2.ObservabilityPipelineConfig{
Destinations: []datadogV2.ObservabilityPipelineConfigDestinationItem{
datadogV2.ObservabilityPipelineConfigDestinationItem{
ObservabilityPipelineDatadogLogsDestination: &datadogV2.ObservabilityPipelineDatadogLogsDestination{
Id: "datadog-logs-destination",
Inputs: []string{
"my-processor-group",
},
Type: datadogV2.OBSERVABILITYPIPELINEDATADOGLOGSDESTINATIONTYPE_DATADOG_LOGS,
}},
},
ProcessorGroups: []datadogV2.ObservabilityPipelineConfigProcessorGroup{
{
Enabled: true,
Id: "my-processor-group",
Include: "service:my-service",
Inputs: []string{
"datadog-agent-source",
},
Processors: []datadogV2.ObservabilityPipelineConfigProcessorItem{
datadogV2.ObservabilityPipelineConfigProcessorItem{
ObservabilityPipelineFilterProcessor: &datadogV2.ObservabilityPipelineFilterProcessor{
Enabled: true,
Id: "filter-processor",
Include: "status:error",
Type: datadogV2.OBSERVABILITYPIPELINEFILTERPROCESSORTYPE_FILTER,
}},
},
},
},
Sources: []datadogV2.ObservabilityPipelineConfigSourceItem{
datadogV2.ObservabilityPipelineConfigSourceItem{
ObservabilityPipelineDatadogAgentSource: &datadogV2.ObservabilityPipelineDatadogAgentSource{
Id: "datadog-agent-source",
Type: datadogV2.OBSERVABILITYPIPELINEDATADOGAGENTSOURCETYPE_DATADOG_AGENT,
}},
},
},
Name: "Main Observability Pipeline",
},
Type: "pipelines",
},
}
ctx := datadog.NewDefaultContext(context.Background())
ctx = context.WithValue(ctx, datadog.ContextAccessToken, os.Getenv("DD_BEARER_TOKEN"))
configuration := datadog.NewConfiguration()
apiClient := datadog.NewAPIClient(configuration)
api := datadogV2.NewObservabilityPipelinesApi(apiClient)
resp, r, err := api.CreatePipeline(ctx, body)
if err != nil {
fmt.Fprintf(os.Stderr, "Error when calling `ObservabilityPipelinesApi.CreatePipeline`: %v\n", err)
fmt.Fprintf(os.Stderr, "Full HTTP response: %v\n", r)
}
responseContent, _ := json.MarshalIndent(resp, "", " ")
fmt.Fprintf(os.Stdout, "Response from `ObservabilityPipelinesApi.CreatePipeline`:\n%s\n", responseContent)
}
Create a pipeline with dedupe processor with cache returns "OK" response
// Create a pipeline with dedupe processor with cache returns "OK" response
package main
import (
"context"
"encoding/json"
"fmt"
"os"
"github.com/DataDog/datadog-api-client-go/v2/api/datadog"
"github.com/DataDog/datadog-api-client-go/v2/api/datadogV2"
)
func main() {
body := datadogV2.ObservabilityPipelineSpec{
Data: datadogV2.ObservabilityPipelineSpecData{
Attributes: datadogV2.ObservabilityPipelineDataAttributes{
Config: datadogV2.ObservabilityPipelineConfig{
Destinations: []datadogV2.ObservabilityPipelineConfigDestinationItem{
datadogV2.ObservabilityPipelineConfigDestinationItem{
ObservabilityPipelineDatadogLogsDestination: &datadogV2.ObservabilityPipelineDatadogLogsDestination{
Id: "datadog-logs-destination",
Inputs: []string{
"my-processor-group",
},
Type: datadogV2.OBSERVABILITYPIPELINEDATADOGLOGSDESTINATIONTYPE_DATADOG_LOGS,
}},
},
ProcessorGroups: []datadogV2.ObservabilityPipelineConfigProcessorGroup{
{
Enabled: true,
Id: "my-processor-group",
Include: "service:my-service",
Inputs: []string{
"datadog-agent-source",
},
Processors: []datadogV2.ObservabilityPipelineConfigProcessorItem{
datadogV2.ObservabilityPipelineConfigProcessorItem{
ObservabilityPipelineDedupeProcessor: &datadogV2.ObservabilityPipelineDedupeProcessor{
Enabled: true,
Id: "dedupe-processor",
Include: "service:my-service",
Type: datadogV2.OBSERVABILITYPIPELINEDEDUPEPROCESSORTYPE_DEDUPE,
Fields: []string{
"message",
},
Mode: datadogV2.OBSERVABILITYPIPELINEDEDUPEPROCESSORMODE_MATCH,
Cache: &datadogV2.ObservabilityPipelineDedupeProcessorCache{
NumEvents: 5000,
},
}},
},
},
},
Sources: []datadogV2.ObservabilityPipelineConfigSourceItem{
datadogV2.ObservabilityPipelineConfigSourceItem{
ObservabilityPipelineDatadogAgentSource: &datadogV2.ObservabilityPipelineDatadogAgentSource{
Id: "datadog-agent-source",
Type: datadogV2.OBSERVABILITYPIPELINEDATADOGAGENTSOURCETYPE_DATADOG_AGENT,
}},
},
},
Name: "Pipeline with Dedupe Cache",
},
Type: "pipelines",
},
}
ctx := datadog.NewDefaultContext(context.Background())
ctx = context.WithValue(ctx, datadog.ContextAccessToken, os.Getenv("DD_BEARER_TOKEN"))
configuration := datadog.NewConfiguration()
apiClient := datadog.NewAPIClient(configuration)
api := datadogV2.NewObservabilityPipelinesApi(apiClient)
resp, r, err := api.CreatePipeline(ctx, body)
if err != nil {
fmt.Fprintf(os.Stderr, "Error when calling `ObservabilityPipelinesApi.CreatePipeline`: %v\n", err)
fmt.Fprintf(os.Stderr, "Full HTTP response: %v\n", r)
}
responseContent, _ := json.MarshalIndent(resp, "", " ")
fmt.Fprintf(os.Stdout, "Response from `ObservabilityPipelinesApi.CreatePipeline`:\n%s\n", responseContent)
}
Create a pipeline with dedupe processor without cache returns "OK" response
// Create a pipeline with dedupe processor without cache returns "OK" response
package main
import (
"context"
"encoding/json"
"fmt"
"os"
"github.com/DataDog/datadog-api-client-go/v2/api/datadog"
"github.com/DataDog/datadog-api-client-go/v2/api/datadogV2"
)
func main() {
body := datadogV2.ObservabilityPipelineSpec{
Data: datadogV2.ObservabilityPipelineSpecData{
Attributes: datadogV2.ObservabilityPipelineDataAttributes{
Config: datadogV2.ObservabilityPipelineConfig{
Destinations: []datadogV2.ObservabilityPipelineConfigDestinationItem{
datadogV2.ObservabilityPipelineConfigDestinationItem{
ObservabilityPipelineDatadogLogsDestination: &datadogV2.ObservabilityPipelineDatadogLogsDestination{
Id: "datadog-logs-destination",
Inputs: []string{
"my-processor-group",
},
Type: datadogV2.OBSERVABILITYPIPELINEDATADOGLOGSDESTINATIONTYPE_DATADOG_LOGS,
}},
},
ProcessorGroups: []datadogV2.ObservabilityPipelineConfigProcessorGroup{
{
Enabled: true,
Id: "my-processor-group",
Include: "service:my-service",
Inputs: []string{
"datadog-agent-source",
},
Processors: []datadogV2.ObservabilityPipelineConfigProcessorItem{
datadogV2.ObservabilityPipelineConfigProcessorItem{
ObservabilityPipelineDedupeProcessor: &datadogV2.ObservabilityPipelineDedupeProcessor{
Enabled: true,
Id: "dedupe-processor",
Include: "service:my-service",
Type: datadogV2.OBSERVABILITYPIPELINEDEDUPEPROCESSORTYPE_DEDUPE,
Fields: []string{
"message",
},
Mode: datadogV2.OBSERVABILITYPIPELINEDEDUPEPROCESSORMODE_MATCH,
}},
},
},
},
Sources: []datadogV2.ObservabilityPipelineConfigSourceItem{
datadogV2.ObservabilityPipelineConfigSourceItem{
ObservabilityPipelineDatadogAgentSource: &datadogV2.ObservabilityPipelineDatadogAgentSource{
Id: "datadog-agent-source",
Type: datadogV2.OBSERVABILITYPIPELINEDATADOGAGENTSOURCETYPE_DATADOG_AGENT,
}},
},
},
Name: "Pipeline with Dedupe No Cache",
},
Type: "pipelines",
},
}
ctx := datadog.NewDefaultContext(context.Background())
ctx = context.WithValue(ctx, datadog.ContextAccessToken, os.Getenv("DD_BEARER_TOKEN"))
configuration := datadog.NewConfiguration()
apiClient := datadog.NewAPIClient(configuration)
api := datadogV2.NewObservabilityPipelinesApi(apiClient)
resp, r, err := api.CreatePipeline(ctx, body)
if err != nil {
fmt.Fprintf(os.Stderr, "Error when calling `ObservabilityPipelinesApi.CreatePipeline`: %v\n", err)
fmt.Fprintf(os.Stderr, "Full HTTP response: %v\n", r)
}
responseContent, _ := json.MarshalIndent(resp, "", " ")
fmt.Fprintf(os.Stdout, "Response from `ObservabilityPipelinesApi.CreatePipeline`:\n%s\n", responseContent)
}
Instructions
First install the library and its dependencies and then save the example to main.go and run following commands:
DD_SITE="datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" go run "main.go"DD_SITE="us3.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" go run "main.go"DD_SITE="us5.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" go run "main.go"DD_SITE="datadoghq.eu" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" go run "main.go"DD_SITE="ap1.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" go run "main.go"DD_SITE="ap2.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" go run "main.go"DD_SITE="uk1.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" go run "main.go"DD_SITE="ddog-gov.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" go run "main.go"DD_SITE="us2.ddog-gov.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" go run "main.go"Create a new pipeline returns "OK" response
// Create a new pipeline returns "OK" response
import com.datadog.api.client.ApiClient;
import com.datadog.api.client.ApiException;
import com.datadog.api.client.v2.api.ObservabilityPipelinesApi;
import com.datadog.api.client.v2.model.ObservabilityPipeline;
import com.datadog.api.client.v2.model.ObservabilityPipelineConfig;
import com.datadog.api.client.v2.model.ObservabilityPipelineConfigDestinationItem;
import com.datadog.api.client.v2.model.ObservabilityPipelineConfigProcessorGroup;
import com.datadog.api.client.v2.model.ObservabilityPipelineConfigProcessorItem;
import com.datadog.api.client.v2.model.ObservabilityPipelineConfigSourceItem;
import com.datadog.api.client.v2.model.ObservabilityPipelineDataAttributes;
import com.datadog.api.client.v2.model.ObservabilityPipelineDatadogAgentSource;
import com.datadog.api.client.v2.model.ObservabilityPipelineDatadogAgentSourceType;
import com.datadog.api.client.v2.model.ObservabilityPipelineDatadogLogsDestination;
import com.datadog.api.client.v2.model.ObservabilityPipelineDatadogLogsDestinationType;
import com.datadog.api.client.v2.model.ObservabilityPipelineFilterProcessor;
import com.datadog.api.client.v2.model.ObservabilityPipelineFilterProcessorType;
import com.datadog.api.client.v2.model.ObservabilityPipelineSpec;
import com.datadog.api.client.v2.model.ObservabilityPipelineSpecData;
import java.util.Collections;
public class Example {
public static void main(String[] args) {
ApiClient defaultClient = ApiClient.getDefaultApiClient();
defaultClient.setAccessToken(System.getenv("DD_BEARER_TOKEN"));
ObservabilityPipelinesApi apiInstance = new ObservabilityPipelinesApi(defaultClient);
ObservabilityPipelineSpec body =
new ObservabilityPipelineSpec()
.data(
new ObservabilityPipelineSpecData()
.attributes(
new ObservabilityPipelineDataAttributes()
.config(
new ObservabilityPipelineConfig()
.destinations(
Collections.singletonList(
new ObservabilityPipelineConfigDestinationItem(
new ObservabilityPipelineDatadogLogsDestination()
.id("datadog-logs-destination")
.inputs(
Collections.singletonList(
"my-processor-group"))
.type(
ObservabilityPipelineDatadogLogsDestinationType
.DATADOG_LOGS))))
.processorGroups(
Collections.singletonList(
new ObservabilityPipelineConfigProcessorGroup()
.enabled(true)
.id("my-processor-group")
.include("service:my-service")
.inputs(
Collections.singletonList(
"datadog-agent-source"))
.processors(
Collections.singletonList(
new ObservabilityPipelineConfigProcessorItem(
new ObservabilityPipelineFilterProcessor()
.enabled(true)
.id("filter-processor")
.include("status:error")
.type(
ObservabilityPipelineFilterProcessorType
.FILTER))))))
.sources(
Collections.singletonList(
new ObservabilityPipelineConfigSourceItem(
new ObservabilityPipelineDatadogAgentSource()
.id("datadog-agent-source")
.type(
ObservabilityPipelineDatadogAgentSourceType
.DATADOG_AGENT)))))
.name("Main Observability Pipeline"))
.type("pipelines"));
try {
ObservabilityPipeline result = apiInstance.createPipeline(body);
System.out.println(result);
} catch (ApiException e) {
System.err.println("Exception when calling ObservabilityPipelinesApi#createPipeline");
System.err.println("Status code: " + e.getCode());
System.err.println("Reason: " + e.getResponseBody());
System.err.println("Response headers: " + e.getResponseHeaders());
e.printStackTrace();
}
}
}
Create a pipeline with dedupe processor with cache returns "OK" response
// Create a pipeline with dedupe processor with cache returns "OK" response
import com.datadog.api.client.ApiClient;
import com.datadog.api.client.ApiException;
import com.datadog.api.client.v2.api.ObservabilityPipelinesApi;
import com.datadog.api.client.v2.model.ObservabilityPipeline;
import com.datadog.api.client.v2.model.ObservabilityPipelineConfig;
import com.datadog.api.client.v2.model.ObservabilityPipelineConfigDestinationItem;
import com.datadog.api.client.v2.model.ObservabilityPipelineConfigProcessorGroup;
import com.datadog.api.client.v2.model.ObservabilityPipelineConfigProcessorItem;
import com.datadog.api.client.v2.model.ObservabilityPipelineConfigSourceItem;
import com.datadog.api.client.v2.model.ObservabilityPipelineDataAttributes;
import com.datadog.api.client.v2.model.ObservabilityPipelineDatadogAgentSource;
import com.datadog.api.client.v2.model.ObservabilityPipelineDatadogAgentSourceType;
import com.datadog.api.client.v2.model.ObservabilityPipelineDatadogLogsDestination;
import com.datadog.api.client.v2.model.ObservabilityPipelineDatadogLogsDestinationType;
import com.datadog.api.client.v2.model.ObservabilityPipelineDedupeProcessor;
import com.datadog.api.client.v2.model.ObservabilityPipelineDedupeProcessorCache;
import com.datadog.api.client.v2.model.ObservabilityPipelineDedupeProcessorMode;
import com.datadog.api.client.v2.model.ObservabilityPipelineDedupeProcessorType;
import com.datadog.api.client.v2.model.ObservabilityPipelineSpec;
import com.datadog.api.client.v2.model.ObservabilityPipelineSpecData;
import java.util.Collections;
public class Example {
public static void main(String[] args) {
ApiClient defaultClient = ApiClient.getDefaultApiClient();
defaultClient.setAccessToken(System.getenv("DD_BEARER_TOKEN"));
ObservabilityPipelinesApi apiInstance = new ObservabilityPipelinesApi(defaultClient);
ObservabilityPipelineSpec body =
new ObservabilityPipelineSpec()
.data(
new ObservabilityPipelineSpecData()
.attributes(
new ObservabilityPipelineDataAttributes()
.config(
new ObservabilityPipelineConfig()
.destinations(
Collections.singletonList(
new ObservabilityPipelineConfigDestinationItem(
new ObservabilityPipelineDatadogLogsDestination()
.id("datadog-logs-destination")
.inputs(
Collections.singletonList(
"my-processor-group"))
.type(
ObservabilityPipelineDatadogLogsDestinationType
.DATADOG_LOGS))))
.processorGroups(
Collections.singletonList(
new ObservabilityPipelineConfigProcessorGroup()
.enabled(true)
.id("my-processor-group")
.include("service:my-service")
.inputs(
Collections.singletonList(
"datadog-agent-source"))
.processors(
Collections.singletonList(
new ObservabilityPipelineConfigProcessorItem(
new ObservabilityPipelineDedupeProcessor()
.enabled(true)
.id("dedupe-processor")
.include("service:my-service")
.type(
ObservabilityPipelineDedupeProcessorType
.DEDUPE)
.fields(
Collections.singletonList(
"message"))
.mode(
ObservabilityPipelineDedupeProcessorMode
.MATCH)
.cache(
new ObservabilityPipelineDedupeProcessorCache()
.numEvents(5000L)))))))
.sources(
Collections.singletonList(
new ObservabilityPipelineConfigSourceItem(
new ObservabilityPipelineDatadogAgentSource()
.id("datadog-agent-source")
.type(
ObservabilityPipelineDatadogAgentSourceType
.DATADOG_AGENT)))))
.name("Pipeline with Dedupe Cache"))
.type("pipelines"));
try {
ObservabilityPipeline result = apiInstance.createPipeline(body);
System.out.println(result);
} catch (ApiException e) {
System.err.println("Exception when calling ObservabilityPipelinesApi#createPipeline");
System.err.println("Status code: " + e.getCode());
System.err.println("Reason: " + e.getResponseBody());
System.err.println("Response headers: " + e.getResponseHeaders());
e.printStackTrace();
}
}
}
Create a pipeline with dedupe processor without cache returns "OK" response
// Create a pipeline with dedupe processor without cache returns "OK" response
import com.datadog.api.client.ApiClient;
import com.datadog.api.client.ApiException;
import com.datadog.api.client.v2.api.ObservabilityPipelinesApi;
import com.datadog.api.client.v2.model.ObservabilityPipeline;
import com.datadog.api.client.v2.model.ObservabilityPipelineConfig;
import com.datadog.api.client.v2.model.ObservabilityPipelineConfigDestinationItem;
import com.datadog.api.client.v2.model.ObservabilityPipelineConfigProcessorGroup;
import com.datadog.api.client.v2.model.ObservabilityPipelineConfigProcessorItem;
import com.datadog.api.client.v2.model.ObservabilityPipelineConfigSourceItem;
import com.datadog.api.client.v2.model.ObservabilityPipelineDataAttributes;
import com.datadog.api.client.v2.model.ObservabilityPipelineDatadogAgentSource;
import com.datadog.api.client.v2.model.ObservabilityPipelineDatadogAgentSourceType;
import com.datadog.api.client.v2.model.ObservabilityPipelineDatadogLogsDestination;
import com.datadog.api.client.v2.model.ObservabilityPipelineDatadogLogsDestinationType;
import com.datadog.api.client.v2.model.ObservabilityPipelineDedupeProcessor;
import com.datadog.api.client.v2.model.ObservabilityPipelineDedupeProcessorMode;
import com.datadog.api.client.v2.model.ObservabilityPipelineDedupeProcessorType;
import com.datadog.api.client.v2.model.ObservabilityPipelineSpec;
import com.datadog.api.client.v2.model.ObservabilityPipelineSpecData;
import java.util.Collections;
public class Example {
public static void main(String[] args) {
ApiClient defaultClient = ApiClient.getDefaultApiClient();
defaultClient.setAccessToken(System.getenv("DD_BEARER_TOKEN"));
ObservabilityPipelinesApi apiInstance = new ObservabilityPipelinesApi(defaultClient);
ObservabilityPipelineSpec body =
new ObservabilityPipelineSpec()
.data(
new ObservabilityPipelineSpecData()
.attributes(
new ObservabilityPipelineDataAttributes()
.config(
new ObservabilityPipelineConfig()
.destinations(
Collections.singletonList(
new ObservabilityPipelineConfigDestinationItem(
new ObservabilityPipelineDatadogLogsDestination()
.id("datadog-logs-destination")
.inputs(
Collections.singletonList(
"my-processor-group"))
.type(
ObservabilityPipelineDatadogLogsDestinationType
.DATADOG_LOGS))))
.processorGroups(
Collections.singletonList(
new ObservabilityPipelineConfigProcessorGroup()
.enabled(true)
.id("my-processor-group")
.include("service:my-service")
.inputs(
Collections.singletonList(
"datadog-agent-source"))
.processors(
Collections.singletonList(
new ObservabilityPipelineConfigProcessorItem(
new ObservabilityPipelineDedupeProcessor()
.enabled(true)
.id("dedupe-processor")
.include("service:my-service")
.type(
ObservabilityPipelineDedupeProcessorType
.DEDUPE)
.fields(
Collections.singletonList(
"message"))
.mode(
ObservabilityPipelineDedupeProcessorMode
.MATCH))))))
.sources(
Collections.singletonList(
new ObservabilityPipelineConfigSourceItem(
new ObservabilityPipelineDatadogAgentSource()
.id("datadog-agent-source")
.type(
ObservabilityPipelineDatadogAgentSourceType
.DATADOG_AGENT)))))
.name("Pipeline with Dedupe No Cache"))
.type("pipelines"));
try {
ObservabilityPipeline result = apiInstance.createPipeline(body);
System.out.println(result);
} catch (ApiException e) {
System.err.println("Exception when calling ObservabilityPipelinesApi#createPipeline");
System.err.println("Status code: " + e.getCode());
System.err.println("Reason: " + e.getResponseBody());
System.err.println("Response headers: " + e.getResponseHeaders());
e.printStackTrace();
}
}
}
Instructions
First install the library and its dependencies and then save the example to Example.java and run following commands:
DD_SITE="datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" java "Example.java"DD_SITE="us3.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" java "Example.java"DD_SITE="us5.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" java "Example.java"DD_SITE="datadoghq.eu" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" java "Example.java"DD_SITE="ap1.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" java "Example.java"DD_SITE="ap2.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" java "Example.java"DD_SITE="uk1.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" java "Example.java"DD_SITE="ddog-gov.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" java "Example.java"DD_SITE="us2.ddog-gov.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" java "Example.java"Create a new pipeline returns "OK" response
"""
Create a new pipeline returns "OK" response
"""
from os import environ
from datadog_api_client import ApiClient, Configuration
from datadog_api_client.v2.api.observability_pipelines_api import ObservabilityPipelinesApi
from datadog_api_client.v2.model.observability_pipeline_config import ObservabilityPipelineConfig
from datadog_api_client.v2.model.observability_pipeline_config_processor_group import (
ObservabilityPipelineConfigProcessorGroup,
)
from datadog_api_client.v2.model.observability_pipeline_data_attributes import ObservabilityPipelineDataAttributes
from datadog_api_client.v2.model.observability_pipeline_datadog_agent_source import (
ObservabilityPipelineDatadogAgentSource,
)
from datadog_api_client.v2.model.observability_pipeline_datadog_agent_source_type import (
ObservabilityPipelineDatadogAgentSourceType,
)
from datadog_api_client.v2.model.observability_pipeline_datadog_logs_destination import (
ObservabilityPipelineDatadogLogsDestination,
)
from datadog_api_client.v2.model.observability_pipeline_datadog_logs_destination_type import (
ObservabilityPipelineDatadogLogsDestinationType,
)
from datadog_api_client.v2.model.observability_pipeline_filter_processor import ObservabilityPipelineFilterProcessor
from datadog_api_client.v2.model.observability_pipeline_filter_processor_type import (
ObservabilityPipelineFilterProcessorType,
)
from datadog_api_client.v2.model.observability_pipeline_spec import ObservabilityPipelineSpec
from datadog_api_client.v2.model.observability_pipeline_spec_data import ObservabilityPipelineSpecData
body = ObservabilityPipelineSpec(
data=ObservabilityPipelineSpecData(
attributes=ObservabilityPipelineDataAttributes(
config=ObservabilityPipelineConfig(
destinations=[
ObservabilityPipelineDatadogLogsDestination(
id="datadog-logs-destination",
inputs=[
"my-processor-group",
],
type=ObservabilityPipelineDatadogLogsDestinationType.DATADOG_LOGS,
),
],
processor_groups=[
ObservabilityPipelineConfigProcessorGroup(
enabled=True,
id="my-processor-group",
include="service:my-service",
inputs=[
"datadog-agent-source",
],
processors=[
ObservabilityPipelineFilterProcessor(
enabled=True,
id="filter-processor",
include="status:error",
type=ObservabilityPipelineFilterProcessorType.FILTER,
),
],
),
],
sources=[
ObservabilityPipelineDatadogAgentSource(
id="datadog-agent-source",
type=ObservabilityPipelineDatadogAgentSourceType.DATADOG_AGENT,
),
],
),
name="Main Observability Pipeline",
),
type="pipelines",
),
)
configuration = Configuration()
configuration.access_token = environ["DD_BEARER_TOKEN"]
with ApiClient(configuration) as api_client:
api_instance = ObservabilityPipelinesApi(api_client)
response = api_instance.create_pipeline(body=body)
print(response)
Create a pipeline with dedupe processor with cache returns "OK" response
"""
Create a pipeline with dedupe processor with cache returns "OK" response
"""
from os import environ
from datadog_api_client import ApiClient, Configuration
from datadog_api_client.v2.api.observability_pipelines_api import ObservabilityPipelinesApi
from datadog_api_client.v2.model.observability_pipeline_config import ObservabilityPipelineConfig
from datadog_api_client.v2.model.observability_pipeline_config_processor_group import (
ObservabilityPipelineConfigProcessorGroup,
)
from datadog_api_client.v2.model.observability_pipeline_data_attributes import ObservabilityPipelineDataAttributes
from datadog_api_client.v2.model.observability_pipeline_datadog_agent_source import (
ObservabilityPipelineDatadogAgentSource,
)
from datadog_api_client.v2.model.observability_pipeline_datadog_agent_source_type import (
ObservabilityPipelineDatadogAgentSourceType,
)
from datadog_api_client.v2.model.observability_pipeline_datadog_logs_destination import (
ObservabilityPipelineDatadogLogsDestination,
)
from datadog_api_client.v2.model.observability_pipeline_datadog_logs_destination_type import (
ObservabilityPipelineDatadogLogsDestinationType,
)
from datadog_api_client.v2.model.observability_pipeline_dedupe_processor import ObservabilityPipelineDedupeProcessor
from datadog_api_client.v2.model.observability_pipeline_dedupe_processor_cache import (
ObservabilityPipelineDedupeProcessorCache,
)
from datadog_api_client.v2.model.observability_pipeline_dedupe_processor_mode import (
ObservabilityPipelineDedupeProcessorMode,
)
from datadog_api_client.v2.model.observability_pipeline_dedupe_processor_type import (
ObservabilityPipelineDedupeProcessorType,
)
from datadog_api_client.v2.model.observability_pipeline_spec import ObservabilityPipelineSpec
from datadog_api_client.v2.model.observability_pipeline_spec_data import ObservabilityPipelineSpecData
body = ObservabilityPipelineSpec(
data=ObservabilityPipelineSpecData(
attributes=ObservabilityPipelineDataAttributes(
config=ObservabilityPipelineConfig(
destinations=[
ObservabilityPipelineDatadogLogsDestination(
id="datadog-logs-destination",
inputs=[
"my-processor-group",
],
type=ObservabilityPipelineDatadogLogsDestinationType.DATADOG_LOGS,
),
],
processor_groups=[
ObservabilityPipelineConfigProcessorGroup(
enabled=True,
id="my-processor-group",
include="service:my-service",
inputs=[
"datadog-agent-source",
],
processors=[
ObservabilityPipelineDedupeProcessor(
enabled=True,
id="dedupe-processor",
include="service:my-service",
type=ObservabilityPipelineDedupeProcessorType.DEDUPE,
fields=[
"message",
],
mode=ObservabilityPipelineDedupeProcessorMode.MATCH,
cache=ObservabilityPipelineDedupeProcessorCache(
num_events=5000,
),
),
],
),
],
sources=[
ObservabilityPipelineDatadogAgentSource(
id="datadog-agent-source",
type=ObservabilityPipelineDatadogAgentSourceType.DATADOG_AGENT,
),
],
),
name="Pipeline with Dedupe Cache",
),
type="pipelines",
),
)
configuration = Configuration()
configuration.access_token = environ["DD_BEARER_TOKEN"]
with ApiClient(configuration) as api_client:
api_instance = ObservabilityPipelinesApi(api_client)
response = api_instance.create_pipeline(body=body)
print(response)
Create a pipeline with dedupe processor without cache returns "OK" response
"""
Create a pipeline with dedupe processor without cache returns "OK" response
"""
from os import environ
from datadog_api_client import ApiClient, Configuration
from datadog_api_client.v2.api.observability_pipelines_api import ObservabilityPipelinesApi
from datadog_api_client.v2.model.observability_pipeline_config import ObservabilityPipelineConfig
from datadog_api_client.v2.model.observability_pipeline_config_processor_group import (
ObservabilityPipelineConfigProcessorGroup,
)
from datadog_api_client.v2.model.observability_pipeline_data_attributes import ObservabilityPipelineDataAttributes
from datadog_api_client.v2.model.observability_pipeline_datadog_agent_source import (
ObservabilityPipelineDatadogAgentSource,
)
from datadog_api_client.v2.model.observability_pipeline_datadog_agent_source_type import (
ObservabilityPipelineDatadogAgentSourceType,
)
from datadog_api_client.v2.model.observability_pipeline_datadog_logs_destination import (
ObservabilityPipelineDatadogLogsDestination,
)
from datadog_api_client.v2.model.observability_pipeline_datadog_logs_destination_type import (
ObservabilityPipelineDatadogLogsDestinationType,
)
from datadog_api_client.v2.model.observability_pipeline_dedupe_processor import ObservabilityPipelineDedupeProcessor
from datadog_api_client.v2.model.observability_pipeline_dedupe_processor_mode import (
ObservabilityPipelineDedupeProcessorMode,
)
from datadog_api_client.v2.model.observability_pipeline_dedupe_processor_type import (
ObservabilityPipelineDedupeProcessorType,
)
from datadog_api_client.v2.model.observability_pipeline_spec import ObservabilityPipelineSpec
from datadog_api_client.v2.model.observability_pipeline_spec_data import ObservabilityPipelineSpecData
body = ObservabilityPipelineSpec(
data=ObservabilityPipelineSpecData(
attributes=ObservabilityPipelineDataAttributes(
config=ObservabilityPipelineConfig(
destinations=[
ObservabilityPipelineDatadogLogsDestination(
id="datadog-logs-destination",
inputs=[
"my-processor-group",
],
type=ObservabilityPipelineDatadogLogsDestinationType.DATADOG_LOGS,
),
],
processor_groups=[
ObservabilityPipelineConfigProcessorGroup(
enabled=True,
id="my-processor-group",
include="service:my-service",
inputs=[
"datadog-agent-source",
],
processors=[
ObservabilityPipelineDedupeProcessor(
enabled=True,
id="dedupe-processor",
include="service:my-service",
type=ObservabilityPipelineDedupeProcessorType.DEDUPE,
fields=[
"message",
],
mode=ObservabilityPipelineDedupeProcessorMode.MATCH,
),
],
),
],
sources=[
ObservabilityPipelineDatadogAgentSource(
id="datadog-agent-source",
type=ObservabilityPipelineDatadogAgentSourceType.DATADOG_AGENT,
),
],
),
name="Pipeline with Dedupe No Cache",
),
type="pipelines",
),
)
configuration = Configuration()
configuration.access_token = environ["DD_BEARER_TOKEN"]
with ApiClient(configuration) as api_client:
api_instance = ObservabilityPipelinesApi(api_client)
response = api_instance.create_pipeline(body=body)
print(response)
Instructions
First install the library and its dependencies and then save the example to example.py and run following commands:
DD_SITE="datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" python3 "example.py"DD_SITE="us3.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" python3 "example.py"DD_SITE="us5.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" python3 "example.py"DD_SITE="datadoghq.eu" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" python3 "example.py"DD_SITE="ap1.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" python3 "example.py"DD_SITE="ap2.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" python3 "example.py"DD_SITE="uk1.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" python3 "example.py"DD_SITE="ddog-gov.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" python3 "example.py"DD_SITE="us2.ddog-gov.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" python3 "example.py"Create a new pipeline returns "OK" response
# Create a new pipeline returns "OK" response
require "datadog_api_client"
DatadogAPIClient.configure do |config|
config.access_token = ENV["DD_BEARER_TOKEN"]
end
api_instance = DatadogAPIClient::V2::ObservabilityPipelinesAPI.new
body = DatadogAPIClient::V2::ObservabilityPipelineSpec.new({
data: DatadogAPIClient::V2::ObservabilityPipelineSpecData.new({
attributes: DatadogAPIClient::V2::ObservabilityPipelineDataAttributes.new({
config: DatadogAPIClient::V2::ObservabilityPipelineConfig.new({
destinations: [
DatadogAPIClient::V2::ObservabilityPipelineDatadogLogsDestination.new({
id: "datadog-logs-destination",
inputs: [
"my-processor-group",
],
type: DatadogAPIClient::V2::ObservabilityPipelineDatadogLogsDestinationType::DATADOG_LOGS,
}),
],
processor_groups: [
DatadogAPIClient::V2::ObservabilityPipelineConfigProcessorGroup.new({
enabled: true,
id: "my-processor-group",
include: "service:my-service",
inputs: [
"datadog-agent-source",
],
processors: [
DatadogAPIClient::V2::ObservabilityPipelineFilterProcessor.new({
enabled: true,
id: "filter-processor",
include: "status:error",
type: DatadogAPIClient::V2::ObservabilityPipelineFilterProcessorType::FILTER,
}),
],
}),
],
sources: [
DatadogAPIClient::V2::ObservabilityPipelineDatadogAgentSource.new({
id: "datadog-agent-source",
type: DatadogAPIClient::V2::ObservabilityPipelineDatadogAgentSourceType::DATADOG_AGENT,
}),
],
}),
name: "Main Observability Pipeline",
}),
type: "pipelines",
}),
})
p api_instance.create_pipeline(body)
Create a pipeline with dedupe processor with cache returns "OK" response
# Create a pipeline with dedupe processor with cache returns "OK" response
require "datadog_api_client"
DatadogAPIClient.configure do |config|
config.access_token = ENV["DD_BEARER_TOKEN"]
end
api_instance = DatadogAPIClient::V2::ObservabilityPipelinesAPI.new
body = DatadogAPIClient::V2::ObservabilityPipelineSpec.new({
data: DatadogAPIClient::V2::ObservabilityPipelineSpecData.new({
attributes: DatadogAPIClient::V2::ObservabilityPipelineDataAttributes.new({
config: DatadogAPIClient::V2::ObservabilityPipelineConfig.new({
destinations: [
DatadogAPIClient::V2::ObservabilityPipelineDatadogLogsDestination.new({
id: "datadog-logs-destination",
inputs: [
"my-processor-group",
],
type: DatadogAPIClient::V2::ObservabilityPipelineDatadogLogsDestinationType::DATADOG_LOGS,
}),
],
processor_groups: [
DatadogAPIClient::V2::ObservabilityPipelineConfigProcessorGroup.new({
enabled: true,
id: "my-processor-group",
include: "service:my-service",
inputs: [
"datadog-agent-source",
],
processors: [
DatadogAPIClient::V2::ObservabilityPipelineDedupeProcessor.new({
enabled: true,
id: "dedupe-processor",
include: "service:my-service",
type: DatadogAPIClient::V2::ObservabilityPipelineDedupeProcessorType::DEDUPE,
fields: [
"message",
],
mode: DatadogAPIClient::V2::ObservabilityPipelineDedupeProcessorMode::MATCH,
cache: DatadogAPIClient::V2::ObservabilityPipelineDedupeProcessorCache.new({
num_events: 5000,
}),
}),
],
}),
],
sources: [
DatadogAPIClient::V2::ObservabilityPipelineDatadogAgentSource.new({
id: "datadog-agent-source",
type: DatadogAPIClient::V2::ObservabilityPipelineDatadogAgentSourceType::DATADOG_AGENT,
}),
],
}),
name: "Pipeline with Dedupe Cache",
}),
type: "pipelines",
}),
})
p api_instance.create_pipeline(body)
Create a pipeline with dedupe processor without cache returns "OK" response
# Create a pipeline with dedupe processor without cache returns "OK" response
require "datadog_api_client"
DatadogAPIClient.configure do |config|
config.access_token = ENV["DD_BEARER_TOKEN"]
end
api_instance = DatadogAPIClient::V2::ObservabilityPipelinesAPI.new
body = DatadogAPIClient::V2::ObservabilityPipelineSpec.new({
data: DatadogAPIClient::V2::ObservabilityPipelineSpecData.new({
attributes: DatadogAPIClient::V2::ObservabilityPipelineDataAttributes.new({
config: DatadogAPIClient::V2::ObservabilityPipelineConfig.new({
destinations: [
DatadogAPIClient::V2::ObservabilityPipelineDatadogLogsDestination.new({
id: "datadog-logs-destination",
inputs: [
"my-processor-group",
],
type: DatadogAPIClient::V2::ObservabilityPipelineDatadogLogsDestinationType::DATADOG_LOGS,
}),
],
processor_groups: [
DatadogAPIClient::V2::ObservabilityPipelineConfigProcessorGroup.new({
enabled: true,
id: "my-processor-group",
include: "service:my-service",
inputs: [
"datadog-agent-source",
],
processors: [
DatadogAPIClient::V2::ObservabilityPipelineDedupeProcessor.new({
enabled: true,
id: "dedupe-processor",
include: "service:my-service",
type: DatadogAPIClient::V2::ObservabilityPipelineDedupeProcessorType::DEDUPE,
fields: [
"message",
],
mode: DatadogAPIClient::V2::ObservabilityPipelineDedupeProcessorMode::MATCH,
}),
],
}),
],
sources: [
DatadogAPIClient::V2::ObservabilityPipelineDatadogAgentSource.new({
id: "datadog-agent-source",
type: DatadogAPIClient::V2::ObservabilityPipelineDatadogAgentSourceType::DATADOG_AGENT,
}),
],
}),
name: "Pipeline with Dedupe No Cache",
}),
type: "pipelines",
}),
})
p api_instance.create_pipeline(body)
Instructions
First install the library and its dependencies and then save the example to example.rb and run following commands:
DD_SITE="datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" rb "example.rb"DD_SITE="us3.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" rb "example.rb"DD_SITE="us5.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" rb "example.rb"DD_SITE="datadoghq.eu" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" rb "example.rb"DD_SITE="ap1.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" rb "example.rb"DD_SITE="ap2.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" rb "example.rb"DD_SITE="uk1.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" rb "example.rb"DD_SITE="ddog-gov.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" rb "example.rb"DD_SITE="us2.ddog-gov.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" rb "example.rb"Create a new pipeline returns "OK" response
// Create a new pipeline returns "OK" response
use datadog_api_client::datadog;
use datadog_api_client::datadogV2::api_observability_pipelines::ObservabilityPipelinesAPI;
use datadog_api_client::datadogV2::model::ObservabilityPipelineConfig;
use datadog_api_client::datadogV2::model::ObservabilityPipelineConfigDestinationItem;
use datadog_api_client::datadogV2::model::ObservabilityPipelineConfigProcessorGroup;
use datadog_api_client::datadogV2::model::ObservabilityPipelineConfigProcessorItem;
use datadog_api_client::datadogV2::model::ObservabilityPipelineConfigSourceItem;
use datadog_api_client::datadogV2::model::ObservabilityPipelineDataAttributes;
use datadog_api_client::datadogV2::model::ObservabilityPipelineDatadogAgentSource;
use datadog_api_client::datadogV2::model::ObservabilityPipelineDatadogAgentSourceType;
use datadog_api_client::datadogV2::model::ObservabilityPipelineDatadogLogsDestination;
use datadog_api_client::datadogV2::model::ObservabilityPipelineDatadogLogsDestinationType;
use datadog_api_client::datadogV2::model::ObservabilityPipelineFilterProcessor;
use datadog_api_client::datadogV2::model::ObservabilityPipelineFilterProcessorType;
use datadog_api_client::datadogV2::model::ObservabilityPipelineSpec;
use datadog_api_client::datadogV2::model::ObservabilityPipelineSpecData;
#[tokio::main]
async fn main() {
let body =
ObservabilityPipelineSpec::new(
ObservabilityPipelineSpecData::new(
ObservabilityPipelineDataAttributes::new(
ObservabilityPipelineConfig::new(
vec![
ObservabilityPipelineConfigDestinationItem::ObservabilityPipelineDatadogLogsDestination(
Box::new(
ObservabilityPipelineDatadogLogsDestination::new(
"datadog-logs-destination".to_string(),
vec!["my-processor-group".to_string()],
ObservabilityPipelineDatadogLogsDestinationType::DATADOG_LOGS,
),
),
)
],
vec![
ObservabilityPipelineConfigSourceItem::ObservabilityPipelineDatadogAgentSource(
Box::new(
ObservabilityPipelineDatadogAgentSource::new(
"datadog-agent-source".to_string(),
ObservabilityPipelineDatadogAgentSourceType::DATADOG_AGENT,
),
),
)
],
).processor_groups(
vec![
ObservabilityPipelineConfigProcessorGroup::new(
true,
"my-processor-group".to_string(),
"service:my-service".to_string(),
vec!["datadog-agent-source".to_string()],
vec![
ObservabilityPipelineConfigProcessorItem::ObservabilityPipelineFilterProcessor(
Box::new(
ObservabilityPipelineFilterProcessor::new(
true,
"filter-processor".to_string(),
"status:error".to_string(),
ObservabilityPipelineFilterProcessorType::FILTER,
),
),
)
],
)
],
),
"Main Observability Pipeline".to_string(),
),
"pipelines".to_string(),
),
);
let configuration = datadog::Configuration::new();
let api = ObservabilityPipelinesAPI::with_config(configuration);
let resp = api.create_pipeline(body).await;
if let Ok(value) = resp {
println!("{:#?}", value);
} else {
println!("{:#?}", resp.unwrap_err());
}
}
Create a pipeline with dedupe processor with cache returns "OK" response
// Create a pipeline with dedupe processor with cache returns "OK" response
use datadog_api_client::datadog;
use datadog_api_client::datadogV2::api_observability_pipelines::ObservabilityPipelinesAPI;
use datadog_api_client::datadogV2::model::ObservabilityPipelineConfig;
use datadog_api_client::datadogV2::model::ObservabilityPipelineConfigDestinationItem;
use datadog_api_client::datadogV2::model::ObservabilityPipelineConfigProcessorGroup;
use datadog_api_client::datadogV2::model::ObservabilityPipelineConfigProcessorItem;
use datadog_api_client::datadogV2::model::ObservabilityPipelineConfigSourceItem;
use datadog_api_client::datadogV2::model::ObservabilityPipelineDataAttributes;
use datadog_api_client::datadogV2::model::ObservabilityPipelineDatadogAgentSource;
use datadog_api_client::datadogV2::model::ObservabilityPipelineDatadogAgentSourceType;
use datadog_api_client::datadogV2::model::ObservabilityPipelineDatadogLogsDestination;
use datadog_api_client::datadogV2::model::ObservabilityPipelineDatadogLogsDestinationType;
use datadog_api_client::datadogV2::model::ObservabilityPipelineDedupeProcessor;
use datadog_api_client::datadogV2::model::ObservabilityPipelineDedupeProcessorCache;
use datadog_api_client::datadogV2::model::ObservabilityPipelineDedupeProcessorMode;
use datadog_api_client::datadogV2::model::ObservabilityPipelineDedupeProcessorType;
use datadog_api_client::datadogV2::model::ObservabilityPipelineSpec;
use datadog_api_client::datadogV2::model::ObservabilityPipelineSpecData;
#[tokio::main]
async fn main() {
let body =
ObservabilityPipelineSpec::new(
ObservabilityPipelineSpecData::new(
ObservabilityPipelineDataAttributes::new(
ObservabilityPipelineConfig::new(
vec![
ObservabilityPipelineConfigDestinationItem::ObservabilityPipelineDatadogLogsDestination(
Box::new(
ObservabilityPipelineDatadogLogsDestination::new(
"datadog-logs-destination".to_string(),
vec!["my-processor-group".to_string()],
ObservabilityPipelineDatadogLogsDestinationType::DATADOG_LOGS,
),
),
)
],
vec![
ObservabilityPipelineConfigSourceItem::ObservabilityPipelineDatadogAgentSource(
Box::new(
ObservabilityPipelineDatadogAgentSource::new(
"datadog-agent-source".to_string(),
ObservabilityPipelineDatadogAgentSourceType::DATADOG_AGENT,
),
),
)
],
).processor_groups(
vec![
ObservabilityPipelineConfigProcessorGroup::new(
true,
"my-processor-group".to_string(),
"service:my-service".to_string(),
vec!["datadog-agent-source".to_string()],
vec![
ObservabilityPipelineConfigProcessorItem::ObservabilityPipelineDedupeProcessor(
Box::new(
ObservabilityPipelineDedupeProcessor::new(
true,
vec!["message".to_string()],
"dedupe-processor".to_string(),
"service:my-service".to_string(),
ObservabilityPipelineDedupeProcessorMode::MATCH,
ObservabilityPipelineDedupeProcessorType::DEDUPE,
).cache(ObservabilityPipelineDedupeProcessorCache::new(5000)),
),
)
],
)
],
),
"Pipeline with Dedupe Cache".to_string(),
),
"pipelines".to_string(),
),
);
let configuration = datadog::Configuration::new();
let api = ObservabilityPipelinesAPI::with_config(configuration);
let resp = api.create_pipeline(body).await;
if let Ok(value) = resp {
println!("{:#?}", value);
} else {
println!("{:#?}", resp.unwrap_err());
}
}
Create a pipeline with dedupe processor without cache returns "OK" response
// Create a pipeline with dedupe processor without cache returns "OK" response
use datadog_api_client::datadog;
use datadog_api_client::datadogV2::api_observability_pipelines::ObservabilityPipelinesAPI;
use datadog_api_client::datadogV2::model::ObservabilityPipelineConfig;
use datadog_api_client::datadogV2::model::ObservabilityPipelineConfigDestinationItem;
use datadog_api_client::datadogV2::model::ObservabilityPipelineConfigProcessorGroup;
use datadog_api_client::datadogV2::model::ObservabilityPipelineConfigProcessorItem;
use datadog_api_client::datadogV2::model::ObservabilityPipelineConfigSourceItem;
use datadog_api_client::datadogV2::model::ObservabilityPipelineDataAttributes;
use datadog_api_client::datadogV2::model::ObservabilityPipelineDatadogAgentSource;
use datadog_api_client::datadogV2::model::ObservabilityPipelineDatadogAgentSourceType;
use datadog_api_client::datadogV2::model::ObservabilityPipelineDatadogLogsDestination;
use datadog_api_client::datadogV2::model::ObservabilityPipelineDatadogLogsDestinationType;
use datadog_api_client::datadogV2::model::ObservabilityPipelineDedupeProcessor;
use datadog_api_client::datadogV2::model::ObservabilityPipelineDedupeProcessorMode;
use datadog_api_client::datadogV2::model::ObservabilityPipelineDedupeProcessorType;
use datadog_api_client::datadogV2::model::ObservabilityPipelineSpec;
use datadog_api_client::datadogV2::model::ObservabilityPipelineSpecData;
#[tokio::main]
async fn main() {
let body =
ObservabilityPipelineSpec::new(
ObservabilityPipelineSpecData::new(
ObservabilityPipelineDataAttributes::new(
ObservabilityPipelineConfig::new(
vec![
ObservabilityPipelineConfigDestinationItem::ObservabilityPipelineDatadogLogsDestination(
Box::new(
ObservabilityPipelineDatadogLogsDestination::new(
"datadog-logs-destination".to_string(),
vec!["my-processor-group".to_string()],
ObservabilityPipelineDatadogLogsDestinationType::DATADOG_LOGS,
),
),
)
],
vec![
ObservabilityPipelineConfigSourceItem::ObservabilityPipelineDatadogAgentSource(
Box::new(
ObservabilityPipelineDatadogAgentSource::new(
"datadog-agent-source".to_string(),
ObservabilityPipelineDatadogAgentSourceType::DATADOG_AGENT,
),
),
)
],
).processor_groups(
vec![
ObservabilityPipelineConfigProcessorGroup::new(
true,
"my-processor-group".to_string(),
"service:my-service".to_string(),
vec!["datadog-agent-source".to_string()],
vec![
ObservabilityPipelineConfigProcessorItem::ObservabilityPipelineDedupeProcessor(
Box::new(
ObservabilityPipelineDedupeProcessor::new(
true,
vec!["message".to_string()],
"dedupe-processor".to_string(),
"service:my-service".to_string(),
ObservabilityPipelineDedupeProcessorMode::MATCH,
ObservabilityPipelineDedupeProcessorType::DEDUPE,
),
),
)
],
)
],
),
"Pipeline with Dedupe No Cache".to_string(),
),
"pipelines".to_string(),
),
);
let configuration = datadog::Configuration::new();
let api = ObservabilityPipelinesAPI::with_config(configuration);
let resp = api.create_pipeline(body).await;
if let Ok(value) = resp {
println!("{:#?}", value);
} else {
println!("{:#?}", resp.unwrap_err());
}
}
Instructions
First install the library and its dependencies and then save the example to src/main.rs and run following commands:
DD_SITE="datadoghq.com" DD_API_KEY="<API-KEY>" DD_APP_KEY="<APP-KEY>" cargo runDD_SITE="us3.datadoghq.com" DD_API_KEY="<API-KEY>" DD_APP_KEY="<APP-KEY>" cargo runDD_SITE="us5.datadoghq.com" DD_API_KEY="<API-KEY>" DD_APP_KEY="<APP-KEY>" cargo runDD_SITE="datadoghq.eu" DD_API_KEY="<API-KEY>" DD_APP_KEY="<APP-KEY>" cargo runDD_SITE="ap1.datadoghq.com" DD_API_KEY="<API-KEY>" DD_APP_KEY="<APP-KEY>" cargo runDD_SITE="ap2.datadoghq.com" DD_API_KEY="<API-KEY>" DD_APP_KEY="<APP-KEY>" cargo runDD_SITE="uk1.datadoghq.com" DD_API_KEY="<API-KEY>" DD_APP_KEY="<APP-KEY>" cargo runDD_SITE="ddog-gov.com" DD_API_KEY="<API-KEY>" DD_APP_KEY="<APP-KEY>" cargo runDD_SITE="us2.ddog-gov.com" DD_API_KEY="<API-KEY>" DD_APP_KEY="<APP-KEY>" cargo runCreate a new pipeline returns "OK" response
/**
* Create a new pipeline returns "OK" response
*/
import { client, v2 } from "@datadog/datadog-api-client";
const configuration = client.createConfiguration({
authMethods: {
AuthZ: {
accessToken: process.env.DD_BEARER_TOKEN as string,
},
},
});
const apiInstance = new v2.ObservabilityPipelinesApi(configuration);
const params: v2.ObservabilityPipelinesApiCreatePipelineRequest = {
body: {
data: {
attributes: {
config: {
destinations: [
{
id: "datadog-logs-destination",
inputs: ["my-processor-group"],
type: "datadog_logs",
},
],
processorGroups: [
{
enabled: true,
id: "my-processor-group",
include: "service:my-service",
inputs: ["datadog-agent-source"],
processors: [
{
enabled: true,
id: "filter-processor",
include: "status:error",
type: "filter",
},
],
},
],
sources: [
{
id: "datadog-agent-source",
type: "datadog_agent",
},
],
},
name: "Main Observability Pipeline",
},
type: "pipelines",
},
},
};
apiInstance
.createPipeline(params)
.then((data: v2.ObservabilityPipeline) => {
console.log(
"API called successfully. Returned data: " + JSON.stringify(data)
);
})
.catch((error: any) => console.error(error));
Create a pipeline with dedupe processor with cache returns "OK" response
/**
* Create a pipeline with dedupe processor with cache returns "OK" response
*/
import { client, v2 } from "@datadog/datadog-api-client";
const configuration = client.createConfiguration({
authMethods: {
AuthZ: {
accessToken: process.env.DD_BEARER_TOKEN as string,
},
},
});
const apiInstance = new v2.ObservabilityPipelinesApi(configuration);
const params: v2.ObservabilityPipelinesApiCreatePipelineRequest = {
body: {
data: {
attributes: {
config: {
destinations: [
{
id: "datadog-logs-destination",
inputs: ["my-processor-group"],
type: "datadog_logs",
},
],
processorGroups: [
{
enabled: true,
id: "my-processor-group",
include: "service:my-service",
inputs: ["datadog-agent-source"],
processors: [
{
enabled: true,
id: "dedupe-processor",
include: "service:my-service",
type: "dedupe",
fields: ["message"],
mode: "match",
cache: {
numEvents: 5000,
},
},
],
},
],
sources: [
{
id: "datadog-agent-source",
type: "datadog_agent",
},
],
},
name: "Pipeline with Dedupe Cache",
},
type: "pipelines",
},
},
};
apiInstance
.createPipeline(params)
.then((data: v2.ObservabilityPipeline) => {
console.log(
"API called successfully. Returned data: " + JSON.stringify(data)
);
})
.catch((error: any) => console.error(error));
Create a pipeline with dedupe processor without cache returns "OK" response
/**
* Create a pipeline with dedupe processor without cache returns "OK" response
*/
import { client, v2 } from "@datadog/datadog-api-client";
const configuration = client.createConfiguration({
authMethods: {
AuthZ: {
accessToken: process.env.DD_BEARER_TOKEN as string,
},
},
});
const apiInstance = new v2.ObservabilityPipelinesApi(configuration);
const params: v2.ObservabilityPipelinesApiCreatePipelineRequest = {
body: {
data: {
attributes: {
config: {
destinations: [
{
id: "datadog-logs-destination",
inputs: ["my-processor-group"],
type: "datadog_logs",
},
],
processorGroups: [
{
enabled: true,
id: "my-processor-group",
include: "service:my-service",
inputs: ["datadog-agent-source"],
processors: [
{
enabled: true,
id: "dedupe-processor",
include: "service:my-service",
type: "dedupe",
fields: ["message"],
mode: "match",
},
],
},
],
sources: [
{
id: "datadog-agent-source",
type: "datadog_agent",
},
],
},
name: "Pipeline with Dedupe No Cache",
},
type: "pipelines",
},
},
};
apiInstance
.createPipeline(params)
.then((data: v2.ObservabilityPipeline) => {
console.log(
"API called successfully. Returned data: " + JSON.stringify(data)
);
})
.catch((error: any) => console.error(error));
Instructions
First install the library and its dependencies and then save the example to example.ts and run following commands:
DD_SITE="datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" tsc "example.ts"DD_SITE="us3.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" tsc "example.ts"DD_SITE="us5.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" tsc "example.ts"DD_SITE="datadoghq.eu" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" tsc "example.ts"DD_SITE="ap1.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" tsc "example.ts"DD_SITE="ap2.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" tsc "example.ts"DD_SITE="uk1.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" tsc "example.ts"DD_SITE="ddog-gov.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" tsc "example.ts"DD_SITE="us2.ddog-gov.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" tsc "example.ts"