Create a pipeline
https://api.datadoghq.com/api/v1/logs/config/pipelineshttps://api.us3.datadoghq.com/api/v1/logs/config/pipelineshttps://api.us5.datadoghq.com/api/v1/logs/config/pipelineshttps://api.datadoghq.eu/api/v1/logs/config/pipelineshttps://api.ap1.datadoghq.com/api/v1/logs/config/pipelineshttps://api.ap2.datadoghq.com/api/v1/logs/config/pipelineshttps://api.uk1.datadoghq.com/api/v1/logs/config/pipelineshttps://api.ddog-gov.com/api/v1/logs/config/pipelineshttps://api.us2.ddog-gov.com/api/v1/logs/config/pipelinesOverview
Create a pipeline in your organization.
Request
Body Data (required)
Definition of the new pipeline.
descriptionA description of the pipeline.
filterFilter for logs.
queryThe filter query.
is_enabledWhether or not the pipeline is enabled.
name [required]Name of the pipeline.
processorsOrdered list of processors in this pipeline.
<type=grok-parser>Create custom grok rules to parse the full message or a specific attribute of your raw event. For more information, see the parsing section.
grok [required]Set of rules for the grok parser.
match_rules [required]List of match rules for the grok parser, separated by a new line.
support_rulesList of support rules for the grok parser, separated by a new line.
""is_enabledWhether or not the processor is enabled.
falsenameName of the processor.
samplesList of sample logs to test this grok parser.
source [required]Name of the log attribute to parse.
messagetype [required]Type of logs grok parser.
grok-parsergrok-parser<type=date-remapper>As Datadog receives logs, it timestamps them using the value(s) from any of these default attributes.
timestampdate_timestampTimestampeventTimepublished_date
If your logs put their dates in an attribute not in this list, use the log date Remapper Processor to define their date attribute as the official log timestamp. The recognized date formats are ISO8601, UNIX (the milliseconds EPOCH format), and RFC3164.
Note: If your logs don’t contain any of the default attributes and you haven’t defined your own date attribute, Datadog timestamps the logs with the date it received them.
If multiple log date remapper processors can be applied to a given log, only the first one (according to the pipelines order) is taken into account.
is_enabledWhether or not the processor is enabled.
falsenameName of the processor.
sources [required]Array of source attributes.
type [required]Type of logs date remapper.
date-remapperdate-remapper<type=status-remapper>Use this Processor if you want to assign some attributes as the official status.
Each incoming status value is mapped as follows.
- Integers from 0 to 7 map to the Syslog severity standards
- Strings beginning with
emergor f (case-insensitive) map toemerg(0) - Strings beginning with
a(case-insensitive) map toalert(1) - Strings beginning with
c(case-insensitive) map tocritical(2) - Strings beginning with
err(case-insensitive) map toerror(3) - Strings beginning with
w(case-insensitive) map towarning(4) - Strings beginning with
n(case-insensitive) map tonotice(5) - Strings beginning with
i(case-insensitive) map toinfo(6) - Strings beginning with
d,traceorverbose(case-insensitive) map todebug(7) - Strings beginning with
oor matchingOKorSuccess(case-insensitive) map to OK - All others map to
info(6)
Note: If multiple log status remapper processors can be applied to a given log, only the first one (according to the pipelines order) is taken into account.
is_enabledWhether or not the processor is enabled.
falsenameName of the processor.
sources [required]Array of source attributes.
type [required]Type of logs status remapper.
status-remapperstatus-remapper<type=service-remapper>Use this processor if you want to assign one or more attributes as the official service.
Note: If multiple service remapper processors can be applied to a given log, only the first one (according to the pipeline order) is taken into account.
is_enabledWhether or not the processor is enabled.
falsenameName of the processor.
sources [required]Array of source attributes.
type [required]Type of logs service remapper.
service-remapperservice-remapper<type=message-remapper>The message is a key attribute in Datadog. It is displayed in the message column of the Log Explorer and you can do full string search on it. Use this Processor to define one or more attributes as the official log message.
Note: If multiple log message remapper processors can be applied to a given log, only the first one (according to the pipeline order) is taken into account.
is_enabledWhether or not the processor is enabled.
falsenameName of the processor.
sources [required]Array of source attributes.
type [required]Type of logs message remapper.
message-remappermessage-remapper<type=attribute-remapper>The remapper processor remaps any source attribute(s) or tag to another target attribute or tag.
Constraints on the tag/attribute name are explained in the Tag Best Practice documentation.
Some additional constraints are applied as : or , are not allowed in the target tag/attribute name.
is_enabledWhether or not the processor is enabled.
falsenameName of the processor.
override_on_conflictWhether to override the target element if it's already set.
falsepreserve_sourceRemove or preserve the remapped source element.
falsesource_typeDefines if the sources are from log attribute or tag.
attributesources [required]Array of source attributes.
target [required]Final attribute or tag name to remap the sources to.
target_formatIf the target_type of the remapper is attribute, try to cast the value to a new specific type.
If the cast is not possible, the original type is kept. string, integer, or double are the possible types.
If the target_type is tag, this parameter may not be specified.
auto, string, integer, doubletarget_typeDefines if the final attribute or tag name is from log attribute or tag.
attributetype [required]Type of logs attribute remapper.
attribute-remapperattribute-remapper<type=url-parser>This processor extracts query parameters and other important parameters from a URL.
is_enabledWhether or not the processor is enabled.
falsenameName of the processor.
normalize_ending_slashesNormalize the ending slashes or not.
falsesources [required]Array of source attributes.
target [required]Name of the parent attribute that contains all the extracted details from the sources.
http.url_detailstype [required]Type of logs URL parser.
url-parserurl-parser<type=user-agent-parser>The User-Agent parser takes a User-Agent attribute and extracts the OS, browser, device, and other user data. It recognizes major bots like the Google Bot, Yahoo Slurp, and Bing.
is_enabledWhether or not the processor is enabled.
falseis_encodedDefine if the source attribute is URL encoded or not.
falsenameName of the processor.
sources [required]Array of source attributes.
target [required]Name of the parent attribute that contains all the extracted details from the sources.
http.useragent_detailstype [required]Type of logs User-Agent parser.
user-agent-parseruser-agent-parser<type=category-processor>Use the Category Processor to add a new attribute (without spaces or special characters in the new attribute name) to a log matching a provided search query. Use categories to create groups for an analytical view. For example, URL groups, machine groups, environments, and response time buckets.
Notes:
- The syntax of the query is the one of Logs Explorer search bar. The query can be done on any log attribute or tag, whether it is a facet or not. Wildcards can also be used inside your query.
- Once the log has matched one of the Processor queries, it stops. Make sure they are properly ordered in case a log could match several queries.
- The names of the categories must be unique.
- Once defined in the Category Processor, you can map categories to log status using the Log Status Remapper.
categories [required]Array of filters to match or not a log and their
corresponding name to assign a custom value to the log.
filterFilter for logs.
queryThe filter query.
nameValue to assign to the target attribute.
is_enabledWhether or not the processor is enabled.
falsenameName of the processor.
target [required]Name of the target attribute which value is defined by the matching category.
type [required]Type of logs category processor.
category-processorcategory-processor<type=arithmetic-processor>Use the Arithmetic Processor to add a new attribute (without spaces or special characters in the new attribute name) to a log with the result of the provided formula. This enables you to remap different time attributes with different units into a single attribute, or to compute operations on attributes within the same log.
The formula can use parentheses and the basic arithmetic operators -, +, *, /.
By default, the calculation is skipped if an attribute is missing. Select “Replace missing attribute by 0” to automatically populate missing attribute values with 0 to ensure that the calculation is done. An attribute is missing if it is not found in the log attributes, or if it cannot be converted to a number.
Notes:
- The operator
-needs to be space split in the formula as it can also be contained in attribute names. - If the target attribute already exists, it is overwritten by the result of the formula.
- Results are rounded up to the 9th decimal. For example, if the result of the formula is
0.1234567891, the actual value stored for the attribute is0.123456789. - If you need to scale a unit of measure, see Scale Filter.
expression [required]Arithmetic operation between one or more log attributes.
is_enabledWhether or not the processor is enabled.
falseis_replace_missingIf true, it replaces all missing attributes of expression by 0, false
skip the operation if an attribute is missing.
falsenameName of the processor.
target [required]Name of the attribute that contains the result of the arithmetic operation.
type [required]Type of logs arithmetic processor.
arithmetic-processorarithmetic-processor<type=string-builder-processor>Use the string builder processor to add a new attribute (without spaces or special characters) to a log with the result of the provided template. This enables aggregation of different attributes or raw strings into a single attribute.
The template is defined by both raw text and blocks with the syntax %{attribute_path}.
Notes:
- The processor only accepts attributes with values or an array of values in the blocks.
- If an attribute cannot be used (object or array of object), it is replaced by an empty string or the entire operation is skipped depending on your selection.
- If the target attribute already exists, it is overwritten by the result of the template.
- Results of the template cannot exceed 256 characters.
is_enabledWhether or not the processor is enabled.
falseis_replace_missingIf true, it replaces all missing attributes of template by an empty string.
If false (default), skips the operation for missing attributes.
falsenameName of the processor.
target [required]The name of the attribute that contains the result of the template.
template [required]A formula with one or more attributes and raw text.
type [required]Type of logs string builder processor.
string-builder-processorstring-builder-processor<type=pipeline>Nested Pipelines are pipelines within a pipeline. Use Nested Pipelines to split the processing into two steps. For example, first use a high-level filtering such as team and then a second level of filtering based on the integration, service, or any other tag or attribute.
A pipeline can contain Nested Pipelines and Processors whereas a Nested Pipeline can only contain Processors.
descriptionA description of the pipeline.
filterFilter for logs.
queryThe filter query.
is_enabledWhether or not the processor is enabled.
falsenameName of the processor.
processorsOrdered list of processors in this pipeline.
tagsA list of tags associated with the pipeline.
type [required]Type of logs pipeline processor.
pipelinepipeline<type=geo-ip-parser>The GeoIP parser takes an IP address attribute and extracts if available the Continent, Country, Subdivision, and City information in the target attribute path.
is_enabledWhether or not the processor is enabled.
falsenameName of the processor.
sources [required]Array of source attributes.
target [required]Name of the parent attribute that contains all the extracted details from the sources.
network.client.geoiptype [required]Type of GeoIP parser.
geo-ip-parsergeo-ip-parserObject 14Use the Lookup Processor to define a mapping between a log attribute and a human readable value saved in the processors mapping table. For example, you can use the Lookup Processor to map an internal service ID into a human readable service name. Alternatively, you could also use it to check if the MAC address that just attempted to connect to the production environment belongs to your list of stolen machines.
default_lookupValue to set the target attribute if the source value is not found in the list.
is_enabledWhether or not the processor is enabled.
falselookup_table [required]Mapping table of values for the source attribute and their associated target attribute values,
formatted as ["source_key1,target_value1", "source_key2,target_value2"]
nameName of the processor.
source [required]Source attribute used to perform the lookup.
target [required]Name of the attribute that contains the corresponding value in the mapping list
or the default_lookup if not found in the mapping list.
type [required]Type of logs lookup processor.
lookup-processorlookup-processorObject 15Note: Reference Tables are in public beta. Use the Lookup Processor to define a mapping between a log attribute and a human readable value saved in a Reference Table. For example, you can use the Lookup Processor to map an internal service ID into a human readable service name. Alternatively, you could also use it to check if the MAC address that just attempted to connect to the production environment belongs to your list of stolen machines.
is_enabledWhether or not the processor is enabled.
falselookup_enrichment_table [required]Name of the Reference Table for the source attribute and their associated target attribute values.
nameName of the processor.
source [required]Source attribute used to perform the lookup.
target [required]Name of the attribute that contains the corresponding value in the mapping list.
type [required]Type of logs lookup processor.
lookup-processorlookup-processor<type=trace-id-remapper>There are two ways to improve correlation between application traces and logs.
Follow the documentation on how to inject a trace ID in the application logs and by default log integrations take care of all the rest of the setup.
Use the Trace remapper processor to define a log attribute as its associated trace ID.
is_enabledWhether or not the processor is enabled.
falsenameName of the processor.
sourcesArray of source attributes.
type [required]Type of logs trace remapper.
trace-id-remappertrace-id-remapper<type=span-id-remapper>There are two ways to define correlation between application spans and logs:
Follow the documentation on how to inject a span ID in the application logs. Log integrations automatically handle all remaining setup steps by default.
Use the span remapper processor to define a log attribute as its associated span ID.
is_enabledWhether or not the processor is enabled.
falsenameName of the processor.
sourcesArray of source attributes.
type [required]Type of logs span remapper.
span-id-remapperspan-id-remapper<type=array-processor>A processor for extracting, aggregating, or transforming values from JSON arrays within your logs. Supported operations are:
- Select value from matching element
- Compute array length
- Append a value to an array
- Extract key-value pairs from an array
is_enabledWhether or not the processor is enabled.
falsenameName of the processor.
operation [required]Configuration of the array processor operation to perform.
<type=append>Operation that appends a value to a target array attribute.
preserve_sourceRemove or preserve the remapped source element.
truesource [required]Attribute path containing the value to append.
target [required]Attribute path of the array to append to.
type [required]Operation type.
append<type=length>Operation that computes the length of a source array and stores the result in the target attribute.
source [required]Attribute path of the array to measure.
target [required]Attribute that receives the computed length.
type [required]Operation type.
length<type=select>Operation that finds an object in a source array using a filter, and then extracts a specific value into the target attribute.
filter [required]Filter condition expressed as key:value used to find the matching element.
source [required]Attribute path of the array to search into.
target [required]Attribute that receives the extracted value.
type [required]Operation type.
selectvalue_to_extract [required]Key of the value to extract from the matching element.
<type=key-value>Operation that extracts key-value pairs from a source array and stores the result in the target attribute.
key_to_extract [required]Key of the attribute in each array element that holds the name to use for the extracted attribute.
override_on_conflictWhether to override the target element if it's already set.
falsesource [required]Attribute path of the array to extract key-value pairs from.
targetAttribute that receives the extracted key-value pairs. If not specified, the extracted attributes are added at the root level of the log.
type [required]Operation type.
key-valuevalue_to_extract [required]Key of the attribute in each array element that holds the value to use for the extracted attribute.
type [required]Type of logs array processor.
array-processorarray-processor<type=decoder-processor>The decoder processor decodes any source attribute containing a base64/base16-encoded UTF-8/ASCII string back to its original value, storing the result in a target attribute.
binary_to_text_encoding [required]The encoding used to represent the binary data.
base64, base16input_representation [required]The original representation of input string.
utf_8, integeris_enabledWhether the processor is enabled.
falsenameName of the processor.
source [required]Name of the log attribute with the encoded data.
target [required]Name of the log attribute that contains the decoded data.
type [required]Type of logs decoder processor.
decoder-processordecoder-processor<type=schema-processor>A processor that has additional validations and checks for a given schema. Currently supported schema types include OCSF.
is_enabledWhether or not the processor is enabled.
falsemappers [required]The LogsSchemaProcessor mappers.
<type=schema-remapper>The schema remapper maps source log fields to their correct fields.
name [required]Name of the logs schema remapper.
override_on_conflictWhether to override the target element if it's already set.
falsepreserve_sourceRemove or preserve the remapped source element.
falsesources [required]Array of source attributes.
target [required]Target field to map log source field to.
target_formatIf the target_type of the remapper is attribute, try to cast the value to a new specific type.
If the cast is not possible, the original type is kept. string, integer, or double are the possible types.
If the target_type is tag, this parameter may not be specified.
auto, string, integer, doubletype [required]Type of logs schema remapper.
schema-remapper<type=schema-category-mapper>Use the Schema Category Mapper to categorize log event into enum fields. In the case of OCSF, they can be used to map sibling fields which are composed of an ID and a name.
Notes:
- The syntax of the query is the one of Logs Explorer search bar. The query can be done on any log attribute or tag, whether it is a facet or not. Wildcards can also be used inside your query.
- Categories are executed in order and processing stops at the first match. Make sure categories are properly ordered in case a log could match multiple queries.
- Sibling fields always have a numerical ID field and a human-readable string name.
- A fallback section handles cases where the name or ID value matches a specific value. If the name matches "Other" or the ID matches 99, the value of the sibling name field will be pulled from a source field from the original log.
categories [required]Array of filters to match or not a log and their
corresponding name to assign a custom value to the log.
filter [required]Filter for logs.
queryThe filter query.
id [required]ID to inject into the category.
name [required]Value to assign to target schema field.
fallbackUsed to override hardcoded category values with a value pulled from a source attribute on the log.
sourcesFallback sources used to populate value of field.
<any-key>valuesValues that define when the fallback is used.
<any-key>name [required]Name of the logs schema category mapper.
targets [required]Name of the target attributes which value is defined by the matching category.
idID of the field to map log attributes to.
nameName of the field to map log attributes to.
type [required]Type of logs schema category mapper.
schema-category-mappername [required]Name of the processor.
schema [required]Configuration of the schema data to use.
class_name [required]Class name of the schema to use.
class_uid [required]Class UID of the schema to use.
profilesOptional list of profiles to modify the schema.
schema_type [required]Type of schema to use.
version [required]Version of the schema to use.
type [required]Type of logs schema processor.
schema-processorschema-processor<type=exclude-attribute>Use this processor to remove an attribute from a log during processing. The processor strips the specified attribute from the log event, which is useful when the attribute contains sensitive data or is no longer needed downstream.
attribute_to_exclude [required]Name of the log attribute to remove from the log event.
is_enabledWhether or not the processor is enabled.
falsenameName of the processor.
type [required]Type of logs exclude attribute processor.
exclude-attributeexclude-attribute<type=array-map-processor>The array-map processor transforms each element of a source array by applying
sub-processors in order and collecting the results into a target array.
Results can be written to a new array, to the source array (in-place), or to
an existing target array. Sub-processors can read from $sourceElem.<field>
(object element field), bare $sourceElem (primitive element), or any parent
log attribute path. Sub-processors write to $targetElem.<field> (object
output field) or bare $targetElem (primitive output).
is_enabledWhether or not the processor is enabled.
falsenameName of the processor.
preserve_sourceWhen false and source != target, the source attribute is removed after
processing. Cannot be false when source == target.
trueprocessors [required]Sub-processors applied to each element. Allowed types: attribute-remapper,
string-builder-processor, arithmetic-processor, category-processor.
<type=attribute-remapper>An attribute remapper sub-processor for use inside an array-map processor.
Unlike the top-level attribute remapper, is_enabled, source_type, and
target_type are not supported.
nameName of the sub-processor.
override_on_conflictOverride the target element if already set.
falsepreserve_sourceRemove or preserve the remapped source element.
falsesources [required]Array of source attribute paths.
target [required]Target attribute path.
target_formatIf the target_type of the remapper is attribute, try to cast the value to a new specific type.
If the cast is not possible, the original type is kept. string, integer, or double are the possible types.
If the target_type is tag, this parameter may not be specified.
auto, string, integer, doubletype [required]Type of logs attribute remapper.
attribute-remapperattribute-remapper<type=arithmetic-processor>An arithmetic sub-processor for use inside an array-map processor.
Unlike the top-level arithmetic processor, is_enabled is not supported.
expression [required]Arithmetic operation to perform.
is_replace_missingReplace missing attribute values with 0.
falsenameName of the sub-processor.
target [required]Target attribute path for the result.
type [required]Type of logs arithmetic processor.
arithmetic-processorarithmetic-processor<type=string-builder-processor>A string builder sub-processor for use inside an array-map processor.
Unlike the top-level string builder processor, is_enabled is not supported.
is_replace_missingReplace missing attribute values with an empty string.
falsenameName of the sub-processor.
target [required]Target attribute path for the result.
template [required]Formula with one or more attributes and raw text.
type [required]Type of logs string builder processor.
string-builder-processorstring-builder-processor<type=category-processor>A category sub-processor for use inside an array-map processor.
Unlike the top-level category processor, is_enabled is not supported.
categories [required]Array of filters to match against a log and the corresponding value to assign.
filterFilter for logs.
queryThe filter query.
nameValue to assign to the target attribute.
nameName of the sub-processor.
target [required]Target attribute path for the category value.
type [required]Type of logs category processor.
category-processorcategory-processorsource [required]Attribute path of the source array. Elements are read-only via $sourceElem
inside sub-processors.
target [required]Attribute path of the output array. Sub-processors write to $targetElem
(or $targetElem.<field>) to build each output element.
type [required]Type of logs array-map processor.
array-map-processorarray-map-processortagsA list of tags associated with the pipeline.
Create a pipeline with Array Map Processor returns "OK" response
{
"filter": {
"query": "source:python"
},
"name": "testPipelineArrayMap",
"processors": [
{
"type": "array-map-processor",
"is_enabled": true,
"name": "map items",
"source": "items",
"target": "out",
"preserve_source": true,
"processors": [
{
"type": "attribute-remapper",
"sources": [
"$sourceElem.id"
],
"target": "$targetElem.uid",
"preserve_source": true
},
{
"type": "string-builder-processor",
"template": "item-%{$sourceElem.id}",
"target": "$targetElem.label"
}
]
}
],
"tags": []
}Create a pipeline with Array Map Processor using arithmetic sub-processor returns "OK" response
{
"filter": {
"query": "source:python"
},
"name": "testPipelineArrayMapArithmetic",
"processors": [
{
"type": "array-map-processor",
"is_enabled": true,
"name": "double counts",
"source": "items",
"target": "out",
"processors": [
{
"type": "arithmetic-processor",
"expression": "$sourceElem.count * 2",
"target": "$targetElem.doubled"
}
]
}
],
"tags": []
}Create a pipeline with Array Map Processor using category sub-processor returns "OK" response
{
"filter": {
"query": "source:python"
},
"name": "testPipelineArrayMapCategory",
"processors": [
{
"type": "array-map-processor",
"is_enabled": true,
"name": "categorize items",
"source": "items",
"target": "out",
"processors": [
{
"type": "category-processor",
"target": "$targetElem.level",
"categories": [
{
"filter": {
"query": "@$sourceElem.status:error"
},
"name": "error"
},
{
"filter": {
"query": "*"
},
"name": "info"
}
]
}
]
}
],
"tags": []
}Response
OK
Pipelines and processors operate on incoming logs, parsing and transforming them into structured attributes for easier querying.
Note: These endpoints are only available for admin users. Make sure to use an application key created by an admin.
descriptionA description of the pipeline.
filterFilter for logs.
queryThe filter query.
idID of the pipeline.
is_enabledWhether or not the pipeline is enabled.
is_read_onlyWhether or not the pipeline can be edited.
name [required]Name of the pipeline.
processorsOrdered list of processors in this pipeline.
<type=grok-parser>Create custom grok rules to parse the full message or a specific attribute of your raw event. For more information, see the parsing section.
grok [required]Set of rules for the grok parser.
match_rules [required]List of match rules for the grok parser, separated by a new line.
support_rulesList of support rules for the grok parser, separated by a new line.
""is_enabledWhether or not the processor is enabled.
falsenameName of the processor.
samplesList of sample logs to test this grok parser.
source [required]Name of the log attribute to parse.
messagetype [required]Type of logs grok parser.
grok-parsergrok-parser<type=date-remapper>As Datadog receives logs, it timestamps them using the value(s) from any of these default attributes.
timestampdate_timestampTimestampeventTimepublished_date
If your logs put their dates in an attribute not in this list, use the log date Remapper Processor to define their date attribute as the official log timestamp. The recognized date formats are ISO8601, UNIX (the milliseconds EPOCH format), and RFC3164.
Note: If your logs don’t contain any of the default attributes and you haven’t defined your own date attribute, Datadog timestamps the logs with the date it received them.
If multiple log date remapper processors can be applied to a given log, only the first one (according to the pipelines order) is taken into account.
is_enabledWhether or not the processor is enabled.
falsenameName of the processor.
sources [required]Array of source attributes.
type [required]Type of logs date remapper.
date-remapperdate-remapper<type=status-remapper>Use this Processor if you want to assign some attributes as the official status.
Each incoming status value is mapped as follows.
- Integers from 0 to 7 map to the Syslog severity standards
- Strings beginning with
emergor f (case-insensitive) map toemerg(0) - Strings beginning with
a(case-insensitive) map toalert(1) - Strings beginning with
c(case-insensitive) map tocritical(2) - Strings beginning with
err(case-insensitive) map toerror(3) - Strings beginning with
w(case-insensitive) map towarning(4) - Strings beginning with
n(case-insensitive) map tonotice(5) - Strings beginning with
i(case-insensitive) map toinfo(6) - Strings beginning with
d,traceorverbose(case-insensitive) map todebug(7) - Strings beginning with
oor matchingOKorSuccess(case-insensitive) map to OK - All others map to
info(6)
Note: If multiple log status remapper processors can be applied to a given log, only the first one (according to the pipelines order) is taken into account.
is_enabledWhether or not the processor is enabled.
falsenameName of the processor.
sources [required]Array of source attributes.
type [required]Type of logs status remapper.
status-remapperstatus-remapper<type=service-remapper>Use this processor if you want to assign one or more attributes as the official service.
Note: If multiple service remapper processors can be applied to a given log, only the first one (according to the pipeline order) is taken into account.
is_enabledWhether or not the processor is enabled.
falsenameName of the processor.
sources [required]Array of source attributes.
type [required]Type of logs service remapper.
service-remapperservice-remapper<type=message-remapper>The message is a key attribute in Datadog. It is displayed in the message column of the Log Explorer and you can do full string search on it. Use this Processor to define one or more attributes as the official log message.
Note: If multiple log message remapper processors can be applied to a given log, only the first one (according to the pipeline order) is taken into account.
is_enabledWhether or not the processor is enabled.
falsenameName of the processor.
sources [required]Array of source attributes.
type [required]Type of logs message remapper.
message-remappermessage-remapper<type=attribute-remapper>The remapper processor remaps any source attribute(s) or tag to another target attribute or tag.
Constraints on the tag/attribute name are explained in the Tag Best Practice documentation.
Some additional constraints are applied as : or , are not allowed in the target tag/attribute name.
is_enabledWhether or not the processor is enabled.
falsenameName of the processor.
override_on_conflictWhether to override the target element if it's already set.
falsepreserve_sourceRemove or preserve the remapped source element.
falsesource_typeDefines if the sources are from log attribute or tag.
attributesources [required]Array of source attributes.
target [required]Final attribute or tag name to remap the sources to.
target_formatIf the target_type of the remapper is attribute, try to cast the value to a new specific type.
If the cast is not possible, the original type is kept. string, integer, or double are the possible types.
If the target_type is tag, this parameter may not be specified.
auto, string, integer, doubletarget_typeDefines if the final attribute or tag name is from log attribute or tag.
attributetype [required]Type of logs attribute remapper.
attribute-remapperattribute-remapper<type=url-parser>This processor extracts query parameters and other important parameters from a URL.
is_enabledWhether or not the processor is enabled.
falsenameName of the processor.
normalize_ending_slashesNormalize the ending slashes or not.
falsesources [required]Array of source attributes.
target [required]Name of the parent attribute that contains all the extracted details from the sources.
http.url_detailstype [required]Type of logs URL parser.
url-parserurl-parser<type=user-agent-parser>The User-Agent parser takes a User-Agent attribute and extracts the OS, browser, device, and other user data. It recognizes major bots like the Google Bot, Yahoo Slurp, and Bing.
is_enabledWhether or not the processor is enabled.
falseis_encodedDefine if the source attribute is URL encoded or not.
falsenameName of the processor.
sources [required]Array of source attributes.
target [required]Name of the parent attribute that contains all the extracted details from the sources.
http.useragent_detailstype [required]Type of logs User-Agent parser.
user-agent-parseruser-agent-parser<type=category-processor>Use the Category Processor to add a new attribute (without spaces or special characters in the new attribute name) to a log matching a provided search query. Use categories to create groups for an analytical view. For example, URL groups, machine groups, environments, and response time buckets.
Notes:
- The syntax of the query is the one of Logs Explorer search bar. The query can be done on any log attribute or tag, whether it is a facet or not. Wildcards can also be used inside your query.
- Once the log has matched one of the Processor queries, it stops. Make sure they are properly ordered in case a log could match several queries.
- The names of the categories must be unique.
- Once defined in the Category Processor, you can map categories to log status using the Log Status Remapper.
categories [required]Array of filters to match or not a log and their
corresponding name to assign a custom value to the log.
filterFilter for logs.
queryThe filter query.
nameValue to assign to the target attribute.
is_enabledWhether or not the processor is enabled.
falsenameName of the processor.
target [required]Name of the target attribute which value is defined by the matching category.
type [required]Type of logs category processor.
category-processorcategory-processor<type=arithmetic-processor>Use the Arithmetic Processor to add a new attribute (without spaces or special characters in the new attribute name) to a log with the result of the provided formula. This enables you to remap different time attributes with different units into a single attribute, or to compute operations on attributes within the same log.
The formula can use parentheses and the basic arithmetic operators -, +, *, /.
By default, the calculation is skipped if an attribute is missing. Select “Replace missing attribute by 0” to automatically populate missing attribute values with 0 to ensure that the calculation is done. An attribute is missing if it is not found in the log attributes, or if it cannot be converted to a number.
Notes:
- The operator
-needs to be space split in the formula as it can also be contained in attribute names. - If the target attribute already exists, it is overwritten by the result of the formula.
- Results are rounded up to the 9th decimal. For example, if the result of the formula is
0.1234567891, the actual value stored for the attribute is0.123456789. - If you need to scale a unit of measure, see Scale Filter.
expression [required]Arithmetic operation between one or more log attributes.
is_enabledWhether or not the processor is enabled.
falseis_replace_missingIf true, it replaces all missing attributes of expression by 0, false
skip the operation if an attribute is missing.
falsenameName of the processor.
target [required]Name of the attribute that contains the result of the arithmetic operation.
type [required]Type of logs arithmetic processor.
arithmetic-processorarithmetic-processor<type=string-builder-processor>Use the string builder processor to add a new attribute (without spaces or special characters) to a log with the result of the provided template. This enables aggregation of different attributes or raw strings into a single attribute.
The template is defined by both raw text and blocks with the syntax %{attribute_path}.
Notes:
- The processor only accepts attributes with values or an array of values in the blocks.
- If an attribute cannot be used (object or array of object), it is replaced by an empty string or the entire operation is skipped depending on your selection.
- If the target attribute already exists, it is overwritten by the result of the template.
- Results of the template cannot exceed 256 characters.
is_enabledWhether or not the processor is enabled.
falseis_replace_missingIf true, it replaces all missing attributes of template by an empty string.
If false (default), skips the operation for missing attributes.
falsenameName of the processor.
target [required]The name of the attribute that contains the result of the template.
template [required]A formula with one or more attributes and raw text.
type [required]Type of logs string builder processor.
string-builder-processorstring-builder-processor<type=pipeline>Nested Pipelines are pipelines within a pipeline. Use Nested Pipelines to split the processing into two steps. For example, first use a high-level filtering such as team and then a second level of filtering based on the integration, service, or any other tag or attribute.
A pipeline can contain Nested Pipelines and Processors whereas a Nested Pipeline can only contain Processors.
descriptionA description of the pipeline.
filterFilter for logs.
queryThe filter query.
is_enabledWhether or not the processor is enabled.
falsenameName of the processor.
processorsOrdered list of processors in this pipeline.
tagsA list of tags associated with the pipeline.
type [required]Type of logs pipeline processor.
pipelinepipeline<type=geo-ip-parser>The GeoIP parser takes an IP address attribute and extracts if available the Continent, Country, Subdivision, and City information in the target attribute path.
is_enabledWhether or not the processor is enabled.
falsenameName of the processor.
sources [required]Array of source attributes.
target [required]Name of the parent attribute that contains all the extracted details from the sources.
network.client.geoiptype [required]Type of GeoIP parser.
geo-ip-parsergeo-ip-parserObject 14Use the Lookup Processor to define a mapping between a log attribute and a human readable value saved in the processors mapping table. For example, you can use the Lookup Processor to map an internal service ID into a human readable service name. Alternatively, you could also use it to check if the MAC address that just attempted to connect to the production environment belongs to your list of stolen machines.
default_lookupValue to set the target attribute if the source value is not found in the list.
is_enabledWhether or not the processor is enabled.
falselookup_table [required]Mapping table of values for the source attribute and their associated target attribute values,
formatted as ["source_key1,target_value1", "source_key2,target_value2"]
nameName of the processor.
source [required]Source attribute used to perform the lookup.
target [required]Name of the attribute that contains the corresponding value in the mapping list
or the default_lookup if not found in the mapping list.
type [required]Type of logs lookup processor.
lookup-processorlookup-processorObject 15Note: Reference Tables are in public beta. Use the Lookup Processor to define a mapping between a log attribute and a human readable value saved in a Reference Table. For example, you can use the Lookup Processor to map an internal service ID into a human readable service name. Alternatively, you could also use it to check if the MAC address that just attempted to connect to the production environment belongs to your list of stolen machines.
is_enabledWhether or not the processor is enabled.
falselookup_enrichment_table [required]Name of the Reference Table for the source attribute and their associated target attribute values.
nameName of the processor.
source [required]Source attribute used to perform the lookup.
target [required]Name of the attribute that contains the corresponding value in the mapping list.
type [required]Type of logs lookup processor.
lookup-processorlookup-processor<type=trace-id-remapper>There are two ways to improve correlation between application traces and logs.
Follow the documentation on how to inject a trace ID in the application logs and by default log integrations take care of all the rest of the setup.
Use the Trace remapper processor to define a log attribute as its associated trace ID.
is_enabledWhether or not the processor is enabled.
falsenameName of the processor.
sourcesArray of source attributes.
type [required]Type of logs trace remapper.
trace-id-remappertrace-id-remapper<type=span-id-remapper>There are two ways to define correlation between application spans and logs:
Follow the documentation on how to inject a span ID in the application logs. Log integrations automatically handle all remaining setup steps by default.
Use the span remapper processor to define a log attribute as its associated span ID.
is_enabledWhether or not the processor is enabled.
falsenameName of the processor.
sourcesArray of source attributes.
type [required]Type of logs span remapper.
span-id-remapperspan-id-remapper<type=array-processor>A processor for extracting, aggregating, or transforming values from JSON arrays within your logs. Supported operations are:
- Select value from matching element
- Compute array length
- Append a value to an array
- Extract key-value pairs from an array
is_enabledWhether or not the processor is enabled.
falsenameName of the processor.
operation [required]Configuration of the array processor operation to perform.
<type=append>Operation that appends a value to a target array attribute.
preserve_sourceRemove or preserve the remapped source element.
truesource [required]Attribute path containing the value to append.
target [required]Attribute path of the array to append to.
type [required]Operation type.
append<type=length>Operation that computes the length of a source array and stores the result in the target attribute.
source [required]Attribute path of the array to measure.
target [required]Attribute that receives the computed length.
type [required]Operation type.
length<type=select>Operation that finds an object in a source array using a filter, and then extracts a specific value into the target attribute.
filter [required]Filter condition expressed as key:value used to find the matching element.
source [required]Attribute path of the array to search into.
target [required]Attribute that receives the extracted value.
type [required]Operation type.
selectvalue_to_extract [required]Key of the value to extract from the matching element.
<type=key-value>Operation that extracts key-value pairs from a source array and stores the result in the target attribute.
key_to_extract [required]Key of the attribute in each array element that holds the name to use for the extracted attribute.
override_on_conflictWhether to override the target element if it's already set.
falsesource [required]Attribute path of the array to extract key-value pairs from.
targetAttribute that receives the extracted key-value pairs. If not specified, the extracted attributes are added at the root level of the log.
type [required]Operation type.
key-valuevalue_to_extract [required]Key of the attribute in each array element that holds the value to use for the extracted attribute.
type [required]Type of logs array processor.
array-processorarray-processor<type=decoder-processor>The decoder processor decodes any source attribute containing a base64/base16-encoded UTF-8/ASCII string back to its original value, storing the result in a target attribute.
binary_to_text_encoding [required]The encoding used to represent the binary data.
base64, base16input_representation [required]The original representation of input string.
utf_8, integeris_enabledWhether the processor is enabled.
falsenameName of the processor.
source [required]Name of the log attribute with the encoded data.
target [required]Name of the log attribute that contains the decoded data.
type [required]Type of logs decoder processor.
decoder-processordecoder-processor<type=schema-processor>A processor that has additional validations and checks for a given schema. Currently supported schema types include OCSF.
is_enabledWhether or not the processor is enabled.
falsemappers [required]The LogsSchemaProcessor mappers.
<type=schema-remapper>The schema remapper maps source log fields to their correct fields.
name [required]Name of the logs schema remapper.
override_on_conflictWhether to override the target element if it's already set.
falsepreserve_sourceRemove or preserve the remapped source element.
falsesources [required]Array of source attributes.
target [required]Target field to map log source field to.
target_formatIf the target_type of the remapper is attribute, try to cast the value to a new specific type.
If the cast is not possible, the original type is kept. string, integer, or double are the possible types.
If the target_type is tag, this parameter may not be specified.
auto, string, integer, doubletype [required]Type of logs schema remapper.
schema-remapper<type=schema-category-mapper>Use the Schema Category Mapper to categorize log event into enum fields. In the case of OCSF, they can be used to map sibling fields which are composed of an ID and a name.
Notes:
- The syntax of the query is the one of Logs Explorer search bar. The query can be done on any log attribute or tag, whether it is a facet or not. Wildcards can also be used inside your query.
- Categories are executed in order and processing stops at the first match. Make sure categories are properly ordered in case a log could match multiple queries.
- Sibling fields always have a numerical ID field and a human-readable string name.
- A fallback section handles cases where the name or ID value matches a specific value. If the name matches "Other" or the ID matches 99, the value of the sibling name field will be pulled from a source field from the original log.
categories [required]Array of filters to match or not a log and their
corresponding name to assign a custom value to the log.
filter [required]Filter for logs.
queryThe filter query.
id [required]ID to inject into the category.
name [required]Value to assign to target schema field.
fallbackUsed to override hardcoded category values with a value pulled from a source attribute on the log.
sourcesFallback sources used to populate value of field.
<any-key>valuesValues that define when the fallback is used.
<any-key>name [required]Name of the logs schema category mapper.
targets [required]Name of the target attributes which value is defined by the matching category.
idID of the field to map log attributes to.
nameName of the field to map log attributes to.
type [required]Type of logs schema category mapper.
schema-category-mappername [required]Name of the processor.
schema [required]Configuration of the schema data to use.
class_name [required]Class name of the schema to use.
class_uid [required]Class UID of the schema to use.
profilesOptional list of profiles to modify the schema.
schema_type [required]Type of schema to use.
version [required]Version of the schema to use.
type [required]Type of logs schema processor.
schema-processorschema-processor<type=exclude-attribute>Use this processor to remove an attribute from a log during processing. The processor strips the specified attribute from the log event, which is useful when the attribute contains sensitive data or is no longer needed downstream.
attribute_to_exclude [required]Name of the log attribute to remove from the log event.
is_enabledWhether or not the processor is enabled.
falsenameName of the processor.
type [required]Type of logs exclude attribute processor.
exclude-attributeexclude-attribute<type=array-map-processor>The array-map processor transforms each element of a source array by applying
sub-processors in order and collecting the results into a target array.
Results can be written to a new array, to the source array (in-place), or to
an existing target array. Sub-processors can read from $sourceElem.<field>
(object element field), bare $sourceElem (primitive element), or any parent
log attribute path. Sub-processors write to $targetElem.<field> (object
output field) or bare $targetElem (primitive output).
is_enabledWhether or not the processor is enabled.
falsenameName of the processor.
preserve_sourceWhen false and source != target, the source attribute is removed after
processing. Cannot be false when source == target.
trueprocessors [required]Sub-processors applied to each element. Allowed types: attribute-remapper,
string-builder-processor, arithmetic-processor, category-processor.
<type=attribute-remapper>An attribute remapper sub-processor for use inside an array-map processor.
Unlike the top-level attribute remapper, is_enabled, source_type, and
target_type are not supported.
nameName of the sub-processor.
override_on_conflictOverride the target element if already set.
falsepreserve_sourceRemove or preserve the remapped source element.
falsesources [required]Array of source attribute paths.
target [required]Target attribute path.
target_formatIf the target_type of the remapper is attribute, try to cast the value to a new specific type.
If the cast is not possible, the original type is kept. string, integer, or double are the possible types.
If the target_type is tag, this parameter may not be specified.
auto, string, integer, doubletype [required]Type of logs attribute remapper.
attribute-remapperattribute-remapper<type=arithmetic-processor>An arithmetic sub-processor for use inside an array-map processor.
Unlike the top-level arithmetic processor, is_enabled is not supported.
expression [required]Arithmetic operation to perform.
is_replace_missingReplace missing attribute values with 0.
falsenameName of the sub-processor.
target [required]Target attribute path for the result.
type [required]Type of logs arithmetic processor.
arithmetic-processorarithmetic-processor<type=string-builder-processor>A string builder sub-processor for use inside an array-map processor.
Unlike the top-level string builder processor, is_enabled is not supported.
is_replace_missingReplace missing attribute values with an empty string.
falsenameName of the sub-processor.
target [required]Target attribute path for the result.
template [required]Formula with one or more attributes and raw text.
type [required]Type of logs string builder processor.
string-builder-processorstring-builder-processor<type=category-processor>A category sub-processor for use inside an array-map processor.
Unlike the top-level category processor, is_enabled is not supported.
categories [required]Array of filters to match against a log and the corresponding value to assign.
filterFilter for logs.
queryThe filter query.
nameValue to assign to the target attribute.
nameName of the sub-processor.
target [required]Target attribute path for the category value.
type [required]Type of logs category processor.
category-processorcategory-processorsource [required]Attribute path of the source array. Elements are read-only via $sourceElem
inside sub-processors.
target [required]Attribute path of the output array. Sub-processors write to $targetElem
(or $targetElem.<field>) to build each output element.
type [required]Type of logs array-map processor.
array-map-processorarray-map-processortagsA list of tags associated with the pipeline.
typeType of pipeline.
{
"description": "string",
"filter": {
"query": "source:python"
},
"id": "string",
"is_enabled": false,
"is_read_only": false,
"name": "",
"processors": [
{
"grok": {
"match_rules": "rule_name_1 foo\nrule_name_2 bar",
"support_rules": "rule_name_1 foo\nrule_name_2 bar"
},
"is_enabled": false,
"name": "string",
"samples": [],
"source": "message",
"type": "grok-parser"
}
],
"tags": [],
"type": "pipeline"
}Bad Request
Response returned by the Logs API when errors occur.
errorError returned by the Logs API
codeCode identifying the error
detailsAdditional error details
messageError message
{
"error": {
"code": "string",
"details": [],
"message": "string"
}
}Forbidden
Error response object.
errors [required]Array of errors returned by the API.
{
"errors": [
"Bad Request"
]
}Too many requests
Error response object.
errors [required]Array of errors returned by the API.
{
"errors": [
"Bad Request"
]
}Code Example
Create a pipeline with Array Map Processor returns "OK" response
# Set your Datadog site and credentials
export DD_SITE="datadoghq.com"
# Use a Personal Access Token or Service Access Token
export DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>"
curl -X POST "https://api.${DD_SITE}/api/v1/logs/config/pipelines" \
-H "Accept: application/json" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer ${DD_BEARER_TOKEN}" \
-d @- << EOF
{
"filter": {
"query": "source:python"
},
"name": "testPipelineArrayMap",
"processors": [
{
"type": "array-map-processor",
"is_enabled": true,
"name": "map items",
"source": "items",
"target": "out",
"preserve_source": true,
"processors": [
{
"type": "attribute-remapper",
"sources": [
"$sourceElem.id"
],
"target": "$targetElem.uid",
"preserve_source": true
},
{
"type": "string-builder-processor",
"template": "item-%{$sourceElem.id}",
"target": "$targetElem.label"
}
]
}
],
"tags": []
}
EOF# Set your Datadog site and credentials
export DD_SITE="us3.datadoghq.com"
# Use a Personal Access Token or Service Access Token
export DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>"
curl -X POST "https://api.${DD_SITE}/api/v1/logs/config/pipelines" \
-H "Accept: application/json" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer ${DD_BEARER_TOKEN}" \
-d @- << EOF
{
"filter": {
"query": "source:python"
},
"name": "testPipelineArrayMap",
"processors": [
{
"type": "array-map-processor",
"is_enabled": true,
"name": "map items",
"source": "items",
"target": "out",
"preserve_source": true,
"processors": [
{
"type": "attribute-remapper",
"sources": [
"$sourceElem.id"
],
"target": "$targetElem.uid",
"preserve_source": true
},
{
"type": "string-builder-processor",
"template": "item-%{$sourceElem.id}",
"target": "$targetElem.label"
}
]
}
],
"tags": []
}
EOF# Set your Datadog site and credentials
export DD_SITE="us5.datadoghq.com"
# Use a Personal Access Token or Service Access Token
export DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>"
curl -X POST "https://api.${DD_SITE}/api/v1/logs/config/pipelines" \
-H "Accept: application/json" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer ${DD_BEARER_TOKEN}" \
-d @- << EOF
{
"filter": {
"query": "source:python"
},
"name": "testPipelineArrayMap",
"processors": [
{
"type": "array-map-processor",
"is_enabled": true,
"name": "map items",
"source": "items",
"target": "out",
"preserve_source": true,
"processors": [
{
"type": "attribute-remapper",
"sources": [
"$sourceElem.id"
],
"target": "$targetElem.uid",
"preserve_source": true
},
{
"type": "string-builder-processor",
"template": "item-%{$sourceElem.id}",
"target": "$targetElem.label"
}
]
}
],
"tags": []
}
EOF# Set your Datadog site and credentials
export DD_SITE="datadoghq.eu"
# Use a Personal Access Token or Service Access Token
export DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>"
curl -X POST "https://api.${DD_SITE}/api/v1/logs/config/pipelines" \
-H "Accept: application/json" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer ${DD_BEARER_TOKEN}" \
-d @- << EOF
{
"filter": {
"query": "source:python"
},
"name": "testPipelineArrayMap",
"processors": [
{
"type": "array-map-processor",
"is_enabled": true,
"name": "map items",
"source": "items",
"target": "out",
"preserve_source": true,
"processors": [
{
"type": "attribute-remapper",
"sources": [
"$sourceElem.id"
],
"target": "$targetElem.uid",
"preserve_source": true
},
{
"type": "string-builder-processor",
"template": "item-%{$sourceElem.id}",
"target": "$targetElem.label"
}
]
}
],
"tags": []
}
EOF# Set your Datadog site and credentials
export DD_SITE="ap1.datadoghq.com"
# Use a Personal Access Token or Service Access Token
export DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>"
curl -X POST "https://api.${DD_SITE}/api/v1/logs/config/pipelines" \
-H "Accept: application/json" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer ${DD_BEARER_TOKEN}" \
-d @- << EOF
{
"filter": {
"query": "source:python"
},
"name": "testPipelineArrayMap",
"processors": [
{
"type": "array-map-processor",
"is_enabled": true,
"name": "map items",
"source": "items",
"target": "out",
"preserve_source": true,
"processors": [
{
"type": "attribute-remapper",
"sources": [
"$sourceElem.id"
],
"target": "$targetElem.uid",
"preserve_source": true
},
{
"type": "string-builder-processor",
"template": "item-%{$sourceElem.id}",
"target": "$targetElem.label"
}
]
}
],
"tags": []
}
EOF# Set your Datadog site and credentials
export DD_SITE="ap2.datadoghq.com"
# Use a Personal Access Token or Service Access Token
export DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>"
curl -X POST "https://api.${DD_SITE}/api/v1/logs/config/pipelines" \
-H "Accept: application/json" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer ${DD_BEARER_TOKEN}" \
-d @- << EOF
{
"filter": {
"query": "source:python"
},
"name": "testPipelineArrayMap",
"processors": [
{
"type": "array-map-processor",
"is_enabled": true,
"name": "map items",
"source": "items",
"target": "out",
"preserve_source": true,
"processors": [
{
"type": "attribute-remapper",
"sources": [
"$sourceElem.id"
],
"target": "$targetElem.uid",
"preserve_source": true
},
{
"type": "string-builder-processor",
"template": "item-%{$sourceElem.id}",
"target": "$targetElem.label"
}
]
}
],
"tags": []
}
EOF# Set your Datadog site and credentials
export DD_SITE="uk1.datadoghq.com"
# Use a Personal Access Token or Service Access Token
export DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>"
curl -X POST "https://api.${DD_SITE}/api/v1/logs/config/pipelines" \
-H "Accept: application/json" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer ${DD_BEARER_TOKEN}" \
-d @- << EOF
{
"filter": {
"query": "source:python"
},
"name": "testPipelineArrayMap",
"processors": [
{
"type": "array-map-processor",
"is_enabled": true,
"name": "map items",
"source": "items",
"target": "out",
"preserve_source": true,
"processors": [
{
"type": "attribute-remapper",
"sources": [
"$sourceElem.id"
],
"target": "$targetElem.uid",
"preserve_source": true
},
{
"type": "string-builder-processor",
"template": "item-%{$sourceElem.id}",
"target": "$targetElem.label"
}
]
}
],
"tags": []
}
EOF# Set your Datadog site and credentials
export DD_SITE="ddog-gov.com"
# Use a Personal Access Token or Service Access Token
export DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>"
curl -X POST "https://api.${DD_SITE}/api/v1/logs/config/pipelines" \
-H "Accept: application/json" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer ${DD_BEARER_TOKEN}" \
-d @- << EOF
{
"filter": {
"query": "source:python"
},
"name": "testPipelineArrayMap",
"processors": [
{
"type": "array-map-processor",
"is_enabled": true,
"name": "map items",
"source": "items",
"target": "out",
"preserve_source": true,
"processors": [
{
"type": "attribute-remapper",
"sources": [
"$sourceElem.id"
],
"target": "$targetElem.uid",
"preserve_source": true
},
{
"type": "string-builder-processor",
"template": "item-%{$sourceElem.id}",
"target": "$targetElem.label"
}
]
}
],
"tags": []
}
EOF# Set your Datadog site and credentials
export DD_SITE="us2.ddog-gov.com"
# Use a Personal Access Token or Service Access Token
export DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>"
curl -X POST "https://api.${DD_SITE}/api/v1/logs/config/pipelines" \
-H "Accept: application/json" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer ${DD_BEARER_TOKEN}" \
-d @- << EOF
{
"filter": {
"query": "source:python"
},
"name": "testPipelineArrayMap",
"processors": [
{
"type": "array-map-processor",
"is_enabled": true,
"name": "map items",
"source": "items",
"target": "out",
"preserve_source": true,
"processors": [
{
"type": "attribute-remapper",
"sources": [
"$sourceElem.id"
],
"target": "$targetElem.uid",
"preserve_source": true
},
{
"type": "string-builder-processor",
"template": "item-%{$sourceElem.id}",
"target": "$targetElem.label"
}
]
}
],
"tags": []
}
EOFCreate a pipeline with Array Map Processor using arithmetic sub-processor returns "OK" response
# Set your Datadog site and credentials
export DD_SITE="datadoghq.com"
# Use a Personal Access Token or Service Access Token
export DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>"
curl -X POST "https://api.${DD_SITE}/api/v1/logs/config/pipelines" \
-H "Accept: application/json" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer ${DD_BEARER_TOKEN}" \
-d @- << EOF
{
"filter": {
"query": "source:python"
},
"name": "testPipelineArrayMapArithmetic",
"processors": [
{
"type": "array-map-processor",
"is_enabled": true,
"name": "double counts",
"source": "items",
"target": "out",
"processors": [
{
"type": "arithmetic-processor",
"expression": "$sourceElem.count * 2",
"target": "$targetElem.doubled"
}
]
}
],
"tags": []
}
EOF# Set your Datadog site and credentials
export DD_SITE="us3.datadoghq.com"
# Use a Personal Access Token or Service Access Token
export DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>"
curl -X POST "https://api.${DD_SITE}/api/v1/logs/config/pipelines" \
-H "Accept: application/json" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer ${DD_BEARER_TOKEN}" \
-d @- << EOF
{
"filter": {
"query": "source:python"
},
"name": "testPipelineArrayMapArithmetic",
"processors": [
{
"type": "array-map-processor",
"is_enabled": true,
"name": "double counts",
"source": "items",
"target": "out",
"processors": [
{
"type": "arithmetic-processor",
"expression": "$sourceElem.count * 2",
"target": "$targetElem.doubled"
}
]
}
],
"tags": []
}
EOF# Set your Datadog site and credentials
export DD_SITE="us5.datadoghq.com"
# Use a Personal Access Token or Service Access Token
export DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>"
curl -X POST "https://api.${DD_SITE}/api/v1/logs/config/pipelines" \
-H "Accept: application/json" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer ${DD_BEARER_TOKEN}" \
-d @- << EOF
{
"filter": {
"query": "source:python"
},
"name": "testPipelineArrayMapArithmetic",
"processors": [
{
"type": "array-map-processor",
"is_enabled": true,
"name": "double counts",
"source": "items",
"target": "out",
"processors": [
{
"type": "arithmetic-processor",
"expression": "$sourceElem.count * 2",
"target": "$targetElem.doubled"
}
]
}
],
"tags": []
}
EOF# Set your Datadog site and credentials
export DD_SITE="datadoghq.eu"
# Use a Personal Access Token or Service Access Token
export DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>"
curl -X POST "https://api.${DD_SITE}/api/v1/logs/config/pipelines" \
-H "Accept: application/json" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer ${DD_BEARER_TOKEN}" \
-d @- << EOF
{
"filter": {
"query": "source:python"
},
"name": "testPipelineArrayMapArithmetic",
"processors": [
{
"type": "array-map-processor",
"is_enabled": true,
"name": "double counts",
"source": "items",
"target": "out",
"processors": [
{
"type": "arithmetic-processor",
"expression": "$sourceElem.count * 2",
"target": "$targetElem.doubled"
}
]
}
],
"tags": []
}
EOF# Set your Datadog site and credentials
export DD_SITE="ap1.datadoghq.com"
# Use a Personal Access Token or Service Access Token
export DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>"
curl -X POST "https://api.${DD_SITE}/api/v1/logs/config/pipelines" \
-H "Accept: application/json" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer ${DD_BEARER_TOKEN}" \
-d @- << EOF
{
"filter": {
"query": "source:python"
},
"name": "testPipelineArrayMapArithmetic",
"processors": [
{
"type": "array-map-processor",
"is_enabled": true,
"name": "double counts",
"source": "items",
"target": "out",
"processors": [
{
"type": "arithmetic-processor",
"expression": "$sourceElem.count * 2",
"target": "$targetElem.doubled"
}
]
}
],
"tags": []
}
EOF# Set your Datadog site and credentials
export DD_SITE="ap2.datadoghq.com"
# Use a Personal Access Token or Service Access Token
export DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>"
curl -X POST "https://api.${DD_SITE}/api/v1/logs/config/pipelines" \
-H "Accept: application/json" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer ${DD_BEARER_TOKEN}" \
-d @- << EOF
{
"filter": {
"query": "source:python"
},
"name": "testPipelineArrayMapArithmetic",
"processors": [
{
"type": "array-map-processor",
"is_enabled": true,
"name": "double counts",
"source": "items",
"target": "out",
"processors": [
{
"type": "arithmetic-processor",
"expression": "$sourceElem.count * 2",
"target": "$targetElem.doubled"
}
]
}
],
"tags": []
}
EOF# Set your Datadog site and credentials
export DD_SITE="uk1.datadoghq.com"
# Use a Personal Access Token or Service Access Token
export DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>"
curl -X POST "https://api.${DD_SITE}/api/v1/logs/config/pipelines" \
-H "Accept: application/json" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer ${DD_BEARER_TOKEN}" \
-d @- << EOF
{
"filter": {
"query": "source:python"
},
"name": "testPipelineArrayMapArithmetic",
"processors": [
{
"type": "array-map-processor",
"is_enabled": true,
"name": "double counts",
"source": "items",
"target": "out",
"processors": [
{
"type": "arithmetic-processor",
"expression": "$sourceElem.count * 2",
"target": "$targetElem.doubled"
}
]
}
],
"tags": []
}
EOF# Set your Datadog site and credentials
export DD_SITE="ddog-gov.com"
# Use a Personal Access Token or Service Access Token
export DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>"
curl -X POST "https://api.${DD_SITE}/api/v1/logs/config/pipelines" \
-H "Accept: application/json" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer ${DD_BEARER_TOKEN}" \
-d @- << EOF
{
"filter": {
"query": "source:python"
},
"name": "testPipelineArrayMapArithmetic",
"processors": [
{
"type": "array-map-processor",
"is_enabled": true,
"name": "double counts",
"source": "items",
"target": "out",
"processors": [
{
"type": "arithmetic-processor",
"expression": "$sourceElem.count * 2",
"target": "$targetElem.doubled"
}
]
}
],
"tags": []
}
EOF# Set your Datadog site and credentials
export DD_SITE="us2.ddog-gov.com"
# Use a Personal Access Token or Service Access Token
export DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>"
curl -X POST "https://api.${DD_SITE}/api/v1/logs/config/pipelines" \
-H "Accept: application/json" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer ${DD_BEARER_TOKEN}" \
-d @- << EOF
{
"filter": {
"query": "source:python"
},
"name": "testPipelineArrayMapArithmetic",
"processors": [
{
"type": "array-map-processor",
"is_enabled": true,
"name": "double counts",
"source": "items",
"target": "out",
"processors": [
{
"type": "arithmetic-processor",
"expression": "$sourceElem.count * 2",
"target": "$targetElem.doubled"
}
]
}
],
"tags": []
}
EOFCreate a pipeline with Array Map Processor using category sub-processor returns "OK" response
# Set your Datadog site and credentials
export DD_SITE="datadoghq.com"
# Use a Personal Access Token or Service Access Token
export DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>"
curl -X POST "https://api.${DD_SITE}/api/v1/logs/config/pipelines" \
-H "Accept: application/json" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer ${DD_BEARER_TOKEN}" \
-d @- << EOF
{
"filter": {
"query": "source:python"
},
"name": "testPipelineArrayMapCategory",
"processors": [
{
"type": "array-map-processor",
"is_enabled": true,
"name": "categorize items",
"source": "items",
"target": "out",
"processors": [
{
"type": "category-processor",
"target": "$targetElem.level",
"categories": [
{
"filter": {
"query": "@$sourceElem.status:error"
},
"name": "error"
},
{
"filter": {
"query": "*"
},
"name": "info"
}
]
}
]
}
],
"tags": []
}
EOF# Set your Datadog site and credentials
export DD_SITE="us3.datadoghq.com"
# Use a Personal Access Token or Service Access Token
export DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>"
curl -X POST "https://api.${DD_SITE}/api/v1/logs/config/pipelines" \
-H "Accept: application/json" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer ${DD_BEARER_TOKEN}" \
-d @- << EOF
{
"filter": {
"query": "source:python"
},
"name": "testPipelineArrayMapCategory",
"processors": [
{
"type": "array-map-processor",
"is_enabled": true,
"name": "categorize items",
"source": "items",
"target": "out",
"processors": [
{
"type": "category-processor",
"target": "$targetElem.level",
"categories": [
{
"filter": {
"query": "@$sourceElem.status:error"
},
"name": "error"
},
{
"filter": {
"query": "*"
},
"name": "info"
}
]
}
]
}
],
"tags": []
}
EOF# Set your Datadog site and credentials
export DD_SITE="us5.datadoghq.com"
# Use a Personal Access Token or Service Access Token
export DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>"
curl -X POST "https://api.${DD_SITE}/api/v1/logs/config/pipelines" \
-H "Accept: application/json" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer ${DD_BEARER_TOKEN}" \
-d @- << EOF
{
"filter": {
"query": "source:python"
},
"name": "testPipelineArrayMapCategory",
"processors": [
{
"type": "array-map-processor",
"is_enabled": true,
"name": "categorize items",
"source": "items",
"target": "out",
"processors": [
{
"type": "category-processor",
"target": "$targetElem.level",
"categories": [
{
"filter": {
"query": "@$sourceElem.status:error"
},
"name": "error"
},
{
"filter": {
"query": "*"
},
"name": "info"
}
]
}
]
}
],
"tags": []
}
EOF# Set your Datadog site and credentials
export DD_SITE="datadoghq.eu"
# Use a Personal Access Token or Service Access Token
export DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>"
curl -X POST "https://api.${DD_SITE}/api/v1/logs/config/pipelines" \
-H "Accept: application/json" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer ${DD_BEARER_TOKEN}" \
-d @- << EOF
{
"filter": {
"query": "source:python"
},
"name": "testPipelineArrayMapCategory",
"processors": [
{
"type": "array-map-processor",
"is_enabled": true,
"name": "categorize items",
"source": "items",
"target": "out",
"processors": [
{
"type": "category-processor",
"target": "$targetElem.level",
"categories": [
{
"filter": {
"query": "@$sourceElem.status:error"
},
"name": "error"
},
{
"filter": {
"query": "*"
},
"name": "info"
}
]
}
]
}
],
"tags": []
}
EOF# Set your Datadog site and credentials
export DD_SITE="ap1.datadoghq.com"
# Use a Personal Access Token or Service Access Token
export DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>"
curl -X POST "https://api.${DD_SITE}/api/v1/logs/config/pipelines" \
-H "Accept: application/json" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer ${DD_BEARER_TOKEN}" \
-d @- << EOF
{
"filter": {
"query": "source:python"
},
"name": "testPipelineArrayMapCategory",
"processors": [
{
"type": "array-map-processor",
"is_enabled": true,
"name": "categorize items",
"source": "items",
"target": "out",
"processors": [
{
"type": "category-processor",
"target": "$targetElem.level",
"categories": [
{
"filter": {
"query": "@$sourceElem.status:error"
},
"name": "error"
},
{
"filter": {
"query": "*"
},
"name": "info"
}
]
}
]
}
],
"tags": []
}
EOF# Set your Datadog site and credentials
export DD_SITE="ap2.datadoghq.com"
# Use a Personal Access Token or Service Access Token
export DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>"
curl -X POST "https://api.${DD_SITE}/api/v1/logs/config/pipelines" \
-H "Accept: application/json" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer ${DD_BEARER_TOKEN}" \
-d @- << EOF
{
"filter": {
"query": "source:python"
},
"name": "testPipelineArrayMapCategory",
"processors": [
{
"type": "array-map-processor",
"is_enabled": true,
"name": "categorize items",
"source": "items",
"target": "out",
"processors": [
{
"type": "category-processor",
"target": "$targetElem.level",
"categories": [
{
"filter": {
"query": "@$sourceElem.status:error"
},
"name": "error"
},
{
"filter": {
"query": "*"
},
"name": "info"
}
]
}
]
}
],
"tags": []
}
EOF# Set your Datadog site and credentials
export DD_SITE="uk1.datadoghq.com"
# Use a Personal Access Token or Service Access Token
export DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>"
curl -X POST "https://api.${DD_SITE}/api/v1/logs/config/pipelines" \
-H "Accept: application/json" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer ${DD_BEARER_TOKEN}" \
-d @- << EOF
{
"filter": {
"query": "source:python"
},
"name": "testPipelineArrayMapCategory",
"processors": [
{
"type": "array-map-processor",
"is_enabled": true,
"name": "categorize items",
"source": "items",
"target": "out",
"processors": [
{
"type": "category-processor",
"target": "$targetElem.level",
"categories": [
{
"filter": {
"query": "@$sourceElem.status:error"
},
"name": "error"
},
{
"filter": {
"query": "*"
},
"name": "info"
}
]
}
]
}
],
"tags": []
}
EOF# Set your Datadog site and credentials
export DD_SITE="ddog-gov.com"
# Use a Personal Access Token or Service Access Token
export DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>"
curl -X POST "https://api.${DD_SITE}/api/v1/logs/config/pipelines" \
-H "Accept: application/json" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer ${DD_BEARER_TOKEN}" \
-d @- << EOF
{
"filter": {
"query": "source:python"
},
"name": "testPipelineArrayMapCategory",
"processors": [
{
"type": "array-map-processor",
"is_enabled": true,
"name": "categorize items",
"source": "items",
"target": "out",
"processors": [
{
"type": "category-processor",
"target": "$targetElem.level",
"categories": [
{
"filter": {
"query": "@$sourceElem.status:error"
},
"name": "error"
},
{
"filter": {
"query": "*"
},
"name": "info"
}
]
}
]
}
],
"tags": []
}
EOF# Set your Datadog site and credentials
export DD_SITE="us2.ddog-gov.com"
# Use a Personal Access Token or Service Access Token
export DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>"
curl -X POST "https://api.${DD_SITE}/api/v1/logs/config/pipelines" \
-H "Accept: application/json" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer ${DD_BEARER_TOKEN}" \
-d @- << EOF
{
"filter": {
"query": "source:python"
},
"name": "testPipelineArrayMapCategory",
"processors": [
{
"type": "array-map-processor",
"is_enabled": true,
"name": "categorize items",
"source": "items",
"target": "out",
"processors": [
{
"type": "category-processor",
"target": "$targetElem.level",
"categories": [
{
"filter": {
"query": "@$sourceElem.status:error"
},
"name": "error"
},
{
"filter": {
"query": "*"
},
"name": "info"
}
]
}
]
}
],
"tags": []
}
EOFCreate a pipeline with Array Map Processor returns "OK" response
// Create a pipeline with Array Map Processor returns "OK" response
package main
import (
"context"
"encoding/json"
"fmt"
"os"
"github.com/DataDog/datadog-api-client-go/v2/api/datadog"
"github.com/DataDog/datadog-api-client-go/v2/api/datadogV1"
)
func main() {
body := datadogV1.LogsPipeline{
Filter: &datadogV1.LogsFilter{
Query: datadog.PtrString("source:python"),
},
Name: "testPipelineArrayMap",
Processors: []datadogV1.LogsProcessor{
datadogV1.LogsProcessor{
LogsArrayMapProcessor: &datadogV1.LogsArrayMapProcessor{
Type: datadogV1.LOGSARRAYMAPPROCESSORTYPE_ARRAY_MAP_PROCESSOR,
IsEnabled: datadog.PtrBool(true),
Name: datadog.PtrString("map items"),
Source: "items",
Target: "out",
PreserveSource: datadog.PtrBool(true),
Processors: []datadogV1.LogsArrayMapSubProcessor{
datadogV1.LogsArrayMapSubProcessor{
LogsArrayMapAttributeRemapper: &datadogV1.LogsArrayMapAttributeRemapper{
Type: datadogV1.LOGSATTRIBUTEREMAPPERTYPE_ATTRIBUTE_REMAPPER,
Sources: []string{
"$sourceElem.id",
},
Target: "$targetElem.uid",
PreserveSource: datadog.PtrBool(true),
}},
datadogV1.LogsArrayMapSubProcessor{
LogsArrayMapStringBuilderSubProcessor: &datadogV1.LogsArrayMapStringBuilderSubProcessor{
Type: datadogV1.LOGSSTRINGBUILDERPROCESSORTYPE_STRING_BUILDER_PROCESSOR,
Template: "item-%{$sourceElem.id}",
Target: "$targetElem.label",
}},
},
}},
},
Tags: []string{},
}
ctx := datadog.NewDefaultContext(context.Background())
ctx = context.WithValue(ctx, datadog.ContextAccessToken, os.Getenv("DD_BEARER_TOKEN"))
configuration := datadog.NewConfiguration()
apiClient := datadog.NewAPIClient(configuration)
api := datadogV1.NewLogsPipelinesApi(apiClient)
resp, r, err := api.CreateLogsPipeline(ctx, body)
if err != nil {
fmt.Fprintf(os.Stderr, "Error when calling `LogsPipelinesApi.CreateLogsPipeline`: %v\n", err)
fmt.Fprintf(os.Stderr, "Full HTTP response: %v\n", r)
}
responseContent, _ := json.MarshalIndent(resp, "", " ")
fmt.Fprintf(os.Stdout, "Response from `LogsPipelinesApi.CreateLogsPipeline`:\n%s\n", responseContent)
}
Create a pipeline with Array Map Processor using arithmetic sub-processor returns "OK" response
// Create a pipeline with Array Map Processor using arithmetic sub-processor returns "OK" response
package main
import (
"context"
"encoding/json"
"fmt"
"os"
"github.com/DataDog/datadog-api-client-go/v2/api/datadog"
"github.com/DataDog/datadog-api-client-go/v2/api/datadogV1"
)
func main() {
body := datadogV1.LogsPipeline{
Filter: &datadogV1.LogsFilter{
Query: datadog.PtrString("source:python"),
},
Name: "testPipelineArrayMapArithmetic",
Processors: []datadogV1.LogsProcessor{
datadogV1.LogsProcessor{
LogsArrayMapProcessor: &datadogV1.LogsArrayMapProcessor{
Type: datadogV1.LOGSARRAYMAPPROCESSORTYPE_ARRAY_MAP_PROCESSOR,
IsEnabled: datadog.PtrBool(true),
Name: datadog.PtrString("double counts"),
Source: "items",
Target: "out",
Processors: []datadogV1.LogsArrayMapSubProcessor{
datadogV1.LogsArrayMapSubProcessor{
LogsArrayMapArithmeticSubProcessor: &datadogV1.LogsArrayMapArithmeticSubProcessor{
Type: datadogV1.LOGSARITHMETICPROCESSORTYPE_ARITHMETIC_PROCESSOR,
Expression: "$sourceElem.count * 2",
Target: "$targetElem.doubled",
}},
},
}},
},
Tags: []string{},
}
ctx := datadog.NewDefaultContext(context.Background())
ctx = context.WithValue(ctx, datadog.ContextAccessToken, os.Getenv("DD_BEARER_TOKEN"))
configuration := datadog.NewConfiguration()
apiClient := datadog.NewAPIClient(configuration)
api := datadogV1.NewLogsPipelinesApi(apiClient)
resp, r, err := api.CreateLogsPipeline(ctx, body)
if err != nil {
fmt.Fprintf(os.Stderr, "Error when calling `LogsPipelinesApi.CreateLogsPipeline`: %v\n", err)
fmt.Fprintf(os.Stderr, "Full HTTP response: %v\n", r)
}
responseContent, _ := json.MarshalIndent(resp, "", " ")
fmt.Fprintf(os.Stdout, "Response from `LogsPipelinesApi.CreateLogsPipeline`:\n%s\n", responseContent)
}
Create a pipeline with Array Map Processor using category sub-processor returns "OK" response
// Create a pipeline with Array Map Processor using category sub-processor returns "OK" response
package main
import (
"context"
"encoding/json"
"fmt"
"os"
"github.com/DataDog/datadog-api-client-go/v2/api/datadog"
"github.com/DataDog/datadog-api-client-go/v2/api/datadogV1"
)
func main() {
body := datadogV1.LogsPipeline{
Filter: &datadogV1.LogsFilter{
Query: datadog.PtrString("source:python"),
},
Name: "testPipelineArrayMapCategory",
Processors: []datadogV1.LogsProcessor{
datadogV1.LogsProcessor{
LogsArrayMapProcessor: &datadogV1.LogsArrayMapProcessor{
Type: datadogV1.LOGSARRAYMAPPROCESSORTYPE_ARRAY_MAP_PROCESSOR,
IsEnabled: datadog.PtrBool(true),
Name: datadog.PtrString("categorize items"),
Source: "items",
Target: "out",
Processors: []datadogV1.LogsArrayMapSubProcessor{
datadogV1.LogsArrayMapSubProcessor{
LogsArrayMapCategorySubProcessor: &datadogV1.LogsArrayMapCategorySubProcessor{
Type: datadogV1.LOGSCATEGORYPROCESSORTYPE_CATEGORY_PROCESSOR,
Target: "$targetElem.level",
Categories: []datadogV1.LogsCategoryProcessorCategory{
{
Filter: &datadogV1.LogsFilter{
Query: datadog.PtrString("@$sourceElem.status:error"),
},
Name: datadog.PtrString("error"),
},
{
Filter: &datadogV1.LogsFilter{
Query: datadog.PtrString("*"),
},
Name: datadog.PtrString("info"),
},
},
}},
},
}},
},
Tags: []string{},
}
ctx := datadog.NewDefaultContext(context.Background())
ctx = context.WithValue(ctx, datadog.ContextAccessToken, os.Getenv("DD_BEARER_TOKEN"))
configuration := datadog.NewConfiguration()
apiClient := datadog.NewAPIClient(configuration)
api := datadogV1.NewLogsPipelinesApi(apiClient)
resp, r, err := api.CreateLogsPipeline(ctx, body)
if err != nil {
fmt.Fprintf(os.Stderr, "Error when calling `LogsPipelinesApi.CreateLogsPipeline`: %v\n", err)
fmt.Fprintf(os.Stderr, "Full HTTP response: %v\n", r)
}
responseContent, _ := json.MarshalIndent(resp, "", " ")
fmt.Fprintf(os.Stdout, "Response from `LogsPipelinesApi.CreateLogsPipeline`:\n%s\n", responseContent)
}
Instructions
First install the library and its dependencies and then save the example to main.go and run following commands:
DD_SITE="datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" go run "main.go"DD_SITE="us3.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" go run "main.go"DD_SITE="us5.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" go run "main.go"DD_SITE="datadoghq.eu" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" go run "main.go"DD_SITE="ap1.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" go run "main.go"DD_SITE="ap2.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" go run "main.go"DD_SITE="uk1.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" go run "main.go"DD_SITE="ddog-gov.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" go run "main.go"DD_SITE="us2.ddog-gov.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" go run "main.go"Create a pipeline with Array Map Processor returns "OK" response
// Create a pipeline with Array Map Processor returns "OK" response
import com.datadog.api.client.ApiClient;
import com.datadog.api.client.ApiException;
import com.datadog.api.client.v1.api.LogsPipelinesApi;
import com.datadog.api.client.v1.model.LogsArrayMapAttributeRemapper;
import com.datadog.api.client.v1.model.LogsArrayMapProcessor;
import com.datadog.api.client.v1.model.LogsArrayMapProcessorType;
import com.datadog.api.client.v1.model.LogsArrayMapStringBuilderSubProcessor;
import com.datadog.api.client.v1.model.LogsArrayMapSubProcessor;
import com.datadog.api.client.v1.model.LogsAttributeRemapperType;
import com.datadog.api.client.v1.model.LogsFilter;
import com.datadog.api.client.v1.model.LogsPipeline;
import com.datadog.api.client.v1.model.LogsProcessor;
import com.datadog.api.client.v1.model.LogsStringBuilderProcessorType;
import java.util.Arrays;
import java.util.Collections;
public class Example {
public static void main(String[] args) {
ApiClient defaultClient = ApiClient.getDefaultApiClient();
defaultClient.setAccessToken(System.getenv("DD_BEARER_TOKEN"));
LogsPipelinesApi apiInstance = new LogsPipelinesApi(defaultClient);
LogsPipeline body =
new LogsPipeline()
.filter(new LogsFilter().query("source:python"))
.name("testPipelineArrayMap")
.processors(
Collections.singletonList(
new LogsProcessor(
new LogsArrayMapProcessor()
.type(LogsArrayMapProcessorType.ARRAY_MAP_PROCESSOR)
.isEnabled(true)
.name("map items")
.source("items")
.target("out")
.preserveSource(true)
.processors(
Arrays.asList(
new LogsArrayMapSubProcessor(
new LogsArrayMapAttributeRemapper()
.type(LogsAttributeRemapperType.ATTRIBUTE_REMAPPER)
.sources(Collections.singletonList("$sourceElem.id"))
.target("$targetElem.uid")
.preserveSource(true)),
new LogsArrayMapSubProcessor(
new LogsArrayMapStringBuilderSubProcessor()
.type(
LogsStringBuilderProcessorType
.STRING_BUILDER_PROCESSOR)
.template("item-%{$sourceElem.id}")
.target("$targetElem.label")))))));
try {
LogsPipeline result = apiInstance.createLogsPipeline(body);
System.out.println(result);
} catch (ApiException e) {
System.err.println("Exception when calling LogsPipelinesApi#createLogsPipeline");
System.err.println("Status code: " + e.getCode());
System.err.println("Reason: " + e.getResponseBody());
System.err.println("Response headers: " + e.getResponseHeaders());
e.printStackTrace();
}
}
}
Create a pipeline with Array Map Processor using arithmetic sub-processor returns "OK" response
// Create a pipeline with Array Map Processor using arithmetic sub-processor returns "OK" response
import com.datadog.api.client.ApiClient;
import com.datadog.api.client.ApiException;
import com.datadog.api.client.v1.api.LogsPipelinesApi;
import com.datadog.api.client.v1.model.LogsArithmeticProcessorType;
import com.datadog.api.client.v1.model.LogsArrayMapArithmeticSubProcessor;
import com.datadog.api.client.v1.model.LogsArrayMapProcessor;
import com.datadog.api.client.v1.model.LogsArrayMapProcessorType;
import com.datadog.api.client.v1.model.LogsArrayMapSubProcessor;
import com.datadog.api.client.v1.model.LogsFilter;
import com.datadog.api.client.v1.model.LogsPipeline;
import com.datadog.api.client.v1.model.LogsProcessor;
import java.util.Collections;
public class Example {
public static void main(String[] args) {
ApiClient defaultClient = ApiClient.getDefaultApiClient();
defaultClient.setAccessToken(System.getenv("DD_BEARER_TOKEN"));
LogsPipelinesApi apiInstance = new LogsPipelinesApi(defaultClient);
LogsPipeline body =
new LogsPipeline()
.filter(new LogsFilter().query("source:python"))
.name("testPipelineArrayMapArithmetic")
.processors(
Collections.singletonList(
new LogsProcessor(
new LogsArrayMapProcessor()
.type(LogsArrayMapProcessorType.ARRAY_MAP_PROCESSOR)
.isEnabled(true)
.name("double counts")
.source("items")
.target("out")
.processors(
Collections.singletonList(
new LogsArrayMapSubProcessor(
new LogsArrayMapArithmeticSubProcessor()
.type(LogsArithmeticProcessorType.ARITHMETIC_PROCESSOR)
.expression("$sourceElem.count * 2")
.target("$targetElem.doubled")))))));
try {
LogsPipeline result = apiInstance.createLogsPipeline(body);
System.out.println(result);
} catch (ApiException e) {
System.err.println("Exception when calling LogsPipelinesApi#createLogsPipeline");
System.err.println("Status code: " + e.getCode());
System.err.println("Reason: " + e.getResponseBody());
System.err.println("Response headers: " + e.getResponseHeaders());
e.printStackTrace();
}
}
}
Create a pipeline with Array Map Processor using category sub-processor returns "OK" response
// Create a pipeline with Array Map Processor using category sub-processor returns "OK" response
import com.datadog.api.client.ApiClient;
import com.datadog.api.client.ApiException;
import com.datadog.api.client.v1.api.LogsPipelinesApi;
import com.datadog.api.client.v1.model.LogsArrayMapCategorySubProcessor;
import com.datadog.api.client.v1.model.LogsArrayMapProcessor;
import com.datadog.api.client.v1.model.LogsArrayMapProcessorType;
import com.datadog.api.client.v1.model.LogsArrayMapSubProcessor;
import com.datadog.api.client.v1.model.LogsCategoryProcessorCategory;
import com.datadog.api.client.v1.model.LogsCategoryProcessorType;
import com.datadog.api.client.v1.model.LogsFilter;
import com.datadog.api.client.v1.model.LogsPipeline;
import com.datadog.api.client.v1.model.LogsProcessor;
import java.util.Arrays;
import java.util.Collections;
public class Example {
public static void main(String[] args) {
ApiClient defaultClient = ApiClient.getDefaultApiClient();
defaultClient.setAccessToken(System.getenv("DD_BEARER_TOKEN"));
LogsPipelinesApi apiInstance = new LogsPipelinesApi(defaultClient);
LogsPipeline body =
new LogsPipeline()
.filter(new LogsFilter().query("source:python"))
.name("testPipelineArrayMapCategory")
.processors(
Collections.singletonList(
new LogsProcessor(
new LogsArrayMapProcessor()
.type(LogsArrayMapProcessorType.ARRAY_MAP_PROCESSOR)
.isEnabled(true)
.name("categorize items")
.source("items")
.target("out")
.processors(
Collections.singletonList(
new LogsArrayMapSubProcessor(
new LogsArrayMapCategorySubProcessor()
.type(LogsCategoryProcessorType.CATEGORY_PROCESSOR)
.target("$targetElem.level")
.categories(
Arrays.asList(
new LogsCategoryProcessorCategory()
.filter(
new LogsFilter()
.query("@$sourceElem.status:error"))
.name("error"),
new LogsCategoryProcessorCategory()
.filter(new LogsFilter().query("*"))
.name("info")))))))));
try {
LogsPipeline result = apiInstance.createLogsPipeline(body);
System.out.println(result);
} catch (ApiException e) {
System.err.println("Exception when calling LogsPipelinesApi#createLogsPipeline");
System.err.println("Status code: " + e.getCode());
System.err.println("Reason: " + e.getResponseBody());
System.err.println("Response headers: " + e.getResponseHeaders());
e.printStackTrace();
}
}
}
Instructions
First install the library and its dependencies and then save the example to Example.java and run following commands:
DD_SITE="datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" java "Example.java"DD_SITE="us3.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" java "Example.java"DD_SITE="us5.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" java "Example.java"DD_SITE="datadoghq.eu" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" java "Example.java"DD_SITE="ap1.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" java "Example.java"DD_SITE="ap2.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" java "Example.java"DD_SITE="uk1.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" java "Example.java"DD_SITE="ddog-gov.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" java "Example.java"DD_SITE="us2.ddog-gov.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" java "Example.java"Create a pipeline with Array Map Processor returns "OK" response
"""
Create a pipeline with Array Map Processor returns "OK" response
"""
from os import environ
from datadog_api_client import ApiClient, Configuration
from datadog_api_client.v1.api.logs_pipelines_api import LogsPipelinesApi
from datadog_api_client.v1.model.logs_array_map_attribute_remapper import LogsArrayMapAttributeRemapper
from datadog_api_client.v1.model.logs_array_map_processor import LogsArrayMapProcessor
from datadog_api_client.v1.model.logs_array_map_processor_type import LogsArrayMapProcessorType
from datadog_api_client.v1.model.logs_array_map_string_builder_sub_processor import (
LogsArrayMapStringBuilderSubProcessor,
)
from datadog_api_client.v1.model.logs_attribute_remapper_type import LogsAttributeRemapperType
from datadog_api_client.v1.model.logs_filter import LogsFilter
from datadog_api_client.v1.model.logs_pipeline import LogsPipeline
from datadog_api_client.v1.model.logs_string_builder_processor_type import LogsStringBuilderProcessorType
body = LogsPipeline(
filter=LogsFilter(
query="source:python",
),
name="testPipelineArrayMap",
processors=[
LogsArrayMapProcessor(
type=LogsArrayMapProcessorType.ARRAY_MAP_PROCESSOR,
is_enabled=True,
name="map items",
source="items",
target="out",
preserve_source=True,
processors=[
LogsArrayMapAttributeRemapper(
type=LogsAttributeRemapperType.ATTRIBUTE_REMAPPER,
sources=[
"$sourceElem.id",
],
target="$targetElem.uid",
preserve_source=True,
),
LogsArrayMapStringBuilderSubProcessor(
type=LogsStringBuilderProcessorType.STRING_BUILDER_PROCESSOR,
template="item-%{$sourceElem.id}",
target="$targetElem.label",
),
],
),
],
tags=[],
)
configuration = Configuration()
configuration.access_token = environ["DD_BEARER_TOKEN"]
with ApiClient(configuration) as api_client:
api_instance = LogsPipelinesApi(api_client)
response = api_instance.create_logs_pipeline(body=body)
print(response)
Create a pipeline with Array Map Processor using arithmetic sub-processor returns "OK" response
"""
Create a pipeline with Array Map Processor using arithmetic sub-processor returns "OK" response
"""
from os import environ
from datadog_api_client import ApiClient, Configuration
from datadog_api_client.v1.api.logs_pipelines_api import LogsPipelinesApi
from datadog_api_client.v1.model.logs_arithmetic_processor_type import LogsArithmeticProcessorType
from datadog_api_client.v1.model.logs_array_map_arithmetic_sub_processor import LogsArrayMapArithmeticSubProcessor
from datadog_api_client.v1.model.logs_array_map_processor import LogsArrayMapProcessor
from datadog_api_client.v1.model.logs_array_map_processor_type import LogsArrayMapProcessorType
from datadog_api_client.v1.model.logs_filter import LogsFilter
from datadog_api_client.v1.model.logs_pipeline import LogsPipeline
body = LogsPipeline(
filter=LogsFilter(
query="source:python",
),
name="testPipelineArrayMapArithmetic",
processors=[
LogsArrayMapProcessor(
type=LogsArrayMapProcessorType.ARRAY_MAP_PROCESSOR,
is_enabled=True,
name="double counts",
source="items",
target="out",
processors=[
LogsArrayMapArithmeticSubProcessor(
type=LogsArithmeticProcessorType.ARITHMETIC_PROCESSOR,
expression="$sourceElem.count * 2",
target="$targetElem.doubled",
),
],
),
],
tags=[],
)
configuration = Configuration()
configuration.access_token = environ["DD_BEARER_TOKEN"]
with ApiClient(configuration) as api_client:
api_instance = LogsPipelinesApi(api_client)
response = api_instance.create_logs_pipeline(body=body)
print(response)
Create a pipeline with Array Map Processor using category sub-processor returns "OK" response
"""
Create a pipeline with Array Map Processor using category sub-processor returns "OK" response
"""
from os import environ
from datadog_api_client import ApiClient, Configuration
from datadog_api_client.v1.api.logs_pipelines_api import LogsPipelinesApi
from datadog_api_client.v1.model.logs_array_map_category_sub_processor import LogsArrayMapCategorySubProcessor
from datadog_api_client.v1.model.logs_array_map_processor import LogsArrayMapProcessor
from datadog_api_client.v1.model.logs_array_map_processor_type import LogsArrayMapProcessorType
from datadog_api_client.v1.model.logs_category_processor_category import LogsCategoryProcessorCategory
from datadog_api_client.v1.model.logs_category_processor_type import LogsCategoryProcessorType
from datadog_api_client.v1.model.logs_filter import LogsFilter
from datadog_api_client.v1.model.logs_pipeline import LogsPipeline
body = LogsPipeline(
filter=LogsFilter(
query="source:python",
),
name="testPipelineArrayMapCategory",
processors=[
LogsArrayMapProcessor(
type=LogsArrayMapProcessorType.ARRAY_MAP_PROCESSOR,
is_enabled=True,
name="categorize items",
source="items",
target="out",
processors=[
LogsArrayMapCategorySubProcessor(
type=LogsCategoryProcessorType.CATEGORY_PROCESSOR,
target="$targetElem.level",
categories=[
LogsCategoryProcessorCategory(
filter=LogsFilter(
query="@$sourceElem.status:error",
),
name="error",
),
LogsCategoryProcessorCategory(
filter=LogsFilter(
query="*",
),
name="info",
),
],
),
],
),
],
tags=[],
)
configuration = Configuration()
configuration.access_token = environ["DD_BEARER_TOKEN"]
with ApiClient(configuration) as api_client:
api_instance = LogsPipelinesApi(api_client)
response = api_instance.create_logs_pipeline(body=body)
print(response)
Instructions
First install the library and its dependencies and then save the example to example.py and run following commands:
DD_SITE="datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" python3 "example.py"DD_SITE="us3.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" python3 "example.py"DD_SITE="us5.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" python3 "example.py"DD_SITE="datadoghq.eu" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" python3 "example.py"DD_SITE="ap1.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" python3 "example.py"DD_SITE="ap2.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" python3 "example.py"DD_SITE="uk1.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" python3 "example.py"DD_SITE="ddog-gov.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" python3 "example.py"DD_SITE="us2.ddog-gov.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" python3 "example.py"Create a pipeline with Array Map Processor returns "OK" response
# Create a pipeline with Array Map Processor returns "OK" response
require "datadog_api_client"
DatadogAPIClient.configure do |config|
config.access_token = ENV["DD_BEARER_TOKEN"]
end
api_instance = DatadogAPIClient::V1::LogsPipelinesAPI.new
body = DatadogAPIClient::V1::LogsPipeline.new({
filter: DatadogAPIClient::V1::LogsFilter.new({
query: "source:python",
}),
name: "testPipelineArrayMap",
processors: [
DatadogAPIClient::V1::LogsArrayMapProcessor.new({
type: DatadogAPIClient::V1::LogsArrayMapProcessorType::ARRAY_MAP_PROCESSOR,
is_enabled: true,
name: "map items",
source: "items",
target: "out",
preserve_source: true,
processors: [
DatadogAPIClient::V1::LogsArrayMapAttributeRemapper.new({
type: DatadogAPIClient::V1::LogsAttributeRemapperType::ATTRIBUTE_REMAPPER,
sources: [
"$sourceElem.id",
],
target: "$targetElem.uid",
preserve_source: true,
}),
DatadogAPIClient::V1::LogsArrayMapStringBuilderSubProcessor.new({
type: DatadogAPIClient::V1::LogsStringBuilderProcessorType::STRING_BUILDER_PROCESSOR,
template: "item-%{$sourceElem.id}",
target: "$targetElem.label",
}),
],
}),
],
tags: [],
})
p api_instance.create_logs_pipeline(body)
Create a pipeline with Array Map Processor using arithmetic sub-processor returns "OK" response
# Create a pipeline with Array Map Processor using arithmetic sub-processor returns "OK" response
require "datadog_api_client"
DatadogAPIClient.configure do |config|
config.access_token = ENV["DD_BEARER_TOKEN"]
end
api_instance = DatadogAPIClient::V1::LogsPipelinesAPI.new
body = DatadogAPIClient::V1::LogsPipeline.new({
filter: DatadogAPIClient::V1::LogsFilter.new({
query: "source:python",
}),
name: "testPipelineArrayMapArithmetic",
processors: [
DatadogAPIClient::V1::LogsArrayMapProcessor.new({
type: DatadogAPIClient::V1::LogsArrayMapProcessorType::ARRAY_MAP_PROCESSOR,
is_enabled: true,
name: "double counts",
source: "items",
target: "out",
processors: [
DatadogAPIClient::V1::LogsArrayMapArithmeticSubProcessor.new({
type: DatadogAPIClient::V1::LogsArithmeticProcessorType::ARITHMETIC_PROCESSOR,
expression: "$sourceElem.count * 2",
target: "$targetElem.doubled",
}),
],
}),
],
tags: [],
})
p api_instance.create_logs_pipeline(body)
Create a pipeline with Array Map Processor using category sub-processor returns "OK" response
# Create a pipeline with Array Map Processor using category sub-processor returns "OK" response
require "datadog_api_client"
DatadogAPIClient.configure do |config|
config.access_token = ENV["DD_BEARER_TOKEN"]
end
api_instance = DatadogAPIClient::V1::LogsPipelinesAPI.new
body = DatadogAPIClient::V1::LogsPipeline.new({
filter: DatadogAPIClient::V1::LogsFilter.new({
query: "source:python",
}),
name: "testPipelineArrayMapCategory",
processors: [
DatadogAPIClient::V1::LogsArrayMapProcessor.new({
type: DatadogAPIClient::V1::LogsArrayMapProcessorType::ARRAY_MAP_PROCESSOR,
is_enabled: true,
name: "categorize items",
source: "items",
target: "out",
processors: [
DatadogAPIClient::V1::LogsArrayMapCategorySubProcessor.new({
type: DatadogAPIClient::V1::LogsCategoryProcessorType::CATEGORY_PROCESSOR,
target: "$targetElem.level",
categories: [
DatadogAPIClient::V1::LogsCategoryProcessorCategory.new({
filter: DatadogAPIClient::V1::LogsFilter.new({
query: "@$sourceElem.status:error",
}),
name: "error",
}),
DatadogAPIClient::V1::LogsCategoryProcessorCategory.new({
filter: DatadogAPIClient::V1::LogsFilter.new({
query: "*",
}),
name: "info",
}),
],
}),
],
}),
],
tags: [],
})
p api_instance.create_logs_pipeline(body)
Instructions
First install the library and its dependencies and then save the example to example.rb and run following commands:
DD_SITE="datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" rb "example.rb"DD_SITE="us3.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" rb "example.rb"DD_SITE="us5.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" rb "example.rb"DD_SITE="datadoghq.eu" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" rb "example.rb"DD_SITE="ap1.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" rb "example.rb"DD_SITE="ap2.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" rb "example.rb"DD_SITE="uk1.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" rb "example.rb"DD_SITE="ddog-gov.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" rb "example.rb"DD_SITE="us2.ddog-gov.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" rb "example.rb"Create a pipeline with Array Map Processor returns "OK" response
// Create a pipeline with Array Map Processor returns "OK" response
use datadog_api_client::datadog;
use datadog_api_client::datadogV1::api_logs_pipelines::LogsPipelinesAPI;
use datadog_api_client::datadogV1::model::LogsArrayMapAttributeRemapper;
use datadog_api_client::datadogV1::model::LogsArrayMapProcessor;
use datadog_api_client::datadogV1::model::LogsArrayMapProcessorType;
use datadog_api_client::datadogV1::model::LogsArrayMapStringBuilderSubProcessor;
use datadog_api_client::datadogV1::model::LogsArrayMapSubProcessor;
use datadog_api_client::datadogV1::model::LogsAttributeRemapperType;
use datadog_api_client::datadogV1::model::LogsFilter;
use datadog_api_client::datadogV1::model::LogsPipeline;
use datadog_api_client::datadogV1::model::LogsProcessor;
use datadog_api_client::datadogV1::model::LogsStringBuilderProcessorType;
#[tokio::main]
async fn main() {
let body = LogsPipeline::new("testPipelineArrayMap".to_string())
.filter(LogsFilter::new().query("source:python".to_string()))
.processors(vec![LogsProcessor::LogsArrayMapProcessor(Box::new(
LogsArrayMapProcessor::new(
vec![
LogsArrayMapSubProcessor::LogsArrayMapAttributeRemapper(Box::new(
LogsArrayMapAttributeRemapper::new(
vec!["$sourceElem.id".to_string()],
"$targetElem.uid".to_string(),
LogsAttributeRemapperType::ATTRIBUTE_REMAPPER,
)
.preserve_source(true),
)),
LogsArrayMapSubProcessor::LogsArrayMapStringBuilderSubProcessor(Box::new(
LogsArrayMapStringBuilderSubProcessor::new(
"$targetElem.label".to_string(),
"item-%{$sourceElem.id}".to_string(),
LogsStringBuilderProcessorType::STRING_BUILDER_PROCESSOR,
),
)),
],
"items".to_string(),
"out".to_string(),
LogsArrayMapProcessorType::ARRAY_MAP_PROCESSOR,
)
.is_enabled(true)
.name("map items".to_string())
.preserve_source(true),
))])
.tags(vec![]);
let configuration = datadog::Configuration::new();
let api = LogsPipelinesAPI::with_config(configuration);
let resp = api.create_logs_pipeline(body).await;
if let Ok(value) = resp {
println!("{:#?}", value);
} else {
println!("{:#?}", resp.unwrap_err());
}
}
Create a pipeline with Array Map Processor using arithmetic sub-processor returns "OK" response
// Create a pipeline with Array Map Processor using arithmetic sub-processor
// returns "OK" response
use datadog_api_client::datadog;
use datadog_api_client::datadogV1::api_logs_pipelines::LogsPipelinesAPI;
use datadog_api_client::datadogV1::model::LogsArithmeticProcessorType;
use datadog_api_client::datadogV1::model::LogsArrayMapArithmeticSubProcessor;
use datadog_api_client::datadogV1::model::LogsArrayMapProcessor;
use datadog_api_client::datadogV1::model::LogsArrayMapProcessorType;
use datadog_api_client::datadogV1::model::LogsArrayMapSubProcessor;
use datadog_api_client::datadogV1::model::LogsFilter;
use datadog_api_client::datadogV1::model::LogsPipeline;
use datadog_api_client::datadogV1::model::LogsProcessor;
#[tokio::main]
async fn main() {
let body = LogsPipeline::new("testPipelineArrayMapArithmetic".to_string())
.filter(LogsFilter::new().query("source:python".to_string()))
.processors(vec![LogsProcessor::LogsArrayMapProcessor(Box::new(
LogsArrayMapProcessor::new(
vec![
LogsArrayMapSubProcessor::LogsArrayMapArithmeticSubProcessor(Box::new(
LogsArrayMapArithmeticSubProcessor::new(
"$sourceElem.count * 2".to_string(),
"$targetElem.doubled".to_string(),
LogsArithmeticProcessorType::ARITHMETIC_PROCESSOR,
),
)),
],
"items".to_string(),
"out".to_string(),
LogsArrayMapProcessorType::ARRAY_MAP_PROCESSOR,
)
.is_enabled(true)
.name("double counts".to_string()),
))])
.tags(vec![]);
let configuration = datadog::Configuration::new();
let api = LogsPipelinesAPI::with_config(configuration);
let resp = api.create_logs_pipeline(body).await;
if let Ok(value) = resp {
println!("{:#?}", value);
} else {
println!("{:#?}", resp.unwrap_err());
}
}
Create a pipeline with Array Map Processor using category sub-processor returns "OK" response
// Create a pipeline with Array Map Processor using category sub-processor returns
// "OK" response
use datadog_api_client::datadog;
use datadog_api_client::datadogV1::api_logs_pipelines::LogsPipelinesAPI;
use datadog_api_client::datadogV1::model::LogsArrayMapCategorySubProcessor;
use datadog_api_client::datadogV1::model::LogsArrayMapProcessor;
use datadog_api_client::datadogV1::model::LogsArrayMapProcessorType;
use datadog_api_client::datadogV1::model::LogsArrayMapSubProcessor;
use datadog_api_client::datadogV1::model::LogsCategoryProcessorCategory;
use datadog_api_client::datadogV1::model::LogsCategoryProcessorType;
use datadog_api_client::datadogV1::model::LogsFilter;
use datadog_api_client::datadogV1::model::LogsPipeline;
use datadog_api_client::datadogV1::model::LogsProcessor;
#[tokio::main]
async fn main() {
let body = LogsPipeline::new("testPipelineArrayMapCategory".to_string())
.filter(LogsFilter::new().query("source:python".to_string()))
.processors(vec![LogsProcessor::LogsArrayMapProcessor(Box::new(
LogsArrayMapProcessor::new(
vec![LogsArrayMapSubProcessor::LogsArrayMapCategorySubProcessor(
Box::new(LogsArrayMapCategorySubProcessor::new(
vec![
LogsCategoryProcessorCategory::new()
.filter(
LogsFilter::new()
.query("@$sourceElem.status:error".to_string()),
)
.name("error".to_string()),
LogsCategoryProcessorCategory::new()
.filter(LogsFilter::new().query("*".to_string()))
.name("info".to_string()),
],
"$targetElem.level".to_string(),
LogsCategoryProcessorType::CATEGORY_PROCESSOR,
)),
)],
"items".to_string(),
"out".to_string(),
LogsArrayMapProcessorType::ARRAY_MAP_PROCESSOR,
)
.is_enabled(true)
.name("categorize items".to_string()),
))])
.tags(vec![]);
let configuration = datadog::Configuration::new();
let api = LogsPipelinesAPI::with_config(configuration);
let resp = api.create_logs_pipeline(body).await;
if let Ok(value) = resp {
println!("{:#?}", value);
} else {
println!("{:#?}", resp.unwrap_err());
}
}
Instructions
First install the library and its dependencies and then save the example to src/main.rs and run following commands:
DD_SITE="datadoghq.com" DD_API_KEY="<API-KEY>" DD_APP_KEY="<APP-KEY>" cargo runDD_SITE="us3.datadoghq.com" DD_API_KEY="<API-KEY>" DD_APP_KEY="<APP-KEY>" cargo runDD_SITE="us5.datadoghq.com" DD_API_KEY="<API-KEY>" DD_APP_KEY="<APP-KEY>" cargo runDD_SITE="datadoghq.eu" DD_API_KEY="<API-KEY>" DD_APP_KEY="<APP-KEY>" cargo runDD_SITE="ap1.datadoghq.com" DD_API_KEY="<API-KEY>" DD_APP_KEY="<APP-KEY>" cargo runDD_SITE="ap2.datadoghq.com" DD_API_KEY="<API-KEY>" DD_APP_KEY="<APP-KEY>" cargo runDD_SITE="uk1.datadoghq.com" DD_API_KEY="<API-KEY>" DD_APP_KEY="<APP-KEY>" cargo runDD_SITE="ddog-gov.com" DD_API_KEY="<API-KEY>" DD_APP_KEY="<APP-KEY>" cargo runDD_SITE="us2.ddog-gov.com" DD_API_KEY="<API-KEY>" DD_APP_KEY="<APP-KEY>" cargo runCreate a pipeline with Array Map Processor returns "OK" response
/**
* Create a pipeline with Array Map Processor returns "OK" response
*/
import { client, v1 } from "@datadog/datadog-api-client";
const configuration = client.createConfiguration({
authMethods: {
AuthZ: {
accessToken: process.env.DD_BEARER_TOKEN as string,
},
},
});
const apiInstance = new v1.LogsPipelinesApi(configuration);
const params: v1.LogsPipelinesApiCreateLogsPipelineRequest = {
body: {
filter: {
query: "source:python",
},
name: "testPipelineArrayMap",
processors: [
{
type: "array-map-processor",
isEnabled: true,
name: "map items",
source: "items",
target: "out",
preserveSource: true,
processors: [
{
type: "attribute-remapper",
sources: ["$sourceElem.id"],
target: "$targetElem.uid",
preserveSource: true,
},
{
type: "string-builder-processor",
template: "item-%{$sourceElem.id}",
target: "$targetElem.label",
},
],
},
],
tags: [],
},
};
apiInstance
.createLogsPipeline(params)
.then((data: v1.LogsPipeline) => {
console.log(
"API called successfully. Returned data: " + JSON.stringify(data)
);
})
.catch((error: any) => console.error(error));
Create a pipeline with Array Map Processor using arithmetic sub-processor returns "OK" response
/**
* Create a pipeline with Array Map Processor using arithmetic sub-processor returns "OK" response
*/
import { client, v1 } from "@datadog/datadog-api-client";
const configuration = client.createConfiguration({
authMethods: {
AuthZ: {
accessToken: process.env.DD_BEARER_TOKEN as string,
},
},
});
const apiInstance = new v1.LogsPipelinesApi(configuration);
const params: v1.LogsPipelinesApiCreateLogsPipelineRequest = {
body: {
filter: {
query: "source:python",
},
name: "testPipelineArrayMapArithmetic",
processors: [
{
type: "array-map-processor",
isEnabled: true,
name: "double counts",
source: "items",
target: "out",
processors: [
{
type: "arithmetic-processor",
expression: "$sourceElem.count * 2",
target: "$targetElem.doubled",
},
],
},
],
tags: [],
},
};
apiInstance
.createLogsPipeline(params)
.then((data: v1.LogsPipeline) => {
console.log(
"API called successfully. Returned data: " + JSON.stringify(data)
);
})
.catch((error: any) => console.error(error));
Create a pipeline with Array Map Processor using category sub-processor returns "OK" response
/**
* Create a pipeline with Array Map Processor using category sub-processor returns "OK" response
*/
import { client, v1 } from "@datadog/datadog-api-client";
const configuration = client.createConfiguration({
authMethods: {
AuthZ: {
accessToken: process.env.DD_BEARER_TOKEN as string,
},
},
});
const apiInstance = new v1.LogsPipelinesApi(configuration);
const params: v1.LogsPipelinesApiCreateLogsPipelineRequest = {
body: {
filter: {
query: "source:python",
},
name: "testPipelineArrayMapCategory",
processors: [
{
type: "array-map-processor",
isEnabled: true,
name: "categorize items",
source: "items",
target: "out",
processors: [
{
type: "category-processor",
target: "$targetElem.level",
categories: [
{
filter: {
query: "@$sourceElem.status:error",
},
name: "error",
},
{
filter: {
query: "*",
},
name: "info",
},
],
},
],
},
],
tags: [],
},
};
apiInstance
.createLogsPipeline(params)
.then((data: v1.LogsPipeline) => {
console.log(
"API called successfully. Returned data: " + JSON.stringify(data)
);
})
.catch((error: any) => console.error(error));
Instructions
First install the library and its dependencies and then save the example to example.ts and run following commands:
DD_SITE="datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" tsc "example.ts"DD_SITE="us3.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" tsc "example.ts"DD_SITE="us5.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" tsc "example.ts"DD_SITE="datadoghq.eu" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" tsc "example.ts"DD_SITE="ap1.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" tsc "example.ts"DD_SITE="ap2.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" tsc "example.ts"DD_SITE="uk1.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" tsc "example.ts"DD_SITE="ddog-gov.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" tsc "example.ts"DD_SITE="us2.ddog-gov.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" tsc "example.ts"