Mettre à jour un pipeline

The filter query.

Create custom grok rules to parse the full message or a specific attribute of your raw event. For more information, see the parsing section.

As Datadog receives logs, it timestamps them using the value(s) from any of these default attributes.

• timestamp
• date
• _timestamp
• Timestamp
• eventTime
• published_date

If your logs put their dates in an attribute not in this list, use the log date Remapper Processor to define their date attribute as the official log timestamp. The recognized date formats are ISO8601, UNIX (the milliseconds EPOCH format), and RFC3164.

Note: If your logs don’t contain any of the default attributes and you haven’t defined your own date attribute, Datadog timestamps the logs with the date it received them.

If multiple log date remapper processors can be applied to a given log, only the first one (according to the pipelines order) is taken into account.

Use this Processor if you want to assign some attributes as the official status.

Each incoming status value is mapped as follows.

• Integers from 0 to 7 map to the Syslog severity standards
• Strings beginning with emerg or f (case-insensitive) map to emerg (0)
• Strings beginning with a (case-insensitive) map to alert (1)
• Strings beginning with c (case-insensitive) map to critical (2)
• Strings beginning with err (case-insensitive) map to error (3)
• Strings beginning with w (case-insensitive) map to warning (4)
• Strings beginning with n (case-insensitive) map to notice (5)
• Strings beginning with i (case-insensitive) map to info (6)
• Strings beginning with d, trace or verbose (case-insensitive) map to debug (7)
• Strings beginning with o or matching OK or Success (case-insensitive) map to OK
• All others map to info (6)

Note: If multiple log status remapper processors can be applied to a given log, only the first one (according to the pipelines order) is taken into account.

Use this processor if you want to assign one or more attributes as the official service.

Note: If multiple service remapper processors can be applied to a given log, only the first one (according to the pipeline order) is taken into account.

The message is a key attribute in Datadog. It is displayed in the message column of the Log Explorer and you can do full string search on it. Use this Processor to define one or more attributes as the official log message.

Note: If multiple log message remapper processors can be applied to a given log, only the first one (according to the pipeline order) is taken into account.

The remapper processor remaps any source attribute(s) or tag to another target attribute or tag. Constraints on the tag/attribute name are explained in the Tag Best Practice documentation. Some additional constraints are applied as : or , are not allowed in the target tag/attribute name.

This processor extracts query parameters and other important parameters from a URL.

The User-Agent parser takes a User-Agent attribute and extracts the OS, browser, device, and other user data. It recognizes major bots like the Google Bot, Yahoo Slurp, and Bing.

Use the Category Processor to add a new attribute (without spaces or special characters in the new attribute name) to a log matching a provided search query. Use categories to create groups for an analytical view. For example, URL groups, machine groups, environments, and response time buckets.

Notes:

• The syntax of the query is the one of Logs Explorer search bar. The query can be done on any log attribute or tag, whether it is a facet or not. Wildcards can also be used inside your query.
• Once the log has matched one of the Processor queries, it stops. Make sure they are properly ordered in case a log could match several queries.
• The names of the categories must be unique.
• Once defined in the Category Processor, you can map categories to log status using the Log Status Remapper.

Use the Arithmetic Processor to add a new attribute (without spaces or special characters in the new attribute name) to a log with the result of the provided formula. This enables you to remap different time attributes with different units into a single attribute, or to compute operations on attributes within the same log.

The formula can use parentheses and the basic arithmetic operators -, +, *, /.

By default, the calculation is skipped if an attribute is missing. Select “Replace missing attribute by 0” to automatically populate missing attribute values with 0 to ensure that the calculation is done. An attribute is missing if it is not found in the log attributes, or if it cannot be converted to a number.

Notes:

• The operator - needs to be space split in the formula as it can also be contained in attribute names.
• If the target attribute already exists, it is overwritten by the result of the formula.
• Results are rounded up to the 9th decimal. For example, if the result of the formula is 0.1234567891, the actual value stored for the attribute is 0.123456789.
• If you need to scale a unit of measure, see Scale Filter.

Use the string builder processor to add a new attribute (without spaces or special characters) to a log with the result of the provided template. This enables aggregation of different attributes or raw strings into a single attribute.

The template is defined by both raw text and blocks with the syntax %{attribute_path}.

Notes:

• The processor only accepts attributes with values or an array of values in the blocks.
• If an attribute cannot be used (object or array of object), it is replaced by an empty string or the entire operation is skipped depending on your selection.
• If the target attribute already exists, it is overwritten by the result of the template.
• Results of the template cannot exceed 256 characters.

Nested Pipelines are pipelines within a pipeline. Use Nested Pipelines to split the processing into two steps. For example, first use a high-level filtering such as team and then a second level of filtering based on the integration, service, or any other tag or attribute.

A pipeline can contain Nested Pipelines and Processors whereas a Nested Pipeline can only contain Processors.

The GeoIP parser takes an IP address attribute and extracts if available the Continent, Country, Subdivision, and City information in the target attribute path.

Use the Lookup Processor to define a mapping between a log attribute and a human readable value saved in the processors mapping table. For example, you can use the Lookup Processor to map an internal service ID into a human readable service name. Alternatively, you could also use it to check if the MAC address that just attempted to connect to the production environment belongs to your list of stolen machines.

Note: Reference Tables are in public beta. Use the Lookup Processor to define a mapping between a log attribute and a human readable value saved in a Reference Table. For example, you can use the Lookup Processor to map an internal service ID into a human readable service name. Alternatively, you could also use it to check if the MAC address that just attempted to connect to the production environment belongs to your list of stolen machines.

There are two ways to improve correlation between application traces and logs.

1. Follow the documentation on how to inject a trace ID in the application logs and by default log integrations take care of all the rest of the setup.

2. Use the Trace remapper processor to define a log attribute as its associated trace ID.

There are two ways to define correlation between application spans and logs:

1. Follow the documentation on how to inject a span ID in the application logs. Log integrations automatically handle all remaining setup steps by default.

2. Use the span remapper processor to define a log attribute as its associated span ID.

A processor for extracting, aggregating, or transforming values from JSON arrays within your logs. Supported operations are:

• Select value from matching element
• Compute array length
• Append a value to an array
• Extract key-value pairs from an array

The decoder processor decodes any source attribute containing a base64/base16-encoded UTF-8/ASCII string back to its original value, storing the result in a target attribute.

A processor that has additional validations and checks for a given schema. Currently supported schema types include OCSF.

Use this processor to remove an attribute from a log during processing. The processor strips the specified attribute from the log event, which is useful when the attribute contains sensitive data or is no longer needed downstream.

The array-map processor transforms each element of a source array by applying sub-processors in order and collecting the results into a target array. Results can be written to a new array, to the source array (in-place), or to an existing target array. Sub-processors can read from $sourceElem.<field> (object element field), bare $sourceElem (primitive element), or any parent log attribute path. Sub-processors write to $targetElem.<field> (object output field) or bare $targetElem (primitive output).