Create a monitor

Overview

Create a monitor using the specified options.

Monitor Types

The type of monitor chosen from:

• anomaly: query alert
• APM: query alert or trace-analytics alert
• composite: composite
• custom: service check
• forecast: query alert
• host: service check
• integration: query alert or service check
• live process: process alert
• logs: log alert
• metric: query alert
• network: service check
• outlier: query alert
• process: service check
• rum: rum alert
• SLO: slo alert
• watchdog: event-v2 alert
• event-v2: event-v2 alert
• audit: audit alert
• error-tracking: error-tracking alert
• database-monitoring: database-monitoring alert
• network-performance: network-performance alert
• cloud cost: cost alert
• network-path: network-path alert

Notes:

• Synthetic monitors are created through the Synthetics API. See the Synthetics API documentation for more information.
• Log monitors require an unscoped App Key.

Query Types

Metric Alert Query

Example: time_aggr(time_window):space_aggr:metric{tags} [by {key}] operator #

• time_aggr: avg, sum, max, min, change, or pct_change
• time_window: last_#m (with # between 1 and 10080 depending on the monitor type) or last_#h(with # between 1 and 168 depending on the monitor type) or last_1d, or last_1w
• space_aggr: avg, sum, min, or max
• tags: one or more tags (comma-separated), or *
• key: a 'key' in key:value tag syntax; defines a separate alert for each tag in the group (multi-alert)
• operator: <, <=, >, >=, ==, or !=
• #: an integer or decimal number used to set the threshold

To use a dynamic threshold on a metric monitor with a formula query, replace # with the threshold keyword (for example, ... > threshold) and provide the threshold as a query via critical_query on options.thresholds. This feature is in preview.

If you are using the _change_ or _pct_change_ time aggregator, instead use change_aggr(time_aggr(time_window), timeshift):space_aggr:metric{tags} [by {key}] operator # with:

• change_aggr change, pct_change
• time_aggr avg, sum, max, min Learn more
• time_window last_#m (between 1 and 2880 depending on the monitor type), last_#h (between 1 and 48 depending on the monitor type), or last_#d (1 or 2)
• timeshift #m_ago (5, 10, 15, or 30), #h_ago (1, 2, or 4), or 1d_ago

Use this to create an outlier monitor using the following query: avg(last_30m):outliers(avg:system.cpu.user{role:es-events-data} by {host}, 'dbscan', 7) > 0

Service Check Query

Example: "check".over(tags).last(count).by(group).count_by_status()

• check name of the check, for example datadog.agent.up
• tags one or more quoted tags (comma-separated), or "*". for example: .over("env:prod", "role:db"); over cannot be blank.
• count must be at greater than or equal to your max threshold (defined in the options). It is limited to 100. For example, if you've specified to notify on 1 critical, 3 ok, and 2 warn statuses, count should be at least 3.
• group must be specified for check monitors. Per-check grouping is already explicitly known for some service checks. For example, Postgres integration monitors are tagged by db, host, and port, and Network monitors by host, instance, and url. See Service Checks documentation for more information.

Event Alert Query

Note: The Event Alert Query has been replaced by the Event V2 Alert Query. For more information, see the Event Migration guide.

Event V2 Alert Query

Example: events(query).rollup(rollup_method[, measure]).last(time_window) operator #

• query The search query - following the Log search syntax.
• rollup_method The stats roll-up method - supports count, avg and cardinality.
• measure For avg and cardinality rollup_method - specify the measure or the facet name you want to use.
• time_window #m (between 1 and 2880), #h (between 1 and 48).
• operator <, <=, >, >=, ==, or !=.
• # an integer or decimal number used to set the threshold.

Process Alert Query

Example: processes(search).over(tags).rollup('count').last(timeframe) operator #

• search free text search string for querying processes. Matching processes match results on the Live Processes page.
• tags one or more tags (comma-separated)
• timeframe the timeframe to roll up the counts. Examples: 10m, 4h. Supported timeframes: s, m, h and d
• operator <, <=, >, >=, ==, or !=
• # an integer or decimal number used to set the threshold

Logs Alert Query

Example: logs(query).index(index_name).rollup(rollup_method[, measure]).last(time_window) operator #

• query The search query - following the Log search syntax.
• index_name For multi-index organizations, the log index in which the request is performed.
• rollup_method The stats roll-up method - supports count, avg and cardinality.
• measure For avg and cardinality rollup_method - specify the measure or the facet name you want to use.
• time_window #m (between 1 and 2880), #h (between 1 and 48).
• operator <, <=, >, >=, ==, or !=.
• # an integer or decimal number used to set the threshold.

Composite Query

Example: 12345 && 67890, where 12345 and 67890 are the IDs of non-composite monitors

• name [required, default = dynamic, based on query]: The name of the alert.
• message [required, default = dynamic, based on query]: A message to include with notifications for this monitor. Email notifications can be sent to specific users by using the same '@username' notation as events.
• tags [optional, default = empty list]: A list of tags to associate with your monitor. When getting all monitor details via the API, use the monitor_tags argument to filter results by these tags. It is only available via the API and isn't visible or editable in the Datadog UI.

SLO Alert Query

Example: error_budget("slo_id").over("time_window") operator #

• slo_id: The alphanumeric SLO ID of the SLO you are configuring the alert for.
• time_window: The time window of the SLO target you wish to alert on. Valid options: 7d, 30d, 90d.
• operator: >= or >

Audit Alert Query

Example: audits(query).rollup(rollup_method[, measure]).last(time_window) operator #

• query The search query - following the Log search syntax.
• rollup_method The stats roll-up method - supports count, avg and cardinality.
• measure For avg and cardinality rollup_method - specify the measure or the facet name you want to use.
• time_window #m (between 1 and 2880), #h (between 1 and 48).
• operator <, <=, >, >=, ==, or !=.
• # an integer or decimal number used to set the threshold.

CI Pipelines Alert Query

Example: ci-pipelines(query).rollup(rollup_method[, measure]).last(time_window) operator #

• query The search query - following the Log search syntax.
• rollup_method The stats roll-up method - supports count, avg, and cardinality.
• measure For avg and cardinality rollup_method - specify the measure or the facet name you want to use.
• time_window #m (between 1 and 2880), #h (between 1 and 48).
• operator <, <=, >, >=, ==, or !=.
• # an integer or decimal number used to set the threshold.

CI Tests Alert Query

Example: ci-tests(query).rollup(rollup_method[, measure]).last(time_window) operator #

• query The search query - following the Log search syntax.
• rollup_method The stats roll-up method - supports count, avg, and cardinality.
• measure For avg and cardinality rollup_method - specify the measure or the facet name you want to use.
• time_window #m (between 1 and 2880), #h (between 1 and 48).
• operator <, <=, >, >=, ==, or !=.
• # an integer or decimal number used to set the threshold.

Error Tracking Alert Query

"New issue" example: error-tracking(query).source(issue_source).new().rollup(rollup_method[, measure]).by(group_by).last(time_window) operator # "High impact issue" example: error-tracking(query).source(issue_source).impact().rollup(rollup_method[, measure]).by(group_by).last(time_window) operator #

• query The search query - following the Log search syntax.
• issue_source The issue source - supports all, browser, mobile and backend and defaults to all if omitted.
• rollup_method The stats roll-up method - supports count, avg, and cardinality and defaults to count if omitted.
• measure For avg and cardinality rollup_method - specify the measure or the facet name you want to use.
• group by Comma-separated list of attributes to group by - should contain at least issue.id.
• time_window #m (between 1 and 2880), #h (between 1 and 48).
• operator <, <=, >, >=, ==, or !=.
• # an integer or decimal number used to set the threshold.

Database Monitoring Alert Query

Example: database-monitoring(query).rollup(rollup_method[, measure]).last(time_window) operator #

• query The search query - following the Log search syntax.
• rollup_method The stats roll-up method - supports count, avg, and cardinality.
• measure For avg and cardinality rollup_method - specify the measure or the facet name you want to use.
• time_window #m (between 1 and 2880), #h (between 1 and 48).
• operator <, <=, >, >=, ==, or !=.
• # an integer or decimal number used to set the threshold.

Network Performance Alert Query

Example: network-performance(query).rollup(rollup_method[, measure]).last(time_window) operator #

• query The search query - following the Log search syntax.
• rollup_method The stats roll-up method - supports count, avg, and cardinality.
• measure For avg and cardinality rollup_method - specify the measure or the facet name you want to use.
• time_window #m (between 1 and 2880), #h (between 1 and 48).
• operator <, <=, >, >=, ==, or !=.
• # an integer or decimal number used to set the threshold.

Cost Alert Query

Example: formula(query).timeframe_type(time_window).function(parameter) operator #

• query The search query - following the Log search syntax.
• timeframe_type The timeframe type to evaluate the cost - for forecast supports current - for change, anomaly, threshold supports last
• time_window - supports daily roll-up e.g. 7d
• function - [optional, defaults to threshold monitor if omitted] supports change, anomaly, forecast
• parameter Specify the parameter of the type
  • for change:
    • supports relative, absolute
    • [optional] supports #, where # is an integer or decimal number used to set the threshold
  • for anomaly:
    • supports direction=both, direction=above, direction=below
    • [optional] supports threshold=#, where # is an integer or decimal number used to set the threshold
• operator
  • for threshold supports <, <=, >, >=, ==, or !=
  • for change supports >, <
  • for anomaly supports >=
  • for forecast supports >
• # an integer or decimal number used to set the threshold.

Network Path Alert Query

Example: network-path(query).index(index_name).rollup(rollup_method[, measure]).last(time_window) operator #

• query The search query - following the Log search syntax.
• index_name The data type to monitor on - supports netpath-path and netpath-hop.
• rollup_method The stats roll-up method - supports count, avg, and cardinality.
• measure For avg and cardinality rollup_method - specify the measure or the facet name you want to use.
• time_window #m (between 1 and 2880), #h (between 1 and 48).
• operator <, <=, >, >=, ==, or !=.
• # an integer or decimal number used to set the threshold.

{
  "name": "Example Monitor",
  "type": "cost alert",
  "query": "formula(\"exclude_null(query1)\").last(\"7d\").anomaly(direction=\"above\", threshold=10) >= 5",
  "message": "some message Notify: @hipchat-channel",
  "tags": [
    "test:examplemonitor",
    "env:ci"
  ],
  "priority": 3,
  "options": {
    "thresholds": {
      "critical": 5,
      "warning": 3
    },
    "variables": [
      {
        "data_source": "cloud_cost",
        "query": "sum:aws.cost.net.amortized.shared.resources.allocated{aws_product IN (amplify ,athena, backup, bedrock ) } by {aws_product}.rollup(sum, 86400)",
        "name": "query1",
        "aggregator": "sum"
      }
    ],
    "include_tags": true
  }
}

{
  "name": "Example-Monitor",
  "type": "data-jobs alert",
  "query": "formula(\"failed_runs(run_query)\").by(job_name,workspace_name).last(10d) > 0",
  "message": "Data jobs alert triggered",
  "tags": [
    "test:examplemonitor",
    "env:ci"
  ],
  "options": {
    "thresholds": {
      "critical": 0
    },
    "variables": [
      {
        "name": "run_query",
        "jobs_query": "job_name:*",
        "job_type": "databricks.job",
        "query_dialect": "metric"
      }
    ]
  }
}

{
  "name": "Example-Monitor",
  "type": "data-quality alert",
  "query": "formula(\"query1\").last(\"5m\") > 100",
  "message": "Data quality alert triggered",
  "tags": [
    "test:examplemonitor",
    "env:ci"
  ],
  "priority": 3,
  "options": {
    "thresholds": {
      "critical": 100
    },
    "variables": [
      {
        "name": "query1",
        "data_source": "data_quality_metrics",
        "measure": "row_count",
        "filter": "search for column where `database:production AND table:users`",
        "group_by": [
          "entity_id"
        ]
      }
    ]
  }
}

{
  "assets": [
    {
      "category": "runbook",
      "name": "Monitor Runbook",
      "resource_key": "12345",
      "resource_type": "string",
      "url": "/notebooks/12345"
    }
  ],
  "created": "2019-09-19T10:00:00.000Z",
  "creator": {
    "email": "string",
    "handle": "string",
    "name": "string"
  },
  "deleted": "2019-09-19T10:00:00.000Z",
  "draft_status": "string",
  "id": "integer",
  "matching_downtimes": [
    {
      "end": 1412792983,
      "id": 1625,
      "scope": [
        "env:staging"
      ],
      "start": 1412792983
    }
  ],
  "message": "string",
  "modified": "2019-09-19T10:00:00.000Z",
  "multi": false,
  "name": "My monitor",
  "options": {
    "aggregation": {
      "group_by": "host",
      "metric": "metrics.name",
      "type": "count"
    },
    "device_ids": [],
    "enable_logs_sample": false,
    "enable_samples": false,
    "escalation_message": "string",
    "evaluation_delay": "integer",
    "group_retention_duration": "string",
    "groupby_simple_monitor": false,
    "include_tags": false,
    "locked": false,
    "min_failure_duration": "integer",
    "min_location_failed": "integer",
    "new_group_delay": "integer",
    "new_host_delay": "integer",
    "no_data_timeframe": "integer",
    "notification_preset_name": "string",
    "notify_audit": false,
    "notify_by": [],
    "notify_no_data": false,
    "on_missing_data": "string",
    "renotify_interval": "integer",
    "renotify_occurrences": "integer",
    "renotify_statuses": [],
    "require_full_window": false,
    "scheduling_options": {
      "custom_schedule": {
        "recurrences": [
          {
            "rrule": "FREQ=WEEKLY;BYDAY=MO,TU,WE,TH,FR",
            "start": "2023-08-31T16:30:00",
            "timezone": "Europe/Paris"
          }
        ]
      },
      "evaluation_window": {
        "day_starts": "04:00",
        "hour_starts": 0,
        "month_starts": 1,
        "timezone": "Europe/Paris"
      }
    },
    "silenced": {
      "<any-key>": "integer"
    },
    "synthetics_check_id": "string",
    "threshold_windows": {
      "recovery_window": "string",
      "trigger_window": "string"
    },
    "thresholds": {
      "critical": "number",
      "critical_query": "formula(\"2 * query1\").rollup(\"avg\").last(\"6mo\")",
      "critical_recovery": "number",
      "critical_recovery_query": "formula(\"1.5 * query1\").rollup(\"avg\").last(\"3mo\")",
      "ok": "number",
      "unknown": "number",
      "warning": "number",
      "warning_recovery": "number"
    },
    "timeout_h": "integer",
    "variables": [
      {
        "compute": {
          "aggregation": "avg",
          "interval": 60000,
          "metric": "@duration",
          "name": "compute_result",
          "source": "filter_query"
        },
        "data_source": "rum",
        "group_by": [
          {
            "facet": "status",
            "limit": 10,
            "sort": {
              "aggregation": "avg",
              "metric": "string",
              "order": "asc"
            },
            "source": "filter_query"
          }
        ],
        "indexes": [
          "days-3",
          "days-7"
        ],
        "name": "query_errors",
        "search": {
          "query": "service:query"
        }
      }
    ]
  },
  "overall_state": "string",
  "priority": "integer",
  "query": "avg(last_5m):sum:system.net.bytes_rcvd{host:host0} > 100",
  "restricted_roles": [],
  "state": {
    "groups": {
      "<any-key>": {
        "last_nodata_ts": "integer",
        "last_notified_ts": "integer",
        "last_resolved_ts": "integer",
        "last_triggered_ts": "integer",
        "name": "string",
        "status": "string"
      }
    }
  },
  "tags": [],
  "type": "query alert"
}