Create or update a custom evaluator configuration
警告Note: This endpoint is in preview and is subject to change.
If you have any feedback, contact Datadog support.
PUT
https://api.datadoghq.com/api/unstable/llm-obs/config/evaluators/custom/{eval_name}https://api.us3.datadoghq.com/api/unstable/llm-obs/config/evaluators/custom/{eval_name}https://api.us5.datadoghq.com/api/unstable/llm-obs/config/evaluators/custom/{eval_name}https://api.datadoghq.eu/api/unstable/llm-obs/config/evaluators/custom/{eval_name}https://api.ap1.datadoghq.com/api/unstable/llm-obs/config/evaluators/custom/{eval_name}https://api.ap2.datadoghq.com/api/unstable/llm-obs/config/evaluators/custom/{eval_name}https://api.uk1.datadoghq.com/api/unstable/llm-obs/config/evaluators/custom/{eval_name}https://api.ddog-gov.com/api/unstable/llm-obs/config/evaluators/custom/{eval_name}https://api.us2.ddog-gov.com/api/unstable/llm-obs/config/evaluators/custom/{eval_name}概要
Create or update a custom Agent Observability evaluator configuration by its name.
引数
パスパラメーター
名前
タイプ
説明
eval_name [required]string
The name of the custom Agent Observability evaluator configuration.
リクエスト
Body Data (required)
Custom evaluator configuration payload.
フィールド
タイプ
説明
data [required]object
Data object for creating or updating a custom Agent Observability evaluator configuration.
attributes [required]object
Attributes for creating or updating a custom Agent Observability evaluator configuration.
categorystring
Category of the evaluator.
eval_namestring
Name of the custom evaluator. If provided, must match the eval_name path parameter.
llm_judge_configobject
LLM judge configuration for a custom evaluator.
assessment_criteriaobject
Criteria used to assess the pass/fail result of a custom evaluator.
max_thresholddouble
Maximum numeric threshold for a passing result.
min_thresholddouble
Minimum numeric threshold for a passing result.
pass_values[string]
Specific output values considered as a passing result.
pass_whenboolean
When true, a boolean output of true is treated as passing.
context_querystring
Query used to extract additional context for the evaluation.
inference_params [required]object
LLM inference parameters for a custom evaluator.
frequency_penaltydouble
Frequency penalty to reduce repetition.
max_tokensint64
Maximum number of tokens to generate.
presence_penaltydouble
Presence penalty to reduce repetition.
temperaturedouble
Sampling temperature for the LLM.
top_kint64
Top-k sampling parameter.
top_pdouble
Top-p (nucleus) sampling parameter.
last_used_library_prompt_template_namestring
Name of the last library prompt template used.
modified_library_prompt_templateboolean
Whether the library prompt template was modified.
output_schemaobject
JSON schema describing the expected output format of the LLM judge.
parsing_typeenum
Output parsing type for a custom LLM judge evaluator.
Allowed enum values:
structured_output, json, keyword_searchprompt_template[object]
List of messages forming the LLM judge prompt template.
contentstring
Text content of the message.
contents[object]
Multi-part content blocks for the message.
type [required]string
Content block type.
value [required]object
Value of a prompt message content block.
textstring
Text content of the message block.
tool_callobject
A tool call within a prompt message.
argumentsstring
JSON-encoded arguments for the tool call.
idstring
Unique identifier of the tool call.
namestring
Name of the tool being called.
typestring
Type of the tool call.
tool_call_resultobject
A tool call result within a prompt message.
namestring
Name of the tool that produced this result.
resultstring
The result returned by the tool.
tool_idstring
Identifier of the tool call this result corresponds to.
typestring
Type of the tool result.
role [required]string
Role of the message author.
target_querystring
Query used to extract the target value to evaluate.
user_specified_json_post_processing_functionstring
User-provided function applied to post-process the JSON output of the LLM judge.
llm_providerobject
LLM provider configuration for a custom evaluator.
bedrockobject
AWS Bedrock-specific options for LLM provider configuration.
inference_profilestring
Bedrock inference profile identifier, such as an application inference profile ARN.
regionstring
AWS region for Bedrock.
integration_account_idstring
Integration account identifier.
integration_providerenum
Name of the LLM integration provider.
Allowed enum values:
openai, amazon-bedrock, anthropic, azure-openai, vertex-ai, llm-proxymodel_namestring
Name of the LLM model.
vertex_aiobject
Google Vertex AI-specific options for LLM provider configuration.
locationstring
Google Cloud region.
projectstring
Google Cloud project ID.
target [required]object
Target application configuration for a custom evaluator.
application_name [required]string
Name of the ML application this evaluator targets.
enabled [required]boolean
Whether the evaluator is active for the target application.
eval_scopeenum
Scope at which to evaluate spans.
Allowed enum values:
span, trace, sessionexperiment_project_ids[string]
Experiment project IDs this evaluator is scoped to.
filterstring
Filter expression to select which spans to evaluate.
root_spans_onlyboolean
When true, only root spans are evaluated.
sampling_percentagedouble
Percentage of traces to evaluate. Must be greater than 0 and at most 100.
idstring
Name of the evaluator. If provided, must match the eval_name path parameter.
type [required]enum
Type of the custom Agent Observability evaluator configuration resource.
Allowed enum values:
evaluator_config{
"data": {
"attributes": {
"category": "Custom",
"eval_name": "my-custom-evaluator",
"llm_judge_config": {
"assessment_criteria": {
"max_threshold": 1,
"min_threshold": 0.7,
"pass_values": [
"pass",
"yes"
],
"pass_when": true
},
"context_query": "@input.context",
"inference_params": {
"frequency_penalty": 0,
"max_tokens": 1024,
"presence_penalty": 0,
"temperature": 0.7,
"top_k": 50,
"top_p": 1
},
"last_used_library_prompt_template_name": "sentiment-analysis-v1",
"modified_library_prompt_template": false,
"output_schema": {},
"parsing_type": "structured_output",
"prompt_template": [
{
"content": "Rate the quality of the following response:",
"contents": [
{
"type": "text",
"value": {
"text": "What is the sentiment of this review?",
"tool_call": {
"arguments": "{\"location\": \"San Francisco\"}",
"id": "call_abc123",
"name": "get_weather",
"type": "function"
},
"tool_call_result": {
"name": "get_weather",
"result": "sunny, 72F",
"tool_id": "call_abc123",
"type": "function"
}
}
}
],
"role": "user"
}
],
"target_query": "@output.value",
"user_specified_json_post_processing_function": "string"
},
"llm_provider": {
"bedrock": {
"inference_profile": "arn:aws:bedrock:us-east-1:123456789012:application-inference-profile/abc123",
"region": "us-east-1"
},
"integration_account_id": "my-account-id",
"integration_provider": "openai",
"model_name": "gpt-4o",
"vertex_ai": {
"location": "us-central1",
"project": "my-gcp-project"
}
},
"target": {
"application_name": "my-llm-app",
"enabled": true,
"eval_scope": "span",
"experiment_project_ids": [],
"filter": "@service:my-service",
"root_spans_only": true,
"sampling_percentage": 50
}
},
"id": "my-custom-evaluator",
"type": "evaluator_config"
}
}応答
OK
Bad Request
API error response.
フィールド
タイプ
説明
errors [required][object]
A list of errors.
detailstring
A human-readable explanation specific to this occurrence of the error.
metaobject
Non-standard meta-information about the error
sourceobject
References to the source of the error.
headerstring
A string indicating the name of a single request header which caused the error.
parameterstring
A string indicating which URI query parameter caused the error.
pointerstring
A JSON pointer to the value in the request document that caused the error.
statusstring
Status code of the response.
titlestring
Short human-readable summary of the error.
{
"errors": [
{
"detail": "Missing required attribute in body",
"meta": {},
"source": {
"header": "Authorization",
"parameter": "limit",
"pointer": "/data/attributes/title"
},
"status": "400",
"title": "Bad Request"
}
]
}Unauthorized
API error response.
フィールド
タイプ
説明
errors [required][object]
A list of errors.
detailstring
A human-readable explanation specific to this occurrence of the error.
metaobject
Non-standard meta-information about the error
sourceobject
References to the source of the error.
headerstring
A string indicating the name of a single request header which caused the error.
parameterstring
A string indicating which URI query parameter caused the error.
pointerstring
A JSON pointer to the value in the request document that caused the error.
statusstring
Status code of the response.
titlestring
Short human-readable summary of the error.
{
"errors": [
{
"detail": "Missing required attribute in body",
"meta": {},
"source": {
"header": "Authorization",
"parameter": "limit",
"pointer": "/data/attributes/title"
},
"status": "400",
"title": "Bad Request"
}
]
}Forbidden
API error response.
フィールド
タイプ
説明
errors [required][object]
A list of errors.
detailstring
A human-readable explanation specific to this occurrence of the error.
metaobject
Non-standard meta-information about the error
sourceobject
References to the source of the error.
headerstring
A string indicating the name of a single request header which caused the error.
parameterstring
A string indicating which URI query parameter caused the error.
pointerstring
A JSON pointer to the value in the request document that caused the error.
statusstring
Status code of the response.
titlestring
Short human-readable summary of the error.
{
"errors": [
{
"detail": "Missing required attribute in body",
"meta": {},
"source": {
"header": "Authorization",
"parameter": "limit",
"pointer": "/data/attributes/title"
},
"status": "400",
"title": "Bad Request"
}
]
}Not Found
API error response.
フィールド
タイプ
説明
errors [required][object]
A list of errors.
detailstring
A human-readable explanation specific to this occurrence of the error.
metaobject
Non-standard meta-information about the error
sourceobject
References to the source of the error.
headerstring
A string indicating the name of a single request header which caused the error.
parameterstring
A string indicating which URI query parameter caused the error.
pointerstring
A JSON pointer to the value in the request document that caused the error.
statusstring
Status code of the response.
titlestring
Short human-readable summary of the error.
{
"errors": [
{
"detail": "Missing required attribute in body",
"meta": {},
"source": {
"header": "Authorization",
"parameter": "limit",
"pointer": "/data/attributes/title"
},
"status": "400",
"title": "Bad Request"
}
]
}Unprocessable Entity
API error response.
フィールド
タイプ
説明
errors [required][object]
A list of errors.
detailstring
A human-readable explanation specific to this occurrence of the error.
metaobject
Non-standard meta-information about the error
sourceobject
References to the source of the error.
headerstring
A string indicating the name of a single request header which caused the error.
parameterstring
A string indicating which URI query parameter caused the error.
pointerstring
A JSON pointer to the value in the request document that caused the error.
statusstring
Status code of the response.
titlestring
Short human-readable summary of the error.
{
"errors": [
{
"detail": "Missing required attribute in body",
"meta": {},
"source": {
"header": "Authorization",
"parameter": "limit",
"pointer": "/data/attributes/title"
},
"status": "400",
"title": "Bad Request"
}
]
}Too many requests
API error response.
フィールド
タイプ
説明
errors [required][string]
A list of errors.
{
"errors": [
"Bad Request"
]
}コード例
Create or update a custom evaluator configuration
## default
# Path parameters
export eval_name="my-custom-evaluator"
# Set your Datadog site and credentials
export DD_SITE="datadoghq.com"
# Use a Personal Access Token or Service Access Token
export DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>"
curl -X PUT "https://api.${DD_SITE}/api/unstable/llm-obs/config/evaluators/custom/${eval_name}" \
-H "Accept: application/json" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer ${DD_BEARER_TOKEN}" \
-d @- << EOF
{
"data": {
"attributes": {
"llm_judge_config": {
"inference_params": {
"max_tokens": 1024,
"temperature": 0.7
},
"parsing_type": "structured_output"
},
"llm_provider": {
"integration_provider": "openai",
"model_name": "gpt-4o"
},
"target": {
"application_name": "my-llm-app",
"enabled": true,
"sampling_percentage": 50
}
},
"id": "my-custom-evaluator",
"type": "evaluator_config"
}
}
EOF
## Full example with prompt template, output schema, and assessment criteria
# Path parameters
export eval_name="my-custom-evaluator"
# Set your Datadog site and credentials
export DD_SITE="datadoghq.com"
# Use a Personal Access Token or Service Access Token
export DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>"
curl -X PUT "https://api.${DD_SITE}/api/unstable/llm-obs/config/evaluators/custom/${eval_name}" \
-H "Accept: application/json" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer ${DD_BEARER_TOKEN}" \
-d @- << EOF
{
"data": {
"attributes": {
"category": "Custom",
"eval_name": "my-custom-evaluator",
"llm_judge_config": {
"assessment_criteria": {
"pass_when": false
},
"inference_params": {
"frequency_penalty": 0,
"max_tokens": 4096,
"presence_penalty": 0,
"temperature": 1,
"top_p": 1
},
"output_schema": {
"name": "boolean_eval",
"strict": true
},
"parsing_type": "structured_output",
"prompt_template": [
{
"content": "You are a judge LLM.",
"role": "system"
},
{
"content": "{{span_output}}",
"role": "user"
}
]
},
"llm_provider": {
"integration_account_id": "your-account-uuid",
"integration_provider": "openai",
"model_name": "gpt-4o"
},
"target": {
"application_name": "my-llm-app",
"enabled": true,
"eval_scope": "span",
"filter": "@meta.span.kind:llm",
"root_spans_only": false,
"sampling_percentage": 100
}
},
"id": "my-custom-evaluator",
"type": "evaluator_config"
}
}
EOF## default
# Path parameters
export eval_name="my-custom-evaluator"
# Set your Datadog site and credentials
export DD_SITE="us3.datadoghq.com"
# Use a Personal Access Token or Service Access Token
export DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>"
curl -X PUT "https://api.${DD_SITE}/api/unstable/llm-obs/config/evaluators/custom/${eval_name}" \
-H "Accept: application/json" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer ${DD_BEARER_TOKEN}" \
-d @- << EOF
{
"data": {
"attributes": {
"llm_judge_config": {
"inference_params": {
"max_tokens": 1024,
"temperature": 0.7
},
"parsing_type": "structured_output"
},
"llm_provider": {
"integration_provider": "openai",
"model_name": "gpt-4o"
},
"target": {
"application_name": "my-llm-app",
"enabled": true,
"sampling_percentage": 50
}
},
"id": "my-custom-evaluator",
"type": "evaluator_config"
}
}
EOF
## Full example with prompt template, output schema, and assessment criteria
# Path parameters
export eval_name="my-custom-evaluator"
# Set your Datadog site and credentials
export DD_SITE="us3.datadoghq.com"
# Use a Personal Access Token or Service Access Token
export DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>"
curl -X PUT "https://api.${DD_SITE}/api/unstable/llm-obs/config/evaluators/custom/${eval_name}" \
-H "Accept: application/json" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer ${DD_BEARER_TOKEN}" \
-d @- << EOF
{
"data": {
"attributes": {
"category": "Custom",
"eval_name": "my-custom-evaluator",
"llm_judge_config": {
"assessment_criteria": {
"pass_when": false
},
"inference_params": {
"frequency_penalty": 0,
"max_tokens": 4096,
"presence_penalty": 0,
"temperature": 1,
"top_p": 1
},
"output_schema": {
"name": "boolean_eval",
"strict": true
},
"parsing_type": "structured_output",
"prompt_template": [
{
"content": "You are a judge LLM.",
"role": "system"
},
{
"content": "{{span_output}}",
"role": "user"
}
]
},
"llm_provider": {
"integration_account_id": "your-account-uuid",
"integration_provider": "openai",
"model_name": "gpt-4o"
},
"target": {
"application_name": "my-llm-app",
"enabled": true,
"eval_scope": "span",
"filter": "@meta.span.kind:llm",
"root_spans_only": false,
"sampling_percentage": 100
}
},
"id": "my-custom-evaluator",
"type": "evaluator_config"
}
}
EOF## default
# Path parameters
export eval_name="my-custom-evaluator"
# Set your Datadog site and credentials
export DD_SITE="us5.datadoghq.com"
# Use a Personal Access Token or Service Access Token
export DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>"
curl -X PUT "https://api.${DD_SITE}/api/unstable/llm-obs/config/evaluators/custom/${eval_name}" \
-H "Accept: application/json" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer ${DD_BEARER_TOKEN}" \
-d @- << EOF
{
"data": {
"attributes": {
"llm_judge_config": {
"inference_params": {
"max_tokens": 1024,
"temperature": 0.7
},
"parsing_type": "structured_output"
},
"llm_provider": {
"integration_provider": "openai",
"model_name": "gpt-4o"
},
"target": {
"application_name": "my-llm-app",
"enabled": true,
"sampling_percentage": 50
}
},
"id": "my-custom-evaluator",
"type": "evaluator_config"
}
}
EOF
## Full example with prompt template, output schema, and assessment criteria
# Path parameters
export eval_name="my-custom-evaluator"
# Set your Datadog site and credentials
export DD_SITE="us5.datadoghq.com"
# Use a Personal Access Token or Service Access Token
export DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>"
curl -X PUT "https://api.${DD_SITE}/api/unstable/llm-obs/config/evaluators/custom/${eval_name}" \
-H "Accept: application/json" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer ${DD_BEARER_TOKEN}" \
-d @- << EOF
{
"data": {
"attributes": {
"category": "Custom",
"eval_name": "my-custom-evaluator",
"llm_judge_config": {
"assessment_criteria": {
"pass_when": false
},
"inference_params": {
"frequency_penalty": 0,
"max_tokens": 4096,
"presence_penalty": 0,
"temperature": 1,
"top_p": 1
},
"output_schema": {
"name": "boolean_eval",
"strict": true
},
"parsing_type": "structured_output",
"prompt_template": [
{
"content": "You are a judge LLM.",
"role": "system"
},
{
"content": "{{span_output}}",
"role": "user"
}
]
},
"llm_provider": {
"integration_account_id": "your-account-uuid",
"integration_provider": "openai",
"model_name": "gpt-4o"
},
"target": {
"application_name": "my-llm-app",
"enabled": true,
"eval_scope": "span",
"filter": "@meta.span.kind:llm",
"root_spans_only": false,
"sampling_percentage": 100
}
},
"id": "my-custom-evaluator",
"type": "evaluator_config"
}
}
EOF## default
# Path parameters
export eval_name="my-custom-evaluator"
# Set your Datadog site and credentials
export DD_SITE="datadoghq.eu"
# Use a Personal Access Token or Service Access Token
export DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>"
curl -X PUT "https://api.${DD_SITE}/api/unstable/llm-obs/config/evaluators/custom/${eval_name}" \
-H "Accept: application/json" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer ${DD_BEARER_TOKEN}" \
-d @- << EOF
{
"data": {
"attributes": {
"llm_judge_config": {
"inference_params": {
"max_tokens": 1024,
"temperature": 0.7
},
"parsing_type": "structured_output"
},
"llm_provider": {
"integration_provider": "openai",
"model_name": "gpt-4o"
},
"target": {
"application_name": "my-llm-app",
"enabled": true,
"sampling_percentage": 50
}
},
"id": "my-custom-evaluator",
"type": "evaluator_config"
}
}
EOF
## Full example with prompt template, output schema, and assessment criteria
# Path parameters
export eval_name="my-custom-evaluator"
# Set your Datadog site and credentials
export DD_SITE="datadoghq.eu"
# Use a Personal Access Token or Service Access Token
export DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>"
curl -X PUT "https://api.${DD_SITE}/api/unstable/llm-obs/config/evaluators/custom/${eval_name}" \
-H "Accept: application/json" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer ${DD_BEARER_TOKEN}" \
-d @- << EOF
{
"data": {
"attributes": {
"category": "Custom",
"eval_name": "my-custom-evaluator",
"llm_judge_config": {
"assessment_criteria": {
"pass_when": false
},
"inference_params": {
"frequency_penalty": 0,
"max_tokens": 4096,
"presence_penalty": 0,
"temperature": 1,
"top_p": 1
},
"output_schema": {
"name": "boolean_eval",
"strict": true
},
"parsing_type": "structured_output",
"prompt_template": [
{
"content": "You are a judge LLM.",
"role": "system"
},
{
"content": "{{span_output}}",
"role": "user"
}
]
},
"llm_provider": {
"integration_account_id": "your-account-uuid",
"integration_provider": "openai",
"model_name": "gpt-4o"
},
"target": {
"application_name": "my-llm-app",
"enabled": true,
"eval_scope": "span",
"filter": "@meta.span.kind:llm",
"root_spans_only": false,
"sampling_percentage": 100
}
},
"id": "my-custom-evaluator",
"type": "evaluator_config"
}
}
EOF## default
# Path parameters
export eval_name="my-custom-evaluator"
# Set your Datadog site and credentials
export DD_SITE="ap1.datadoghq.com"
# Use a Personal Access Token or Service Access Token
export DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>"
curl -X PUT "https://api.${DD_SITE}/api/unstable/llm-obs/config/evaluators/custom/${eval_name}" \
-H "Accept: application/json" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer ${DD_BEARER_TOKEN}" \
-d @- << EOF
{
"data": {
"attributes": {
"llm_judge_config": {
"inference_params": {
"max_tokens": 1024,
"temperature": 0.7
},
"parsing_type": "structured_output"
},
"llm_provider": {
"integration_provider": "openai",
"model_name": "gpt-4o"
},
"target": {
"application_name": "my-llm-app",
"enabled": true,
"sampling_percentage": 50
}
},
"id": "my-custom-evaluator",
"type": "evaluator_config"
}
}
EOF
## Full example with prompt template, output schema, and assessment criteria
# Path parameters
export eval_name="my-custom-evaluator"
# Set your Datadog site and credentials
export DD_SITE="ap1.datadoghq.com"
# Use a Personal Access Token or Service Access Token
export DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>"
curl -X PUT "https://api.${DD_SITE}/api/unstable/llm-obs/config/evaluators/custom/${eval_name}" \
-H "Accept: application/json" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer ${DD_BEARER_TOKEN}" \
-d @- << EOF
{
"data": {
"attributes": {
"category": "Custom",
"eval_name": "my-custom-evaluator",
"llm_judge_config": {
"assessment_criteria": {
"pass_when": false
},
"inference_params": {
"frequency_penalty": 0,
"max_tokens": 4096,
"presence_penalty": 0,
"temperature": 1,
"top_p": 1
},
"output_schema": {
"name": "boolean_eval",
"strict": true
},
"parsing_type": "structured_output",
"prompt_template": [
{
"content": "You are a judge LLM.",
"role": "system"
},
{
"content": "{{span_output}}",
"role": "user"
}
]
},
"llm_provider": {
"integration_account_id": "your-account-uuid",
"integration_provider": "openai",
"model_name": "gpt-4o"
},
"target": {
"application_name": "my-llm-app",
"enabled": true,
"eval_scope": "span",
"filter": "@meta.span.kind:llm",
"root_spans_only": false,
"sampling_percentage": 100
}
},
"id": "my-custom-evaluator",
"type": "evaluator_config"
}
}
EOF## default
# Path parameters
export eval_name="my-custom-evaluator"
# Set your Datadog site and credentials
export DD_SITE="ap2.datadoghq.com"
# Use a Personal Access Token or Service Access Token
export DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>"
curl -X PUT "https://api.${DD_SITE}/api/unstable/llm-obs/config/evaluators/custom/${eval_name}" \
-H "Accept: application/json" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer ${DD_BEARER_TOKEN}" \
-d @- << EOF
{
"data": {
"attributes": {
"llm_judge_config": {
"inference_params": {
"max_tokens": 1024,
"temperature": 0.7
},
"parsing_type": "structured_output"
},
"llm_provider": {
"integration_provider": "openai",
"model_name": "gpt-4o"
},
"target": {
"application_name": "my-llm-app",
"enabled": true,
"sampling_percentage": 50
}
},
"id": "my-custom-evaluator",
"type": "evaluator_config"
}
}
EOF
## Full example with prompt template, output schema, and assessment criteria
# Path parameters
export eval_name="my-custom-evaluator"
# Set your Datadog site and credentials
export DD_SITE="ap2.datadoghq.com"
# Use a Personal Access Token or Service Access Token
export DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>"
curl -X PUT "https://api.${DD_SITE}/api/unstable/llm-obs/config/evaluators/custom/${eval_name}" \
-H "Accept: application/json" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer ${DD_BEARER_TOKEN}" \
-d @- << EOF
{
"data": {
"attributes": {
"category": "Custom",
"eval_name": "my-custom-evaluator",
"llm_judge_config": {
"assessment_criteria": {
"pass_when": false
},
"inference_params": {
"frequency_penalty": 0,
"max_tokens": 4096,
"presence_penalty": 0,
"temperature": 1,
"top_p": 1
},
"output_schema": {
"name": "boolean_eval",
"strict": true
},
"parsing_type": "structured_output",
"prompt_template": [
{
"content": "You are a judge LLM.",
"role": "system"
},
{
"content": "{{span_output}}",
"role": "user"
}
]
},
"llm_provider": {
"integration_account_id": "your-account-uuid",
"integration_provider": "openai",
"model_name": "gpt-4o"
},
"target": {
"application_name": "my-llm-app",
"enabled": true,
"eval_scope": "span",
"filter": "@meta.span.kind:llm",
"root_spans_only": false,
"sampling_percentage": 100
}
},
"id": "my-custom-evaluator",
"type": "evaluator_config"
}
}
EOF## default
# Path parameters
export eval_name="my-custom-evaluator"
# Set your Datadog site and credentials
export DD_SITE="uk1.datadoghq.com"
# Use a Personal Access Token or Service Access Token
export DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>"
curl -X PUT "https://api.${DD_SITE}/api/unstable/llm-obs/config/evaluators/custom/${eval_name}" \
-H "Accept: application/json" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer ${DD_BEARER_TOKEN}" \
-d @- << EOF
{
"data": {
"attributes": {
"llm_judge_config": {
"inference_params": {
"max_tokens": 1024,
"temperature": 0.7
},
"parsing_type": "structured_output"
},
"llm_provider": {
"integration_provider": "openai",
"model_name": "gpt-4o"
},
"target": {
"application_name": "my-llm-app",
"enabled": true,
"sampling_percentage": 50
}
},
"id": "my-custom-evaluator",
"type": "evaluator_config"
}
}
EOF
## Full example with prompt template, output schema, and assessment criteria
# Path parameters
export eval_name="my-custom-evaluator"
# Set your Datadog site and credentials
export DD_SITE="uk1.datadoghq.com"
# Use a Personal Access Token or Service Access Token
export DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>"
curl -X PUT "https://api.${DD_SITE}/api/unstable/llm-obs/config/evaluators/custom/${eval_name}" \
-H "Accept: application/json" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer ${DD_BEARER_TOKEN}" \
-d @- << EOF
{
"data": {
"attributes": {
"category": "Custom",
"eval_name": "my-custom-evaluator",
"llm_judge_config": {
"assessment_criteria": {
"pass_when": false
},
"inference_params": {
"frequency_penalty": 0,
"max_tokens": 4096,
"presence_penalty": 0,
"temperature": 1,
"top_p": 1
},
"output_schema": {
"name": "boolean_eval",
"strict": true
},
"parsing_type": "structured_output",
"prompt_template": [
{
"content": "You are a judge LLM.",
"role": "system"
},
{
"content": "{{span_output}}",
"role": "user"
}
]
},
"llm_provider": {
"integration_account_id": "your-account-uuid",
"integration_provider": "openai",
"model_name": "gpt-4o"
},
"target": {
"application_name": "my-llm-app",
"enabled": true,
"eval_scope": "span",
"filter": "@meta.span.kind:llm",
"root_spans_only": false,
"sampling_percentage": 100
}
},
"id": "my-custom-evaluator",
"type": "evaluator_config"
}
}
EOF## default
# Path parameters
export eval_name="my-custom-evaluator"
# Set your Datadog site and credentials
export DD_SITE="ddog-gov.com"
# Use a Personal Access Token or Service Access Token
export DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>"
curl -X PUT "https://api.${DD_SITE}/api/unstable/llm-obs/config/evaluators/custom/${eval_name}" \
-H "Accept: application/json" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer ${DD_BEARER_TOKEN}" \
-d @- << EOF
{
"data": {
"attributes": {
"llm_judge_config": {
"inference_params": {
"max_tokens": 1024,
"temperature": 0.7
},
"parsing_type": "structured_output"
},
"llm_provider": {
"integration_provider": "openai",
"model_name": "gpt-4o"
},
"target": {
"application_name": "my-llm-app",
"enabled": true,
"sampling_percentage": 50
}
},
"id": "my-custom-evaluator",
"type": "evaluator_config"
}
}
EOF
## Full example with prompt template, output schema, and assessment criteria
# Path parameters
export eval_name="my-custom-evaluator"
# Set your Datadog site and credentials
export DD_SITE="ddog-gov.com"
# Use a Personal Access Token or Service Access Token
export DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>"
curl -X PUT "https://api.${DD_SITE}/api/unstable/llm-obs/config/evaluators/custom/${eval_name}" \
-H "Accept: application/json" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer ${DD_BEARER_TOKEN}" \
-d @- << EOF
{
"data": {
"attributes": {
"category": "Custom",
"eval_name": "my-custom-evaluator",
"llm_judge_config": {
"assessment_criteria": {
"pass_when": false
},
"inference_params": {
"frequency_penalty": 0,
"max_tokens": 4096,
"presence_penalty": 0,
"temperature": 1,
"top_p": 1
},
"output_schema": {
"name": "boolean_eval",
"strict": true
},
"parsing_type": "structured_output",
"prompt_template": [
{
"content": "You are a judge LLM.",
"role": "system"
},
{
"content": "{{span_output}}",
"role": "user"
}
]
},
"llm_provider": {
"integration_account_id": "your-account-uuid",
"integration_provider": "openai",
"model_name": "gpt-4o"
},
"target": {
"application_name": "my-llm-app",
"enabled": true,
"eval_scope": "span",
"filter": "@meta.span.kind:llm",
"root_spans_only": false,
"sampling_percentage": 100
}
},
"id": "my-custom-evaluator",
"type": "evaluator_config"
}
}
EOF## default
# Path parameters
export eval_name="my-custom-evaluator"
# Set your Datadog site and credentials
export DD_SITE="us2.ddog-gov.com"
# Use a Personal Access Token or Service Access Token
export DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>"
curl -X PUT "https://api.${DD_SITE}/api/unstable/llm-obs/config/evaluators/custom/${eval_name}" \
-H "Accept: application/json" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer ${DD_BEARER_TOKEN}" \
-d @- << EOF
{
"data": {
"attributes": {
"llm_judge_config": {
"inference_params": {
"max_tokens": 1024,
"temperature": 0.7
},
"parsing_type": "structured_output"
},
"llm_provider": {
"integration_provider": "openai",
"model_name": "gpt-4o"
},
"target": {
"application_name": "my-llm-app",
"enabled": true,
"sampling_percentage": 50
}
},
"id": "my-custom-evaluator",
"type": "evaluator_config"
}
}
EOF
## Full example with prompt template, output schema, and assessment criteria
# Path parameters
export eval_name="my-custom-evaluator"
# Set your Datadog site and credentials
export DD_SITE="us2.ddog-gov.com"
# Use a Personal Access Token or Service Access Token
export DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>"
curl -X PUT "https://api.${DD_SITE}/api/unstable/llm-obs/config/evaluators/custom/${eval_name}" \
-H "Accept: application/json" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer ${DD_BEARER_TOKEN}" \
-d @- << EOF
{
"data": {
"attributes": {
"category": "Custom",
"eval_name": "my-custom-evaluator",
"llm_judge_config": {
"assessment_criteria": {
"pass_when": false
},
"inference_params": {
"frequency_penalty": 0,
"max_tokens": 4096,
"presence_penalty": 0,
"temperature": 1,
"top_p": 1
},
"output_schema": {
"name": "boolean_eval",
"strict": true
},
"parsing_type": "structured_output",
"prompt_template": [
{
"content": "You are a judge LLM.",
"role": "system"
},
{
"content": "{{span_output}}",
"role": "user"
}
]
},
"llm_provider": {
"integration_account_id": "your-account-uuid",
"integration_provider": "openai",
"model_name": "gpt-4o"
},
"target": {
"application_name": "my-llm-app",
"enabled": true,
"eval_scope": "span",
"filter": "@meta.span.kind:llm",
"root_spans_only": false,
"sampling_percentage": 100
}
},
"id": "my-custom-evaluator",
"type": "evaluator_config"
}
}
EOFCreate or update a custom evaluator configuration
// Create or update a custom evaluator configuration returns "OK" response
package main
import (
"context"
"fmt"
"os"
"github.com/DataDog/datadog-api-client-go/v2/api/datadog"
"github.com/DataDog/datadog-api-client-go/v2/api/datadogV2"
"github.com/google/uuid"
)
func main() {
body := datadogV2.LLMObsCustomEvalConfigUpdateRequest{
Data: datadogV2.LLMObsCustomEvalConfigUpdateData{
Attributes: datadogV2.LLMObsCustomEvalConfigUpdateAttributes{
Category: datadog.PtrString("Custom"),
EvalName: datadog.PtrString("my-custom-evaluator"),
LlmJudgeConfig: &datadogV2.LLMObsCustomEvalConfigLLMJudgeConfig{
AssessmentCriteria: &datadogV2.LLMObsCustomEvalConfigAssessmentCriteria{
MaxThreshold: *datadog.NewNullableFloat64(datadog.PtrFloat64(1.0)),
MinThreshold: *datadog.NewNullableFloat64(datadog.PtrFloat64(0.7)),
PassValues: *datadog.NewNullableList(&[]string{
"pass",
"yes",
}),
PassWhen: *datadog.NewNullableBool(datadog.PtrBool(true)),
},
ContextQuery: *datadog.NewNullableString(datadog.PtrString("@input.context")),
InferenceParams: datadogV2.LLMObsCustomEvalConfigInferenceParams{
FrequencyPenalty: datadog.PtrFloat64(0.0),
MaxTokens: datadog.PtrInt64(1024),
PresencePenalty: datadog.PtrFloat64(0.0),
Temperature: datadog.PtrFloat64(0.7),
TopK: datadog.PtrInt64(50),
TopP: datadog.PtrFloat64(1.0),
},
LastUsedLibraryPromptTemplateName: *datadog.NewNullableString(datadog.PtrString("sentiment-analysis-v1")),
ModifiedLibraryPromptTemplate: *datadog.NewNullableBool(datadog.PtrBool(false)),
OutputSchema: nil,
ParsingType: datadogV2.LLMOBSCUSTOMEVALCONFIGPARSINGTYPE_STRUCTURED_OUTPUT.Ptr(),
PromptTemplate: []datadogV2.LLMObsCustomEvalConfigPromptMessage{
{
Content: datadog.PtrString("Rate the quality of the following response:"),
Contents: []datadogV2.LLMObsCustomEvalConfigPromptContent{
{
Type: "text",
Value: datadogV2.LLMObsCustomEvalConfigPromptContentValue{
Text: datadog.PtrString("What is the sentiment of this review?"),
ToolCall: &datadogV2.LLMObsCustomEvalConfigPromptToolCall{
Arguments: datadog.PtrString(`{"location": "San Francisco"}`),
Id: datadog.PtrString("call_abc123"),
Name: datadog.PtrString("get_weather"),
Type: datadog.PtrString("function"),
},
ToolCallResult: &datadogV2.LLMObsCustomEvalConfigPromptToolResult{
Name: datadog.PtrString("get_weather"),
Result: datadog.PtrString("sunny, 72F"),
ToolId: datadog.PtrString("call_abc123"),
Type: datadog.PtrString("function"),
},
},
},
},
Role: "user",
},
},
TargetQuery: *datadog.NewNullableString(datadog.PtrString("@output.value")),
UserSpecifiedJsonPostProcessingFunction: *datadog.NewNullableString(nil),
},
LlmProvider: &datadogV2.LLMObsCustomEvalConfigLLMProvider{
Bedrock: &datadogV2.LLMObsCustomEvalConfigBedrockOptions{
InferenceProfile: datadog.PtrString("arn:aws:bedrock:us-east-1:123456789012:application-inference-profile/abc123"),
Region: datadog.PtrString("us-east-1"),
},
IntegrationAccountId: datadog.PtrString("my-account-id"),
IntegrationProvider: datadogV2.LLMOBSCUSTOMEVALCONFIGINTEGRATIONPROVIDER_OPENAI.Ptr(),
ModelName: datadog.PtrString("gpt-4o"),
VertexAi: &datadogV2.LLMObsCustomEvalConfigVertexAIOptions{
Location: datadog.PtrString("us-central1"),
Project: datadog.PtrString("my-gcp-project"),
},
},
Target: datadogV2.LLMObsCustomEvalConfigTarget{
ApplicationName: "my-llm-app",
Enabled: true,
EvalScope: datadogV2.LLMOBSCUSTOMEVALCONFIGEVALSCOPE_SPAN.Ptr(),
ExperimentProjectIds: []uuid.UUID{},
Filter: *datadog.NewNullableString(datadog.PtrString("@service:my-service")),
RootSpansOnly: *datadog.NewNullableBool(datadog.PtrBool(true)),
SamplingPercentage: *datadog.NewNullableFloat64(datadog.PtrFloat64(50.0)),
},
},
Id: datadog.PtrString("my-custom-evaluator"),
Type: datadogV2.LLMOBSCUSTOMEVALCONFIGTYPE_EVALUATOR_CONFIG,
},
}
ctx := datadog.NewDefaultContext(context.Background())
ctx = context.WithValue(ctx, datadog.ContextAccessToken, os.Getenv("DD_BEARER_TOKEN"))
configuration := datadog.NewConfiguration()
configuration.SetUnstableOperationEnabled("v2.UpdateLLMObsCustomEvalConfig", true)
apiClient := datadog.NewAPIClient(configuration)
api := datadogV2.NewAgentObservabilityApi(apiClient)
r, err := api.UpdateLLMObsCustomEvalConfig(ctx, "eval_name", body)
if err != nil {
fmt.Fprintf(os.Stderr, "Error when calling `AgentObservabilityApi.UpdateLLMObsCustomEvalConfig`: %v\n", err)
fmt.Fprintf(os.Stderr, "Full HTTP response: %v\n", r)
}
}
Instructions
First install the library and its dependencies and then save the example to main.go and run following commands:
DD_SITE="datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" go run "main.go"DD_SITE="us3.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" go run "main.go"DD_SITE="us5.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" go run "main.go"DD_SITE="datadoghq.eu" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" go run "main.go"DD_SITE="ap1.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" go run "main.go"DD_SITE="ap2.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" go run "main.go"DD_SITE="uk1.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" go run "main.go"DD_SITE="ddog-gov.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" go run "main.go"DD_SITE="us2.ddog-gov.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" go run "main.go"Create or update a custom evaluator configuration
// Create or update a custom evaluator configuration returns "OK" response
import com.datadog.api.client.ApiClient;
import com.datadog.api.client.ApiException;
import com.datadog.api.client.v2.api.AgentObservabilityApi;
import com.datadog.api.client.v2.model.LLMObsCustomEvalConfigAssessmentCriteria;
import com.datadog.api.client.v2.model.LLMObsCustomEvalConfigBedrockOptions;
import com.datadog.api.client.v2.model.LLMObsCustomEvalConfigEvalScope;
import com.datadog.api.client.v2.model.LLMObsCustomEvalConfigInferenceParams;
import com.datadog.api.client.v2.model.LLMObsCustomEvalConfigIntegrationProvider;
import com.datadog.api.client.v2.model.LLMObsCustomEvalConfigLLMJudgeConfig;
import com.datadog.api.client.v2.model.LLMObsCustomEvalConfigLLMProvider;
import com.datadog.api.client.v2.model.LLMObsCustomEvalConfigParsingType;
import com.datadog.api.client.v2.model.LLMObsCustomEvalConfigPromptContent;
import com.datadog.api.client.v2.model.LLMObsCustomEvalConfigPromptContentValue;
import com.datadog.api.client.v2.model.LLMObsCustomEvalConfigPromptMessage;
import com.datadog.api.client.v2.model.LLMObsCustomEvalConfigPromptToolCall;
import com.datadog.api.client.v2.model.LLMObsCustomEvalConfigPromptToolResult;
import com.datadog.api.client.v2.model.LLMObsCustomEvalConfigTarget;
import com.datadog.api.client.v2.model.LLMObsCustomEvalConfigType;
import com.datadog.api.client.v2.model.LLMObsCustomEvalConfigUpdateAttributes;
import com.datadog.api.client.v2.model.LLMObsCustomEvalConfigUpdateData;
import com.datadog.api.client.v2.model.LLMObsCustomEvalConfigUpdateRequest;
import com.datadog.api.client.v2.model.LLMObsCustomEvalConfigVertexAIOptions;
import java.util.Arrays;
import java.util.Collections;
public class Example {
public static void main(String[] args) {
ApiClient defaultClient = ApiClient.getDefaultApiClient();
defaultClient.setAccessToken(System.getenv("DD_BEARER_TOKEN"));
defaultClient.setUnstableOperationEnabled("v2.updateLLMObsCustomEvalConfig", true);
AgentObservabilityApi apiInstance = new AgentObservabilityApi(defaultClient);
LLMObsCustomEvalConfigUpdateRequest body =
new LLMObsCustomEvalConfigUpdateRequest()
.data(
new LLMObsCustomEvalConfigUpdateData()
.attributes(
new LLMObsCustomEvalConfigUpdateAttributes()
.category("Custom")
.evalName("my-custom-evaluator")
.llmJudgeConfig(
new LLMObsCustomEvalConfigLLMJudgeConfig()
.assessmentCriteria(
new LLMObsCustomEvalConfigAssessmentCriteria()
.maxThreshold(1.0)
.minThreshold(0.7)
.passValues(Arrays.asList("pass", "yes"))
.passWhen(true))
.contextQuery("@input.context")
.inferenceParams(
new LLMObsCustomEvalConfigInferenceParams()
.maxTokens(1024L)
.temperature(0.7)
.topK(50L)
.topP(1.0))
.lastUsedLibraryPromptTemplateName("sentiment-analysis-v1")
.modifiedLibraryPromptTemplate(false)
.outputSchema(null)
.parsingType(
LLMObsCustomEvalConfigParsingType.STRUCTURED_OUTPUT)
.promptTemplate(
Collections.singletonList(
new LLMObsCustomEvalConfigPromptMessage()
.content(
"Rate the quality of the following response:")
.contents(
Collections.singletonList(
new LLMObsCustomEvalConfigPromptContent()
.type("text")
.value(
new LLMObsCustomEvalConfigPromptContentValue()
.text(
"What is the sentiment of"
+ " this review?")
.toolCall(
new LLMObsCustomEvalConfigPromptToolCall()
.arguments(
"""
{"location": "San Francisco"}
""")
.id("call_abc123")
.name("get_weather")
.type("function"))
.toolCallResult(
new LLMObsCustomEvalConfigPromptToolResult()
.name("get_weather")
.result("sunny, 72F")
.toolId("call_abc123")
.type("function")))))
.role("user")))
.targetQuery("@output.value")
.userSpecifiedJsonPostProcessingFunction(null))
.llmProvider(
new LLMObsCustomEvalConfigLLMProvider()
.bedrock(
new LLMObsCustomEvalConfigBedrockOptions()
.inferenceProfile(
"arn:aws:bedrock:us-east-1:123456789012:application-inference-profile/abc123")
.region("us-east-1"))
.integrationAccountId("my-account-id")
.integrationProvider(
LLMObsCustomEvalConfigIntegrationProvider.OPENAI)
.modelName("gpt-4o")
.vertexAi(
new LLMObsCustomEvalConfigVertexAIOptions()
.location("us-central1")
.project("my-gcp-project")))
.target(
new LLMObsCustomEvalConfigTarget()
.applicationName("my-llm-app")
.enabled(true)
.evalScope(LLMObsCustomEvalConfigEvalScope.SPAN)
.filter("@service:my-service")
.rootSpansOnly(true)
.samplingPercentage(50.0)))
.id("my-custom-evaluator")
.type(LLMObsCustomEvalConfigType.EVALUATOR_CONFIG));
try {
apiInstance.updateLLMObsCustomEvalConfig("my-custom-evaluator", body);
} catch (ApiException e) {
System.err.println(
"Exception when calling AgentObservabilityApi#updateLLMObsCustomEvalConfig");
System.err.println("Status code: " + e.getCode());
System.err.println("Reason: " + e.getResponseBody());
System.err.println("Response headers: " + e.getResponseHeaders());
e.printStackTrace();
}
}
}
Instructions
First install the library and its dependencies and then save the example to Example.java and run following commands:
DD_SITE="datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" java "Example.java"DD_SITE="us3.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" java "Example.java"DD_SITE="us5.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" java "Example.java"DD_SITE="datadoghq.eu" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" java "Example.java"DD_SITE="ap1.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" java "Example.java"DD_SITE="ap2.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" java "Example.java"DD_SITE="uk1.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" java "Example.java"DD_SITE="ddog-gov.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" java "Example.java"DD_SITE="us2.ddog-gov.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" java "Example.java"Create or update a custom evaluator configuration
"""
Create or update a custom evaluator configuration returns "OK" response
"""
from os import environ
from datadog_api_client import ApiClient, Configuration
from datadog_api_client.v2.api.agent_observability_api import AgentObservabilityApi
from datadog_api_client.v2.model.llm_obs_custom_eval_config_assessment_criteria import (
LLMObsCustomEvalConfigAssessmentCriteria,
)
from datadog_api_client.v2.model.llm_obs_custom_eval_config_bedrock_options import LLMObsCustomEvalConfigBedrockOptions
from datadog_api_client.v2.model.llm_obs_custom_eval_config_eval_scope import LLMObsCustomEvalConfigEvalScope
from datadog_api_client.v2.model.llm_obs_custom_eval_config_inference_params import (
LLMObsCustomEvalConfigInferenceParams,
)
from datadog_api_client.v2.model.llm_obs_custom_eval_config_integration_provider import (
LLMObsCustomEvalConfigIntegrationProvider,
)
from datadog_api_client.v2.model.llm_obs_custom_eval_config_llm_judge_config import LLMObsCustomEvalConfigLLMJudgeConfig
from datadog_api_client.v2.model.llm_obs_custom_eval_config_llm_provider import LLMObsCustomEvalConfigLLMProvider
from datadog_api_client.v2.model.llm_obs_custom_eval_config_parsing_type import LLMObsCustomEvalConfigParsingType
from datadog_api_client.v2.model.llm_obs_custom_eval_config_prompt_content import LLMObsCustomEvalConfigPromptContent
from datadog_api_client.v2.model.llm_obs_custom_eval_config_prompt_content_value import (
LLMObsCustomEvalConfigPromptContentValue,
)
from datadog_api_client.v2.model.llm_obs_custom_eval_config_prompt_message import LLMObsCustomEvalConfigPromptMessage
from datadog_api_client.v2.model.llm_obs_custom_eval_config_prompt_tool_call import LLMObsCustomEvalConfigPromptToolCall
from datadog_api_client.v2.model.llm_obs_custom_eval_config_prompt_tool_result import (
LLMObsCustomEvalConfigPromptToolResult,
)
from datadog_api_client.v2.model.llm_obs_custom_eval_config_target import LLMObsCustomEvalConfigTarget
from datadog_api_client.v2.model.llm_obs_custom_eval_config_type import LLMObsCustomEvalConfigType
from datadog_api_client.v2.model.llm_obs_custom_eval_config_update_attributes import (
LLMObsCustomEvalConfigUpdateAttributes,
)
from datadog_api_client.v2.model.llm_obs_custom_eval_config_update_data import LLMObsCustomEvalConfigUpdateData
from datadog_api_client.v2.model.llm_obs_custom_eval_config_update_request import LLMObsCustomEvalConfigUpdateRequest
from datadog_api_client.v2.model.llm_obs_custom_eval_config_vertex_ai_options import (
LLMObsCustomEvalConfigVertexAIOptions,
)
body = LLMObsCustomEvalConfigUpdateRequest(
data=LLMObsCustomEvalConfigUpdateData(
attributes=LLMObsCustomEvalConfigUpdateAttributes(
category="Custom",
eval_name="my-custom-evaluator",
llm_judge_config=LLMObsCustomEvalConfigLLMJudgeConfig(
assessment_criteria=LLMObsCustomEvalConfigAssessmentCriteria(
max_threshold=1.0,
min_threshold=0.7,
pass_values=[
"pass",
"yes",
],
pass_when=True,
),
context_query="@input.context",
inference_params=LLMObsCustomEvalConfigInferenceParams(
frequency_penalty=0.0,
max_tokens=1024,
presence_penalty=0.0,
temperature=0.7,
top_k=50,
top_p=1.0,
),
last_used_library_prompt_template_name="sentiment-analysis-v1",
modified_library_prompt_template=False,
output_schema=None,
parsing_type=LLMObsCustomEvalConfigParsingType.STRUCTURED_OUTPUT,
prompt_template=[
LLMObsCustomEvalConfigPromptMessage(
content="Rate the quality of the following response:",
contents=[
LLMObsCustomEvalConfigPromptContent(
type="text",
value=LLMObsCustomEvalConfigPromptContentValue(
text="What is the sentiment of this review?",
tool_call=LLMObsCustomEvalConfigPromptToolCall(
arguments='{"location": "San Francisco"}',
id="call_abc123",
name="get_weather",
type="function",
),
tool_call_result=LLMObsCustomEvalConfigPromptToolResult(
name="get_weather",
result="sunny, 72F",
tool_id="call_abc123",
type="function",
),
),
),
],
role="user",
),
],
target_query="@output.value",
user_specified_json_post_processing_function=None,
),
llm_provider=LLMObsCustomEvalConfigLLMProvider(
bedrock=LLMObsCustomEvalConfigBedrockOptions(
inference_profile="arn:aws:bedrock:us-east-1:123456789012:application-inference-profile/abc123",
region="us-east-1",
),
integration_account_id="my-account-id",
integration_provider=LLMObsCustomEvalConfigIntegrationProvider.OPENAI,
model_name="gpt-4o",
vertex_ai=LLMObsCustomEvalConfigVertexAIOptions(
location="us-central1",
project="my-gcp-project",
),
),
target=LLMObsCustomEvalConfigTarget(
application_name="my-llm-app",
enabled=True,
eval_scope=LLMObsCustomEvalConfigEvalScope.SPAN,
experiment_project_ids=[],
filter="@service:my-service",
root_spans_only=True,
sampling_percentage=50.0,
),
),
id="my-custom-evaluator",
type=LLMObsCustomEvalConfigType.EVALUATOR_CONFIG,
),
)
configuration = Configuration()
configuration.access_token = environ["DD_BEARER_TOKEN"]
configuration.unstable_operations["update_llm_obs_custom_eval_config"] = True
with ApiClient(configuration) as api_client:
api_instance = AgentObservabilityApi(api_client)
api_instance.update_llm_obs_custom_eval_config(eval_name="eval_name", body=body)
Instructions
First install the library and its dependencies and then save the example to example.py and run following commands:
DD_SITE="datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" python3 "example.py"DD_SITE="us3.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" python3 "example.py"DD_SITE="us5.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" python3 "example.py"DD_SITE="datadoghq.eu" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" python3 "example.py"DD_SITE="ap1.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" python3 "example.py"DD_SITE="ap2.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" python3 "example.py"DD_SITE="uk1.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" python3 "example.py"DD_SITE="ddog-gov.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" python3 "example.py"DD_SITE="us2.ddog-gov.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" python3 "example.py"Create or update a custom evaluator configuration
# Create or update a custom evaluator configuration returns "OK" response
require "datadog_api_client"
DatadogAPIClient.configure do |config|
config.access_token = ENV["DD_BEARER_TOKEN"]
config.unstable_operations["v2.update_llm_obs_custom_eval_config".to_sym] = true
end
api_instance = DatadogAPIClient::V2::AgentObservabilityAPI.new
body = DatadogAPIClient::V2::LLMObsCustomEvalConfigUpdateRequest.new({
data: DatadogAPIClient::V2::LLMObsCustomEvalConfigUpdateData.new({
attributes: DatadogAPIClient::V2::LLMObsCustomEvalConfigUpdateAttributes.new({
category: "Custom",
eval_name: "my-custom-evaluator",
llm_judge_config: DatadogAPIClient::V2::LLMObsCustomEvalConfigLLMJudgeConfig.new({
assessment_criteria: DatadogAPIClient::V2::LLMObsCustomEvalConfigAssessmentCriteria.new({
max_threshold: 1.0,
min_threshold: 0.7,
pass_values: [
"pass",
"yes",
],
pass_when: true,
}),
context_query: "@input.context",
inference_params: DatadogAPIClient::V2::LLMObsCustomEvalConfigInferenceParams.new({
frequency_penalty: 0.0,
max_tokens: 1024,
presence_penalty: 0.0,
temperature: 0.7,
top_k: 50,
top_p: 1.0,
}),
last_used_library_prompt_template_name: "sentiment-analysis-v1",
modified_library_prompt_template: false,
output_schema: nil,
parsing_type: DatadogAPIClient::V2::LLMObsCustomEvalConfigParsingType::STRUCTURED_OUTPUT,
prompt_template: [
DatadogAPIClient::V2::LLMObsCustomEvalConfigPromptMessage.new({
content: "Rate the quality of the following response:",
contents: [
DatadogAPIClient::V2::LLMObsCustomEvalConfigPromptContent.new({
type: "text",
value: DatadogAPIClient::V2::LLMObsCustomEvalConfigPromptContentValue.new({
text: "What is the sentiment of this review?",
tool_call: DatadogAPIClient::V2::LLMObsCustomEvalConfigPromptToolCall.new({
arguments: '{"location": "San Francisco"}',
id: "call_abc123",
name: "get_weather",
type: "function",
}),
tool_call_result: DatadogAPIClient::V2::LLMObsCustomEvalConfigPromptToolResult.new({
name: "get_weather",
result: "sunny, 72F",
tool_id: "call_abc123",
type: "function",
}),
}),
}),
],
role: "user",
}),
],
target_query: "@output.value",
user_specified_json_post_processing_function: nil,
}),
llm_provider: DatadogAPIClient::V2::LLMObsCustomEvalConfigLLMProvider.new({
bedrock: DatadogAPIClient::V2::LLMObsCustomEvalConfigBedrockOptions.new({
inference_profile: "arn:aws:bedrock:us-east-1:123456789012:application-inference-profile/abc123",
region: "us-east-1",
}),
integration_account_id: "my-account-id",
integration_provider: DatadogAPIClient::V2::LLMObsCustomEvalConfigIntegrationProvider::OPENAI,
model_name: "gpt-4o",
vertex_ai: DatadogAPIClient::V2::LLMObsCustomEvalConfigVertexAIOptions.new({
location: "us-central1",
project: "my-gcp-project",
}),
}),
target: DatadogAPIClient::V2::LLMObsCustomEvalConfigTarget.new({
application_name: "my-llm-app",
enabled: true,
eval_scope: DatadogAPIClient::V2::LLMObsCustomEvalConfigEvalScope::SPAN,
experiment_project_ids: [],
filter: "@service:my-service",
root_spans_only: true,
sampling_percentage: 50.0,
}),
}),
id: "my-custom-evaluator",
type: DatadogAPIClient::V2::LLMObsCustomEvalConfigType::EVALUATOR_CONFIG,
}),
})
p api_instance.update_llm_obs_custom_eval_config("eval_name", body)
Instructions
First install the library and its dependencies and then save the example to example.rb and run following commands:
DD_SITE="datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" rb "example.rb"DD_SITE="us3.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" rb "example.rb"DD_SITE="us5.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" rb "example.rb"DD_SITE="datadoghq.eu" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" rb "example.rb"DD_SITE="ap1.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" rb "example.rb"DD_SITE="ap2.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" rb "example.rb"DD_SITE="uk1.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" rb "example.rb"DD_SITE="ddog-gov.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" rb "example.rb"DD_SITE="us2.ddog-gov.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" rb "example.rb"Create or update a custom evaluator configuration
// Create or update a custom evaluator configuration returns "OK" response
use datadog_api_client::datadog;
use datadog_api_client::datadogV2::api_agent_observability::AgentObservabilityAPI;
use datadog_api_client::datadogV2::model::LLMObsCustomEvalConfigAssessmentCriteria;
use datadog_api_client::datadogV2::model::LLMObsCustomEvalConfigBedrockOptions;
use datadog_api_client::datadogV2::model::LLMObsCustomEvalConfigEvalScope;
use datadog_api_client::datadogV2::model::LLMObsCustomEvalConfigInferenceParams;
use datadog_api_client::datadogV2::model::LLMObsCustomEvalConfigIntegrationProvider;
use datadog_api_client::datadogV2::model::LLMObsCustomEvalConfigLLMJudgeConfig;
use datadog_api_client::datadogV2::model::LLMObsCustomEvalConfigLLMProvider;
use datadog_api_client::datadogV2::model::LLMObsCustomEvalConfigParsingType;
use datadog_api_client::datadogV2::model::LLMObsCustomEvalConfigPromptContent;
use datadog_api_client::datadogV2::model::LLMObsCustomEvalConfigPromptContentValue;
use datadog_api_client::datadogV2::model::LLMObsCustomEvalConfigPromptMessage;
use datadog_api_client::datadogV2::model::LLMObsCustomEvalConfigPromptToolCall;
use datadog_api_client::datadogV2::model::LLMObsCustomEvalConfigPromptToolResult;
use datadog_api_client::datadogV2::model::LLMObsCustomEvalConfigTarget;
use datadog_api_client::datadogV2::model::LLMObsCustomEvalConfigType;
use datadog_api_client::datadogV2::model::LLMObsCustomEvalConfigUpdateAttributes;
use datadog_api_client::datadogV2::model::LLMObsCustomEvalConfigUpdateData;
use datadog_api_client::datadogV2::model::LLMObsCustomEvalConfigUpdateRequest;
use datadog_api_client::datadogV2::model::LLMObsCustomEvalConfigVertexAIOptions;
#[tokio::main]
async fn main() {
let body =
LLMObsCustomEvalConfigUpdateRequest::new(
LLMObsCustomEvalConfigUpdateData::new(
LLMObsCustomEvalConfigUpdateAttributes::new(
LLMObsCustomEvalConfigTarget::new("my-llm-app".to_string(), true)
.eval_scope(LLMObsCustomEvalConfigEvalScope::SPAN)
.experiment_project_ids(vec![])
.filter(Some("@service:my-service".to_string()))
.root_spans_only(Some(true))
.sampling_percentage(Some(50.0 as f64)),
)
.category("Custom".to_string())
.eval_name("my-custom-evaluator".to_string())
.llm_judge_config(
LLMObsCustomEvalConfigLLMJudgeConfig::new(
LLMObsCustomEvalConfigInferenceParams::new()
.frequency_penalty(0.0 as f64)
.max_tokens(1024)
.presence_penalty(0.0 as f64)
.temperature(0.7 as f64)
.top_k(50)
.top_p(1.0 as f64),
)
.assessment_criteria(
LLMObsCustomEvalConfigAssessmentCriteria::new()
.max_threshold(Some(1.0 as f64))
.min_threshold(Some(0.7 as f64))
.pass_values(Some(vec!["pass".to_string(), "yes".to_string()]))
.pass_when(Some(true)),
)
.context_query(Some("@input.context".to_string()))
.last_used_library_prompt_template_name(Some("sentiment-analysis-v1".to_string()))
.modified_library_prompt_template(Some(false))
.output_schema(None)
.parsing_type(LLMObsCustomEvalConfigParsingType::STRUCTURED_OUTPUT)
.prompt_template(
vec![
LLMObsCustomEvalConfigPromptMessage::new("user".to_string())
.content("Rate the quality of the following response:".to_string())
.contents(
vec![
LLMObsCustomEvalConfigPromptContent::new(
"text".to_string(),
LLMObsCustomEvalConfigPromptContentValue::new()
.text("What is the sentiment of this review?".to_string())
.tool_call(
LLMObsCustomEvalConfigPromptToolCall::new()
.arguments(
r#"{"location": "San Francisco"}"#.to_string(),
)
.id("call_abc123".to_string())
.name("get_weather".to_string())
.type_("function".to_string()),
)
.tool_call_result(
LLMObsCustomEvalConfigPromptToolResult::new()
.name("get_weather".to_string())
.result("sunny, 72F".to_string())
.tool_id("call_abc123".to_string())
.type_("function".to_string()),
),
)
],
)
],
)
.target_query(Some("@output.value".to_string()))
.user_specified_json_post_processing_function(None),
)
.llm_provider(
LLMObsCustomEvalConfigLLMProvider::new()
.bedrock(
LLMObsCustomEvalConfigBedrockOptions::new()
.inference_profile(
"arn:aws:bedrock:us-east-1:123456789012:application-inference-profile/abc123".to_string(),
)
.region("us-east-1".to_string()),
)
.integration_account_id("my-account-id".to_string())
.integration_provider(LLMObsCustomEvalConfigIntegrationProvider::OPENAI)
.model_name("gpt-4o".to_string())
.vertex_ai(
LLMObsCustomEvalConfigVertexAIOptions::new()
.location("us-central1".to_string())
.project("my-gcp-project".to_string()),
),
),
LLMObsCustomEvalConfigType::EVALUATOR_CONFIG,
).id("my-custom-evaluator".to_string()),
);
let mut configuration = datadog::Configuration::new();
configuration.set_unstable_operation_enabled("v2.update_llm_obs_custom_eval_config", true);
let api = AgentObservabilityAPI::with_config(configuration);
let resp = api
.update_llm_obs_custom_eval_config("eval_name".to_string(), body)
.await;
if let Ok(value) = resp {
println!("{:#?}", value);
} else {
println!("{:#?}", resp.unwrap_err());
}
}
Instructions
First install the library and its dependencies and then save the example to src/main.rs and run following commands:
DD_SITE="datadoghq.com" DD_API_KEY="<DD_API_KEY>" DD_APP_KEY="<DD_APP_KEY>" cargo runDD_SITE="us3.datadoghq.com" DD_API_KEY="<DD_API_KEY>" DD_APP_KEY="<DD_APP_KEY>" cargo runDD_SITE="us5.datadoghq.com" DD_API_KEY="<DD_API_KEY>" DD_APP_KEY="<DD_APP_KEY>" cargo runDD_SITE="datadoghq.eu" DD_API_KEY="<DD_API_KEY>" DD_APP_KEY="<DD_APP_KEY>" cargo runDD_SITE="ap1.datadoghq.com" DD_API_KEY="<DD_API_KEY>" DD_APP_KEY="<DD_APP_KEY>" cargo runDD_SITE="ap2.datadoghq.com" DD_API_KEY="<DD_API_KEY>" DD_APP_KEY="<DD_APP_KEY>" cargo runDD_SITE="uk1.datadoghq.com" DD_API_KEY="<DD_API_KEY>" DD_APP_KEY="<DD_APP_KEY>" cargo runDD_SITE="ddog-gov.com" DD_API_KEY="<DD_API_KEY>" DD_APP_KEY="<DD_APP_KEY>" cargo runDD_SITE="us2.ddog-gov.com" DD_API_KEY="<DD_API_KEY>" DD_APP_KEY="<DD_APP_KEY>" cargo runCreate or update a custom evaluator configuration
/**
* Create or update a custom evaluator configuration returns "OK" response
*/
import { client, v2 } from "@datadog/datadog-api-client";
const configuration = client.createConfiguration({
authMethods: {
AuthZ: {
accessToken: process.env.DD_BEARER_TOKEN as string,
},
},
});
configuration.unstableOperations["v2.updateLLMObsCustomEvalConfig"] = true;
const apiInstance = new v2.AgentObservabilityApi(configuration);
const params: v2.AgentObservabilityApiUpdateLLMObsCustomEvalConfigRequest = {
body: {
data: {
attributes: {
category: "Custom",
evalName: "my-custom-evaluator",
llmJudgeConfig: {
assessmentCriteria: {
maxThreshold: 1.0,
minThreshold: 0.7,
passValues: ["pass", "yes"],
passWhen: true,
},
contextQuery: "@input.context",
inferenceParams: {
frequencyPenalty: 0.0,
maxTokens: 1024,
presencePenalty: 0.0,
temperature: 0.7,
topK: 50,
topP: 1.0,
},
lastUsedLibraryPromptTemplateName: "sentiment-analysis-v1",
modifiedLibraryPromptTemplate: false,
outputSchema: undefined,
parsingType: "structured_output",
promptTemplate: [
{
content: "Rate the quality of the following response:",
contents: [
{
type: "text",
value: {
text: "What is the sentiment of this review?",
toolCall: {
arguments: `{"location": "San Francisco"}`,
id: "call_abc123",
name: "get_weather",
type: "function",
},
toolCallResult: {
name: "get_weather",
result: "sunny, 72F",
toolId: "call_abc123",
type: "function",
},
},
},
],
role: "user",
},
],
targetQuery: "@output.value",
userSpecifiedJsonPostProcessingFunction: undefined,
},
llmProvider: {
bedrock: {
inferenceProfile:
"arn:aws:bedrock:us-east-1:123456789012:application-inference-profile/abc123",
region: "us-east-1",
},
integrationAccountId: "my-account-id",
integrationProvider: "openai",
modelName: "gpt-4o",
vertexAi: {
location: "us-central1",
project: "my-gcp-project",
},
},
target: {
applicationName: "my-llm-app",
enabled: true,
evalScope: "span",
experimentProjectIds: [],
filter: "@service:my-service",
rootSpansOnly: true,
samplingPercentage: 50.0,
},
},
id: "my-custom-evaluator",
type: "evaluator_config",
},
},
evalName: "eval_name",
};
apiInstance
.updateLLMObsCustomEvalConfig(params)
.then((data: any) => {
console.log(
"API called successfully. Returned data: " + JSON.stringify(data)
);
})
.catch((error: any) => console.error(error));
Instructions
First install the library and its dependencies and then save the example to example.ts and run following commands:
DD_SITE="datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" tsc "example.ts"DD_SITE="us3.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" tsc "example.ts"DD_SITE="us5.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" tsc "example.ts"DD_SITE="datadoghq.eu" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" tsc "example.ts"DD_SITE="ap1.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" tsc "example.ts"DD_SITE="ap2.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" tsc "example.ts"DD_SITE="uk1.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" tsc "example.ts"DD_SITE="ddog-gov.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" tsc "example.ts"DD_SITE="us2.ddog-gov.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" tsc "example.ts"