Agent Observability 실험을 위한 이벤트 푸시
경고Note: This endpoint is in preview and is subject to change.
If you have any feedback, contact Datadog support.
POST
https://api.datadoghq.com/api/v2/llm-obs/v1/experiments/{experiment_id}/eventshttps://api.us3.datadoghq.com/api/v2/llm-obs/v1/experiments/{experiment_id}/eventshttps://api.us5.datadoghq.com/api/v2/llm-obs/v1/experiments/{experiment_id}/eventshttps://api.datadoghq.eu/api/v2/llm-obs/v1/experiments/{experiment_id}/eventshttps://api.ap1.datadoghq.com/api/v2/llm-obs/v1/experiments/{experiment_id}/eventshttps://api.ap2.datadoghq.com/api/v2/llm-obs/v1/experiments/{experiment_id}/eventshttps://api.uk1.datadoghq.com/api/v2/llm-obs/v1/experiments/{experiment_id}/eventshttps://api.ddog-gov.com/api/v2/llm-obs/v1/experiments/{experiment_id}/eventshttps://api.us2.ddog-gov.com/api/v2/llm-obs/v1/experiments/{experiment_id}/events개요
Agent Observability 실험을 위한 스팬 및 메트릭을 푸시합니다.
인수
경로 파라미터
이름
유형
설명
experiment_id [required]string
The ID of the Agent Observability experiment.
요청
Body Data (required)
실험 이벤트 페이로드입니다.
항목
유형
설명
data [required]object
Agent Observability 실험을 위한 스팬 및 메트릭을 푸시하도록 요청합니다.
attributes [required]object
Attributes for pushing experiment events including spans and metrics.
metrics[object]
List of metrics to push for the experiment.
assessmentenum
Assessment result for an Agent Observability experiment metric.
Allowed enum values:
pass, failboolean_valueboolean
Boolean value. Used when metric_type is boolean.
categorical_valuestring
Categorical value. Used when metric_type is categorical.
errorobject
Error details for an experiment metric evaluation.
messagestring
Error message associated with the metric evaluation.
json_valueobject
JSON value. Used when metric_type is json.
label [required]string
Label or name for the metric.
metadataobject
Arbitrary metadata associated with the metric.
metric_type [required]enum
Type of metric recorded for an Agent Observability experiment.
Allowed enum values:
score, categorical, boolean, jsonreasoningstring
Human-readable reasoning for the metric value.
score_valuedouble
Numeric score value. Used when metric_type is score.
span_id [required]string
The ID of the span this metric measures.
tags[string]
List of tags associated with the metric.
timestamp_ms [required]int64
Timestamp when the metric was recorded, in milliseconds since Unix epoch.
spans[object]
List of spans to push for the experiment.
dataset_id [required]string
Dataset ID associated with this span.
duration [required]int64
Duration of the span in nanoseconds.
metaobject
Metadata associated with an experiment span.
errorobject
Error details for an experiment span.
messagestring
Error message.
stackstring
Stack trace of the error.
typestring
The error type or exception class name.
expected_outputobject
Expected output for the span, used for evaluation.
inputobject <oneOf>
Represents any valid JSON value.
Object 1string
A scalar value represented as a string.
Object 2double
A scalar numeric value.
Object 3object
An arbitrary object value with additional properties.
Object 4[<oneOf>]
An array of arbitrary values.
Object 1string
A scalar value represented as a string.
Object 2double
A scalar numeric value.
Object 3object
An arbitrary object value with additional properties.
Object 4boolean
A scalar boolean value.
Object 5boolean
A scalar boolean value.
outputobject <oneOf>
Represents any valid JSON value.
Object 1string
A scalar value represented as a string.
Object 2double
A scalar numeric value.
Object 3object
An arbitrary object value with additional properties.
Object 4[<oneOf>]
An array of arbitrary values.
Object 1string
A scalar value represented as a string.
Object 2double
A scalar numeric value.
Object 3object
An arbitrary object value with additional properties.
Object 4boolean
A scalar boolean value.
Object 5boolean
A scalar boolean value.
name [required]string
Name of the span.
project_id [required]string
Project ID associated with this span.
span_id [required]string
Unique identifier of the span.
start_ns [required]int64
Start time of the span in nanoseconds since Unix epoch.
status [required]enum
Status of the span.
Allowed enum values:
ok, errortags[string]
List of tags associated with the span.
trace_id [required]string
Trace ID for the span.
type [required]enum
Resource type of an Agent Observability experiment.
Allowed enum values:
experiments{
"data": {
"attributes": {
"metrics": [
{
"assessment": "pass",
"label": "faithfulness",
"metric_type": "score",
"span_id": "span-7a1b2c3d",
"timestamp_ms": 1705314600000
}
],
"spans": [
{
"dataset_id": "9f64e5c7-dc5a-45c8-a17c-1b85f0bec97d",
"duration": 1500000000,
"name": "llm_call",
"project_id": "a33671aa-24fd-4dcd-9b33-a8ec7dde7751",
"span_id": "span-7a1b2c3d",
"start_ns": 1705314600000000000,
"status": "ok",
"trace_id": "abc123def456"
}
]
},
"type": "experiments"
}
}응답
Accepted
Bad Request
API error response.
항목
유형
설명
errors [required][object]
A list of errors.
detailstring
A human-readable explanation specific to this occurrence of the error.
metaobject
Non-standard meta-information about the error
sourceobject
References to the source of the error.
headerstring
A string indicating the name of a single request header which caused the error.
parameterstring
A string indicating which URI query parameter caused the error.
pointerstring
A JSON pointer to the value in the request document that caused the error.
statusstring
Status code of the response.
titlestring
Short human-readable summary of the error.
{
"errors": [
{
"detail": "Missing required attribute in body",
"meta": {},
"source": {
"header": "Authorization",
"parameter": "limit",
"pointer": "/data/attributes/title"
},
"status": "400",
"title": "Bad Request"
}
]
}Unauthorized
API error response.
항목
유형
설명
errors [required][object]
A list of errors.
detailstring
A human-readable explanation specific to this occurrence of the error.
metaobject
Non-standard meta-information about the error
sourceobject
References to the source of the error.
headerstring
A string indicating the name of a single request header which caused the error.
parameterstring
A string indicating which URI query parameter caused the error.
pointerstring
A JSON pointer to the value in the request document that caused the error.
statusstring
Status code of the response.
titlestring
Short human-readable summary of the error.
{
"errors": [
{
"detail": "Missing required attribute in body",
"meta": {},
"source": {
"header": "Authorization",
"parameter": "limit",
"pointer": "/data/attributes/title"
},
"status": "400",
"title": "Bad Request"
}
]
}Forbidden
API error response.
항목
유형
설명
errors [required][object]
A list of errors.
detailstring
A human-readable explanation specific to this occurrence of the error.
metaobject
Non-standard meta-information about the error
sourceobject
References to the source of the error.
headerstring
A string indicating the name of a single request header which caused the error.
parameterstring
A string indicating which URI query parameter caused the error.
pointerstring
A JSON pointer to the value in the request document that caused the error.
statusstring
Status code of the response.
titlestring
Short human-readable summary of the error.
{
"errors": [
{
"detail": "Missing required attribute in body",
"meta": {},
"source": {
"header": "Authorization",
"parameter": "limit",
"pointer": "/data/attributes/title"
},
"status": "400",
"title": "Bad Request"
}
]
}Not Found
API error response.
항목
유형
설명
errors [required][object]
A list of errors.
detailstring
A human-readable explanation specific to this occurrence of the error.
metaobject
Non-standard meta-information about the error
sourceobject
References to the source of the error.
headerstring
A string indicating the name of a single request header which caused the error.
parameterstring
A string indicating which URI query parameter caused the error.
pointerstring
A JSON pointer to the value in the request document that caused the error.
statusstring
Status code of the response.
titlestring
Short human-readable summary of the error.
{
"errors": [
{
"detail": "Missing required attribute in body",
"meta": {},
"source": {
"header": "Authorization",
"parameter": "limit",
"pointer": "/data/attributes/title"
},
"status": "400",
"title": "Bad Request"
}
]
}Too many requests
API error response.
항목
유형
설명
errors [required][string]
A list of errors.
{
"errors": [
"Bad Request"
]
}코드 예시
# Path parameters
export experiment_id="3fd6b5e0-8910-4b1c-a7d0-5b84de329012"
# Set your Datadog site and credentials
export DD_SITE="datadoghq.com"
# Use a Personal Access Token or Service Access Token
export DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>"
curl -X POST "https://api.${DD_SITE}/api/v2/llm-obs/v1/experiments/${experiment_id}/events" \
-H "Accept: application/json" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer ${DD_BEARER_TOKEN}" \
-d @- << EOF
{
"data": {
"attributes": {
"metrics": [
{
"assessment": "pass",
"label": "faithfulness",
"metric_type": "score",
"span_id": "span-7a1b2c3d",
"timestamp_ms": 1705314600000
}
],
"spans": [
{
"dataset_id": "9f64e5c7-dc5a-45c8-a17c-1b85f0bec97d",
"duration": 1500000000,
"name": "llm_call",
"project_id": "a33671aa-24fd-4dcd-9b33-a8ec7dde7751",
"span_id": "span-7a1b2c3d",
"start_ns": 1705314600000000000,
"status": "ok",
"trace_id": "abc123def456"
}
]
},
"type": "experiments"
}
}
EOF# Path parameters
export experiment_id="3fd6b5e0-8910-4b1c-a7d0-5b84de329012"
# Set your Datadog site and credentials
export DD_SITE="us3.datadoghq.com"
# Use a Personal Access Token or Service Access Token
export DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>"
curl -X POST "https://api.${DD_SITE}/api/v2/llm-obs/v1/experiments/${experiment_id}/events" \
-H "Accept: application/json" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer ${DD_BEARER_TOKEN}" \
-d @- << EOF
{
"data": {
"attributes": {
"metrics": [
{
"assessment": "pass",
"label": "faithfulness",
"metric_type": "score",
"span_id": "span-7a1b2c3d",
"timestamp_ms": 1705314600000
}
],
"spans": [
{
"dataset_id": "9f64e5c7-dc5a-45c8-a17c-1b85f0bec97d",
"duration": 1500000000,
"name": "llm_call",
"project_id": "a33671aa-24fd-4dcd-9b33-a8ec7dde7751",
"span_id": "span-7a1b2c3d",
"start_ns": 1705314600000000000,
"status": "ok",
"trace_id": "abc123def456"
}
]
},
"type": "experiments"
}
}
EOF# Path parameters
export experiment_id="3fd6b5e0-8910-4b1c-a7d0-5b84de329012"
# Set your Datadog site and credentials
export DD_SITE="us5.datadoghq.com"
# Use a Personal Access Token or Service Access Token
export DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>"
curl -X POST "https://api.${DD_SITE}/api/v2/llm-obs/v1/experiments/${experiment_id}/events" \
-H "Accept: application/json" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer ${DD_BEARER_TOKEN}" \
-d @- << EOF
{
"data": {
"attributes": {
"metrics": [
{
"assessment": "pass",
"label": "faithfulness",
"metric_type": "score",
"span_id": "span-7a1b2c3d",
"timestamp_ms": 1705314600000
}
],
"spans": [
{
"dataset_id": "9f64e5c7-dc5a-45c8-a17c-1b85f0bec97d",
"duration": 1500000000,
"name": "llm_call",
"project_id": "a33671aa-24fd-4dcd-9b33-a8ec7dde7751",
"span_id": "span-7a1b2c3d",
"start_ns": 1705314600000000000,
"status": "ok",
"trace_id": "abc123def456"
}
]
},
"type": "experiments"
}
}
EOF# Path parameters
export experiment_id="3fd6b5e0-8910-4b1c-a7d0-5b84de329012"
# Set your Datadog site and credentials
export DD_SITE="datadoghq.eu"
# Use a Personal Access Token or Service Access Token
export DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>"
curl -X POST "https://api.${DD_SITE}/api/v2/llm-obs/v1/experiments/${experiment_id}/events" \
-H "Accept: application/json" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer ${DD_BEARER_TOKEN}" \
-d @- << EOF
{
"data": {
"attributes": {
"metrics": [
{
"assessment": "pass",
"label": "faithfulness",
"metric_type": "score",
"span_id": "span-7a1b2c3d",
"timestamp_ms": 1705314600000
}
],
"spans": [
{
"dataset_id": "9f64e5c7-dc5a-45c8-a17c-1b85f0bec97d",
"duration": 1500000000,
"name": "llm_call",
"project_id": "a33671aa-24fd-4dcd-9b33-a8ec7dde7751",
"span_id": "span-7a1b2c3d",
"start_ns": 1705314600000000000,
"status": "ok",
"trace_id": "abc123def456"
}
]
},
"type": "experiments"
}
}
EOF# Path parameters
export experiment_id="3fd6b5e0-8910-4b1c-a7d0-5b84de329012"
# Set your Datadog site and credentials
export DD_SITE="ap1.datadoghq.com"
# Use a Personal Access Token or Service Access Token
export DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>"
curl -X POST "https://api.${DD_SITE}/api/v2/llm-obs/v1/experiments/${experiment_id}/events" \
-H "Accept: application/json" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer ${DD_BEARER_TOKEN}" \
-d @- << EOF
{
"data": {
"attributes": {
"metrics": [
{
"assessment": "pass",
"label": "faithfulness",
"metric_type": "score",
"span_id": "span-7a1b2c3d",
"timestamp_ms": 1705314600000
}
],
"spans": [
{
"dataset_id": "9f64e5c7-dc5a-45c8-a17c-1b85f0bec97d",
"duration": 1500000000,
"name": "llm_call",
"project_id": "a33671aa-24fd-4dcd-9b33-a8ec7dde7751",
"span_id": "span-7a1b2c3d",
"start_ns": 1705314600000000000,
"status": "ok",
"trace_id": "abc123def456"
}
]
},
"type": "experiments"
}
}
EOF# Path parameters
export experiment_id="3fd6b5e0-8910-4b1c-a7d0-5b84de329012"
# Set your Datadog site and credentials
export DD_SITE="ap2.datadoghq.com"
# Use a Personal Access Token or Service Access Token
export DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>"
curl -X POST "https://api.${DD_SITE}/api/v2/llm-obs/v1/experiments/${experiment_id}/events" \
-H "Accept: application/json" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer ${DD_BEARER_TOKEN}" \
-d @- << EOF
{
"data": {
"attributes": {
"metrics": [
{
"assessment": "pass",
"label": "faithfulness",
"metric_type": "score",
"span_id": "span-7a1b2c3d",
"timestamp_ms": 1705314600000
}
],
"spans": [
{
"dataset_id": "9f64e5c7-dc5a-45c8-a17c-1b85f0bec97d",
"duration": 1500000000,
"name": "llm_call",
"project_id": "a33671aa-24fd-4dcd-9b33-a8ec7dde7751",
"span_id": "span-7a1b2c3d",
"start_ns": 1705314600000000000,
"status": "ok",
"trace_id": "abc123def456"
}
]
},
"type": "experiments"
}
}
EOF# Path parameters
export experiment_id="3fd6b5e0-8910-4b1c-a7d0-5b84de329012"
# Set your Datadog site and credentials
export DD_SITE="uk1.datadoghq.com"
# Use a Personal Access Token or Service Access Token
export DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>"
curl -X POST "https://api.${DD_SITE}/api/v2/llm-obs/v1/experiments/${experiment_id}/events" \
-H "Accept: application/json" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer ${DD_BEARER_TOKEN}" \
-d @- << EOF
{
"data": {
"attributes": {
"metrics": [
{
"assessment": "pass",
"label": "faithfulness",
"metric_type": "score",
"span_id": "span-7a1b2c3d",
"timestamp_ms": 1705314600000
}
],
"spans": [
{
"dataset_id": "9f64e5c7-dc5a-45c8-a17c-1b85f0bec97d",
"duration": 1500000000,
"name": "llm_call",
"project_id": "a33671aa-24fd-4dcd-9b33-a8ec7dde7751",
"span_id": "span-7a1b2c3d",
"start_ns": 1705314600000000000,
"status": "ok",
"trace_id": "abc123def456"
}
]
},
"type": "experiments"
}
}
EOF# Path parameters
export experiment_id="3fd6b5e0-8910-4b1c-a7d0-5b84de329012"
# Set your Datadog site and credentials
export DD_SITE="ddog-gov.com"
# Use a Personal Access Token or Service Access Token
export DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>"
curl -X POST "https://api.${DD_SITE}/api/v2/llm-obs/v1/experiments/${experiment_id}/events" \
-H "Accept: application/json" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer ${DD_BEARER_TOKEN}" \
-d @- << EOF
{
"data": {
"attributes": {
"metrics": [
{
"assessment": "pass",
"label": "faithfulness",
"metric_type": "score",
"span_id": "span-7a1b2c3d",
"timestamp_ms": 1705314600000
}
],
"spans": [
{
"dataset_id": "9f64e5c7-dc5a-45c8-a17c-1b85f0bec97d",
"duration": 1500000000,
"name": "llm_call",
"project_id": "a33671aa-24fd-4dcd-9b33-a8ec7dde7751",
"span_id": "span-7a1b2c3d",
"start_ns": 1705314600000000000,
"status": "ok",
"trace_id": "abc123def456"
}
]
},
"type": "experiments"
}
}
EOF# Path parameters
export experiment_id="3fd6b5e0-8910-4b1c-a7d0-5b84de329012"
# Set your Datadog site and credentials
export DD_SITE="us2.ddog-gov.com"
# Use a Personal Access Token or Service Access Token
export DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>"
curl -X POST "https://api.${DD_SITE}/api/v2/llm-obs/v1/experiments/${experiment_id}/events" \
-H "Accept: application/json" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer ${DD_BEARER_TOKEN}" \
-d @- << EOF
{
"data": {
"attributes": {
"metrics": [
{
"assessment": "pass",
"label": "faithfulness",
"metric_type": "score",
"span_id": "span-7a1b2c3d",
"timestamp_ms": 1705314600000
}
],
"spans": [
{
"dataset_id": "9f64e5c7-dc5a-45c8-a17c-1b85f0bec97d",
"duration": 1500000000,
"name": "llm_call",
"project_id": "a33671aa-24fd-4dcd-9b33-a8ec7dde7751",
"span_id": "span-7a1b2c3d",
"start_ns": 1705314600000000000,
"status": "ok",
"trace_id": "abc123def456"
}
]
},
"type": "experiments"
}
}
EOF// Push events for an Agent Observability experiment returns "Accepted" response
package main
import (
"context"
"fmt"
"os"
"github.com/DataDog/datadog-api-client-go/v2/api/datadog"
"github.com/DataDog/datadog-api-client-go/v2/api/datadogV2"
)
func main() {
body := datadogV2.LLMObsExperimentEventsRequest{
Data: datadogV2.LLMObsExperimentEventsDataRequest{
Attributes: datadogV2.LLMObsExperimentEventsDataAttributesRequest{
Metrics: []datadogV2.LLMObsExperimentMetric{
{
Assessment: datadogV2.LLMOBSMETRICASSESSMENT_PASS.Ptr(),
Error: &datadogV2.LLMObsExperimentMetricError{},
Label: "faithfulness",
MetricType: datadogV2.LLMOBSMETRICSCORETYPE_SCORE,
SpanId: "span-7a1b2c3d",
Tags: []string{},
TimestampMs: 1705314600000,
},
},
Spans: []datadogV2.LLMObsExperimentSpan{
{
DatasetId: "9f64e5c7-dc5a-45c8-a17c-1b85f0bec97d",
Duration: 1500000000,
Meta: &datadogV2.LLMObsExperimentSpanMeta{
Error: &datadogV2.LLMObsExperimentSpanError{
Message: datadog.PtrString("Model response timed out"),
Stack: datadog.PtrString(`Traceback (most recent call last):
File "main.py", line 10, in <module>
response = model.generate(input)
File "model.py", line 45, in generate
raise TimeoutError("Model response timed out")
TimeoutError: Model response timed out`),
Type: datadog.PtrString("TimeoutError"),
},
Input: *datadogV2.NewNullableAnyValue(nil),
Output: *datadogV2.NewNullableAnyValue(nil),
},
Name: "llm_call",
ProjectId: "a33671aa-24fd-4dcd-9b33-a8ec7dde7751",
SpanId: "span-7a1b2c3d",
StartNs: 1705314600000000000,
Status: datadogV2.LLMOBSEXPERIMENTSPANSTATUS_OK,
Tags: []string{},
TraceId: "abc123def456",
},
},
},
Type: datadogV2.LLMOBSEVENTTYPE_EVENTS,
},
}
ctx := datadog.NewDefaultContext(context.Background())
ctx = context.WithValue(ctx, datadog.ContextAccessToken, os.Getenv("DD_BEARER_TOKEN"))
configuration := datadog.NewConfiguration()
configuration.SetUnstableOperationEnabled("v2.CreateLLMObsExperimentEvents", true)
apiClient := datadog.NewAPIClient(configuration)
api := datadogV2.NewAgentObservabilityApi(apiClient)
r, err := api.CreateLLMObsExperimentEvents(ctx, "experiment_id", body)
if err != nil {
fmt.Fprintf(os.Stderr, "Error when calling `AgentObservabilityApi.CreateLLMObsExperimentEvents`: %v\n", err)
fmt.Fprintf(os.Stderr, "Full HTTP response: %v\n", r)
}
}
Instructions
First install the library and its dependencies and then save the example to main.go and run following commands:
DD_SITE="datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" go run "main.go"DD_SITE="us3.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" go run "main.go"DD_SITE="us5.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" go run "main.go"DD_SITE="datadoghq.eu" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" go run "main.go"DD_SITE="ap1.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" go run "main.go"DD_SITE="ap2.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" go run "main.go"DD_SITE="uk1.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" go run "main.go"DD_SITE="ddog-gov.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" go run "main.go"DD_SITE="us2.ddog-gov.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" go run "main.go"// Push events for an Agent Observability experiment returns "Accepted" response
import com.datadog.api.client.ApiClient;
import com.datadog.api.client.ApiException;
import com.datadog.api.client.v2.api.AgentObservabilityApi;
import com.datadog.api.client.v2.model.LLMObsEventType;
import com.datadog.api.client.v2.model.LLMObsExperimentEventsDataAttributesRequest;
import com.datadog.api.client.v2.model.LLMObsExperimentEventsDataRequest;
import com.datadog.api.client.v2.model.LLMObsExperimentEventsRequest;
import com.datadog.api.client.v2.model.LLMObsExperimentMetric;
import com.datadog.api.client.v2.model.LLMObsExperimentMetricError;
import com.datadog.api.client.v2.model.LLMObsExperimentSpan;
import com.datadog.api.client.v2.model.LLMObsExperimentSpanError;
import com.datadog.api.client.v2.model.LLMObsExperimentSpanMeta;
import com.datadog.api.client.v2.model.LLMObsExperimentSpanStatus;
import com.datadog.api.client.v2.model.LLMObsMetricAssessment;
import com.datadog.api.client.v2.model.LLMObsMetricScoreType;
import java.util.Collections;
public class Example {
public static void main(String[] args) {
ApiClient defaultClient = ApiClient.getDefaultApiClient();
defaultClient.setAccessToken(System.getenv("DD_BEARER_TOKEN"));
defaultClient.setUnstableOperationEnabled("v2.createLLMObsExperimentEvents", true);
AgentObservabilityApi apiInstance = new AgentObservabilityApi(defaultClient);
LLMObsExperimentEventsRequest body =
new LLMObsExperimentEventsRequest()
.data(
new LLMObsExperimentEventsDataRequest()
.attributes(
new LLMObsExperimentEventsDataAttributesRequest()
.metrics(
Collections.singletonList(
new LLMObsExperimentMetric()
.assessment(LLMObsMetricAssessment.PASS)
.error(new LLMObsExperimentMetricError())
.label("faithfulness")
.metricType(LLMObsMetricScoreType.SCORE)
.spanId("span-7a1b2c3d")
.timestampMs(1705314600000L)))
.spans(
Collections.singletonList(
new LLMObsExperimentSpan()
.datasetId("9f64e5c7-dc5a-45c8-a17c-1b85f0bec97d")
.duration(1500000000L)
.meta(
new LLMObsExperimentSpanMeta()
.error(
new LLMObsExperimentSpanError()
.message("Model response timed out")
.stack(
"""
Traceback (most recent call last):
File "main.py", line 10, in <module>
response = model.generate(input)
File "model.py", line 45, in generate
raise TimeoutError("Model response timed out")
TimeoutError: Model response timed out
""")
.type("TimeoutError"))
.input(null)
.output(null))
.name("llm_call")
.projectId("a33671aa-24fd-4dcd-9b33-a8ec7dde7751")
.spanId("span-7a1b2c3d")
.startNs(1705314600000000000L)
.status(LLMObsExperimentSpanStatus.OK)
.traceId("abc123def456"))))
.type(LLMObsEventType.EVENTS));
try {
apiInstance.createLLMObsExperimentEvents("3fd6b5e0-8910-4b1c-a7d0-5b84de329012", body);
} catch (ApiException e) {
System.err.println(
"Exception when calling AgentObservabilityApi#createLLMObsExperimentEvents");
System.err.println("Status code: " + e.getCode());
System.err.println("Reason: " + e.getResponseBody());
System.err.println("Response headers: " + e.getResponseHeaders());
e.printStackTrace();
}
}
}
Instructions
First install the library and its dependencies and then save the example to Example.java and run following commands:
DD_SITE="datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" java "Example.java"DD_SITE="us3.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" java "Example.java"DD_SITE="us5.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" java "Example.java"DD_SITE="datadoghq.eu" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" java "Example.java"DD_SITE="ap1.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" java "Example.java"DD_SITE="ap2.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" java "Example.java"DD_SITE="uk1.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" java "Example.java"DD_SITE="ddog-gov.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" java "Example.java"DD_SITE="us2.ddog-gov.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" java "Example.java""""
Push events for an Agent Observability experiment returns "Accepted" response
"""
from os import environ
from datadog_api_client import ApiClient, Configuration
from datadog_api_client.v2.api.agent_observability_api import AgentObservabilityApi
from datadog_api_client.v2.model.llm_obs_event_type import LLMObsEventType
from datadog_api_client.v2.model.llm_obs_experiment_events_data_attributes_request import (
LLMObsExperimentEventsDataAttributesRequest,
)
from datadog_api_client.v2.model.llm_obs_experiment_events_data_request import LLMObsExperimentEventsDataRequest
from datadog_api_client.v2.model.llm_obs_experiment_events_request import LLMObsExperimentEventsRequest
from datadog_api_client.v2.model.llm_obs_experiment_metric import LLMObsExperimentMetric
from datadog_api_client.v2.model.llm_obs_experiment_metric_error import LLMObsExperimentMetricError
from datadog_api_client.v2.model.llm_obs_experiment_span import LLMObsExperimentSpan
from datadog_api_client.v2.model.llm_obs_experiment_span_error import LLMObsExperimentSpanError
from datadog_api_client.v2.model.llm_obs_experiment_span_meta import LLMObsExperimentSpanMeta
from datadog_api_client.v2.model.llm_obs_experiment_span_status import LLMObsExperimentSpanStatus
from datadog_api_client.v2.model.llm_obs_metric_assessment import LLMObsMetricAssessment
from datadog_api_client.v2.model.llm_obs_metric_score_type import LLMObsMetricScoreType
body = LLMObsExperimentEventsRequest(
data=LLMObsExperimentEventsDataRequest(
attributes=LLMObsExperimentEventsDataAttributesRequest(
metrics=[
LLMObsExperimentMetric(
assessment=LLMObsMetricAssessment.PASS,
error=LLMObsExperimentMetricError(),
label="faithfulness",
metric_type=LLMObsMetricScoreType.SCORE,
span_id="span-7a1b2c3d",
tags=[],
timestamp_ms=1705314600000,
),
],
spans=[
LLMObsExperimentSpan(
dataset_id="9f64e5c7-dc5a-45c8-a17c-1b85f0bec97d",
duration=1500000000,
meta=LLMObsExperimentSpanMeta(
error=LLMObsExperimentSpanError(
message="Model response timed out",
stack='Traceback (most recent call last):\n File "main.py", line 10, in <module>\n response = model.generate(input)\n File "model.py", line 45, in generate\n raise TimeoutError("Model response timed out")\nTimeoutError: Model response timed out',
type="TimeoutError",
),
input=None,
output=None,
),
name="llm_call",
project_id="a33671aa-24fd-4dcd-9b33-a8ec7dde7751",
span_id="span-7a1b2c3d",
start_ns=1705314600000000000,
status=LLMObsExperimentSpanStatus.OK,
tags=[],
trace_id="abc123def456",
),
],
),
type=LLMObsEventType.EVENTS,
),
)
configuration = Configuration()
configuration.access_token = environ["DD_BEARER_TOKEN"]
configuration.unstable_operations["create_llm_obs_experiment_events"] = True
with ApiClient(configuration) as api_client:
api_instance = AgentObservabilityApi(api_client)
api_instance.create_llm_obs_experiment_events(experiment_id="experiment_id", body=body)
Instructions
First install the library and its dependencies and then save the example to example.py and run following commands:
DD_SITE="datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" python3 "example.py"DD_SITE="us3.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" python3 "example.py"DD_SITE="us5.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" python3 "example.py"DD_SITE="datadoghq.eu" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" python3 "example.py"DD_SITE="ap1.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" python3 "example.py"DD_SITE="ap2.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" python3 "example.py"DD_SITE="uk1.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" python3 "example.py"DD_SITE="ddog-gov.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" python3 "example.py"DD_SITE="us2.ddog-gov.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" python3 "example.py"# Push events for an Agent Observability experiment returns "Accepted" response
require "datadog_api_client"
DatadogAPIClient.configure do |config|
config.access_token = ENV["DD_BEARER_TOKEN"]
config.unstable_operations["v2.create_llm_obs_experiment_events".to_sym] = true
end
api_instance = DatadogAPIClient::V2::AgentObservabilityAPI.new
body = DatadogAPIClient::V2::LLMObsExperimentEventsRequest.new({
data: DatadogAPIClient::V2::LLMObsExperimentEventsDataRequest.new({
attributes: DatadogAPIClient::V2::LLMObsExperimentEventsDataAttributesRequest.new({
metrics: [
DatadogAPIClient::V2::LLMObsExperimentMetric.new({
assessment: DatadogAPIClient::V2::LLMObsMetricAssessment::PASS,
error: DatadogAPIClient::V2::LLMObsExperimentMetricError.new({}),
label: "faithfulness",
metric_type: DatadogAPIClient::V2::LLMObsMetricScoreType::SCORE,
span_id: "span-7a1b2c3d",
tags: [],
timestamp_ms: 1705314600000,
}),
],
spans: [
DatadogAPIClient::V2::LLMObsExperimentSpan.new({
dataset_id: "9f64e5c7-dc5a-45c8-a17c-1b85f0bec97d",
duration: 1500000000,
meta: DatadogAPIClient::V2::LLMObsExperimentSpanMeta.new({
error: DatadogAPIClient::V2::LLMObsExperimentSpanError.new({
message: "Model response timed out",
stack: 'Traceback (most recent call last):\n File "main.py", line 10, in <module>\n response = model.generate(input)\n File "model.py", line 45, in generate\n raise TimeoutError("Model response timed out")\nTimeoutError: Model response timed out',
type: "TimeoutError",
}),
input: nil,
output: nil,
}),
name: "llm_call",
project_id: "a33671aa-24fd-4dcd-9b33-a8ec7dde7751",
span_id: "span-7a1b2c3d",
start_ns: 1705314600000000000,
status: DatadogAPIClient::V2::LLMObsExperimentSpanStatus::OK,
tags: [],
trace_id: "abc123def456",
}),
],
}),
type: DatadogAPIClient::V2::LLMObsEventType::EVENTS,
}),
})
p api_instance.create_llm_obs_experiment_events("experiment_id", body)
Instructions
First install the library and its dependencies and then save the example to example.rb and run following commands:
DD_SITE="datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" rb "example.rb"DD_SITE="us3.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" rb "example.rb"DD_SITE="us5.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" rb "example.rb"DD_SITE="datadoghq.eu" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" rb "example.rb"DD_SITE="ap1.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" rb "example.rb"DD_SITE="ap2.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" rb "example.rb"DD_SITE="uk1.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" rb "example.rb"DD_SITE="ddog-gov.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" rb "example.rb"DD_SITE="us2.ddog-gov.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" rb "example.rb"// Push events for an Agent Observability experiment returns "Accepted" response
use datadog_api_client::datadog;
use datadog_api_client::datadogV2::api_agent_observability::AgentObservabilityAPI;
use datadog_api_client::datadogV2::model::LLMObsEventType;
use datadog_api_client::datadogV2::model::LLMObsExperimentEventsDataAttributesRequest;
use datadog_api_client::datadogV2::model::LLMObsExperimentEventsDataRequest;
use datadog_api_client::datadogV2::model::LLMObsExperimentEventsRequest;
use datadog_api_client::datadogV2::model::LLMObsExperimentMetric;
use datadog_api_client::datadogV2::model::LLMObsExperimentMetricError;
use datadog_api_client::datadogV2::model::LLMObsExperimentSpan;
use datadog_api_client::datadogV2::model::LLMObsExperimentSpanError;
use datadog_api_client::datadogV2::model::LLMObsExperimentSpanMeta;
use datadog_api_client::datadogV2::model::LLMObsExperimentSpanStatus;
use datadog_api_client::datadogV2::model::LLMObsMetricAssessment;
use datadog_api_client::datadogV2::model::LLMObsMetricScoreType;
#[tokio::main]
async fn main() {
let body = LLMObsExperimentEventsRequest::new(LLMObsExperimentEventsDataRequest::new(
LLMObsExperimentEventsDataAttributesRequest::new()
.metrics(vec![LLMObsExperimentMetric::new(
"faithfulness".to_string(),
LLMObsMetricScoreType::SCORE,
"span-7a1b2c3d".to_string(),
1705314600000,
)
.assessment(LLMObsMetricAssessment::PASS)
.error(LLMObsExperimentMetricError::new())
.tags(vec![])])
.spans(vec![LLMObsExperimentSpan::new(
"9f64e5c7-dc5a-45c8-a17c-1b85f0bec97d".to_string(),
1500000000,
"llm_call".to_string(),
"a33671aa-24fd-4dcd-9b33-a8ec7dde7751".to_string(),
"span-7a1b2c3d".to_string(),
1705314600000000000,
LLMObsExperimentSpanStatus::OK,
"abc123def456".to_string(),
)
.meta(
LLMObsExperimentSpanMeta::new()
.error(
LLMObsExperimentSpanError::new()
.message("Model response timed out".to_string())
.stack(
r#"Traceback (most recent call last):
File "main.py", line 10, in <module>
response = model.generate(input)
File "model.py", line 45, in generate
raise TimeoutError("Model response timed out")
TimeoutError: Model response timed out"#
.to_string(),
)
.type_("TimeoutError".to_string()),
)
.input(None)
.output(None),
)
.tags(vec![])]),
LLMObsEventType::EVENTS,
));
let mut configuration = datadog::Configuration::new();
configuration.set_unstable_operation_enabled("v2.create_llm_obs_experiment_events", true);
let api = AgentObservabilityAPI::with_config(configuration);
let resp = api
.create_llm_obs_experiment_events("experiment_id".to_string(), body)
.await;
if let Ok(value) = resp {
println!("{:#?}", value);
} else {
println!("{:#?}", resp.unwrap_err());
}
}
Instructions
First install the library and its dependencies and then save the example to src/main.rs and run following commands:
DD_SITE="datadoghq.com" DD_API_KEY="<DD_API_KEY>" DD_APP_KEY="<DD_APP_KEY>" cargo runDD_SITE="us3.datadoghq.com" DD_API_KEY="<DD_API_KEY>" DD_APP_KEY="<DD_APP_KEY>" cargo runDD_SITE="us5.datadoghq.com" DD_API_KEY="<DD_API_KEY>" DD_APP_KEY="<DD_APP_KEY>" cargo runDD_SITE="datadoghq.eu" DD_API_KEY="<DD_API_KEY>" DD_APP_KEY="<DD_APP_KEY>" cargo runDD_SITE="ap1.datadoghq.com" DD_API_KEY="<DD_API_KEY>" DD_APP_KEY="<DD_APP_KEY>" cargo runDD_SITE="ap2.datadoghq.com" DD_API_KEY="<DD_API_KEY>" DD_APP_KEY="<DD_APP_KEY>" cargo runDD_SITE="uk1.datadoghq.com" DD_API_KEY="<DD_API_KEY>" DD_APP_KEY="<DD_APP_KEY>" cargo runDD_SITE="ddog-gov.com" DD_API_KEY="<DD_API_KEY>" DD_APP_KEY="<DD_APP_KEY>" cargo runDD_SITE="us2.ddog-gov.com" DD_API_KEY="<DD_API_KEY>" DD_APP_KEY="<DD_APP_KEY>" cargo run/**
* Push events for an Agent Observability experiment returns "Accepted" response
*/
import { client, v2 } from "@datadog/datadog-api-client";
const configuration = client.createConfiguration({
authMethods: {
AuthZ: {
accessToken: process.env.DD_BEARER_TOKEN as string,
},
},
});
configuration.unstableOperations["v2.createLLMObsExperimentEvents"] = true;
const apiInstance = new v2.AgentObservabilityApi(configuration);
const params: v2.AgentObservabilityApiCreateLLMObsExperimentEventsRequest = {
body: {
data: {
attributes: {
metrics: [
{
assessment: "pass",
error: {},
label: "faithfulness",
metricType: "score",
spanId: "span-7a1b2c3d",
tags: [],
timestampMs: 1705314600000,
},
],
spans: [
{
datasetId: "9f64e5c7-dc5a-45c8-a17c-1b85f0bec97d",
duration: 1500000000,
meta: {
error: {
message: "Model response timed out",
stack: `Traceback (most recent call last):
File "main.py", line 10, in <module>
response = model.generate(input)
File "model.py", line 45, in generate
raise TimeoutError("Model response timed out")
TimeoutError: Model response timed out`,
type: "TimeoutError",
},
input: undefined,
output: undefined,
},
name: "llm_call",
projectId: "a33671aa-24fd-4dcd-9b33-a8ec7dde7751",
spanId: "span-7a1b2c3d",
startNs: 1705314600000000000,
status: "ok",
tags: [],
traceId: "abc123def456",
},
],
},
type: "events",
},
},
experimentId: "experiment_id",
};
apiInstance
.createLLMObsExperimentEvents(params)
.then((data: any) => {
console.log(
"API called successfully. Returned data: " + JSON.stringify(data)
);
})
.catch((error: any) => console.error(error));
Instructions
First install the library and its dependencies and then save the example to example.ts and run following commands:
DD_SITE="datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" tsc "example.ts"DD_SITE="us3.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" tsc "example.ts"DD_SITE="us5.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" tsc "example.ts"DD_SITE="datadoghq.eu" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" tsc "example.ts"DD_SITE="ap1.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" tsc "example.ts"DD_SITE="ap2.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" tsc "example.ts"DD_SITE="uk1.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" tsc "example.ts"DD_SITE="ddog-gov.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" tsc "example.ts"DD_SITE="us2.ddog-gov.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" tsc "example.ts"