Get SPA Recommendations
https://api.datadoghq.com/api/v2/spa/recommendations/{service}https://api.us3.datadoghq.com/api/v2/spa/recommendations/{service}https://api.us5.datadoghq.com/api/v2/spa/recommendations/{service}https://api.datadoghq.eu/api/v2/spa/recommendations/{service}https://api.ap1.datadoghq.com/api/v2/spa/recommendations/{service}https://api.ap2.datadoghq.com/api/v2/spa/recommendations/{service}https://api.uk1.datadoghq.com/api/v2/spa/recommendations/{service}https://api.ddog-gov.com/api/v2/spa/recommendations/{service}https://api.us2.ddog-gov.com/api/v2/spa/recommendations/{service}Overview
This endpoint is currently experimental and restricted to Datadog internal use only. Retrieve resource recommendations for a Spark job. The caller (Spark Gateway or DJM UI) provides a service name and SPA returns structured recommendations for driver and executor resources. The version with a shard should be preferred, where possible, as it gives more accurate results.
Arguments
Path Parameters
service [required]The service name for a spark job.
Query Strings
bypass_cacheThe recommendation service should not use its metrics cache.
Response
OK
JSON:API document containing a single Recommendation resource. Returned by SPA when the Spark Gateway requests recommendations.
data [required]JSON:API resource object for SPA Recommendation. Includes type, optional ID, and resource attributes with structured recommendations.
attributes [required]Attributes of the SPA Recommendation resource. Contains recommendations for both driver and executor components.
confidence_levelThe confidence level of the recommendation, expressed as a value between 0.0 (low confidence) and 1.0 (high confidence).
driver [required]Resource recommendation for a single Spark component (driver or executor). Contains estimation data used to patch Spark job specs.
estimation [required]Recommended resource values for a Spark driver or executor, derived from recent real usage metrics. Used by SPA to propose more efficient pod sizing.
cpuCPU usage statistics derived from historical Spark job metrics. Provides multiple estimates so users can choose between conservative and cost-saving risk profiles.
maxMaximum CPU usage observed for the job, expressed in millicores. This represents the upper bound of usage.
p7575th percentile of CPU usage (millicores). Represents a cost-saving configuration while covering most workloads.
p9595th percentile of CPU usage (millicores). Balances performance and cost, providing a safer margin than p75.
ephemeral_storageRecommended ephemeral storage allocation (in MiB). Derived from job temporary storage patterns.
heapRecommended JVM heap size (in MiB).
memoryRecommended total memory allocation (in MiB). Includes both heap and overhead.
overheadRecommended JVM overhead (in MiB). Computed as total memory - heap.
executor [required]Resource recommendation for a single Spark component (driver or executor). Contains estimation data used to patch Spark job specs.
estimation [required]Recommended resource values for a Spark driver or executor, derived from recent real usage metrics. Used by SPA to propose more efficient pod sizing.
cpuCPU usage statistics derived from historical Spark job metrics. Provides multiple estimates so users can choose between conservative and cost-saving risk profiles.
maxMaximum CPU usage observed for the job, expressed in millicores. This represents the upper bound of usage.
p7575th percentile of CPU usage (millicores). Represents a cost-saving configuration while covering most workloads.
p9595th percentile of CPU usage (millicores). Balances performance and cost, providing a safer margin than p75.
ephemeral_storageRecommended ephemeral storage allocation (in MiB). Derived from job temporary storage patterns.
heapRecommended JVM heap size (in MiB).
memoryRecommended total memory allocation (in MiB). Includes both heap and overhead.
overheadRecommended JVM overhead (in MiB). Computed as total memory - heap.
matched_paramsOnly returned by the v2 endpoint. The job parameters whose values the recommendation was matched on, as parameter=value pairs joined by |.
An empty string means the service-wide (coarse) recommendation was used.
idResource identifier for the recommendation. Optional in responses.
type [required]JSON:API resource type for Spark Pod Autosizing recommendations. Identifies the Recommendation resource returned by SPA.
recommendationrecommendation{
"data": {
"attributes": {
"confidence_level": "number",
"driver": {
"estimation": {
"cpu": {
"max": "integer",
"p75": "integer",
"p95": "integer"
},
"ephemeral_storage": "integer",
"heap": "integer",
"memory": "integer",
"overhead": "integer"
}
},
"executor": {
"estimation": {
"cpu": {
"max": "integer",
"p75": "integer",
"p95": "integer"
},
"ephemeral_storage": "integer",
"heap": "integer",
"memory": "integer",
"overhead": "integer"
}
},
"matched_params": "detailed_metrics=false|Region=US1"
},
"id": "string",
"type": "recommendation"
}
}Bad Request
API error response.
errors [required]A list of errors.
{
"errors": [
"Bad Request"
]
}Not Authorized
API error response.
errors [required]A list of errors.
{
"errors": [
"Bad Request"
]
}Too many requests
API error response.
errors [required]A list of errors.
{
"errors": [
"Bad Request"
]
}Code Example
# Path parameters
export service="CHANGE_ME"
# Set your Datadog site and credentials
export DD_SITE="datadoghq.com"
# Use a Personal Access Token or Service Access Token
export DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>"
curl -X GET "https://api.${DD_SITE}/api/v2/spa/recommendations/${service}" \
-H "Accept: application/json" \
-H "Authorization: Bearer ${DD_BEARER_TOKEN}"# Path parameters
export service="CHANGE_ME"
# Set your Datadog site and credentials
export DD_SITE="us3.datadoghq.com"
# Use a Personal Access Token or Service Access Token
export DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>"
curl -X GET "https://api.${DD_SITE}/api/v2/spa/recommendations/${service}" \
-H "Accept: application/json" \
-H "Authorization: Bearer ${DD_BEARER_TOKEN}"# Path parameters
export service="CHANGE_ME"
# Set your Datadog site and credentials
export DD_SITE="us5.datadoghq.com"
# Use a Personal Access Token or Service Access Token
export DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>"
curl -X GET "https://api.${DD_SITE}/api/v2/spa/recommendations/${service}" \
-H "Accept: application/json" \
-H "Authorization: Bearer ${DD_BEARER_TOKEN}"# Path parameters
export service="CHANGE_ME"
# Set your Datadog site and credentials
export DD_SITE="datadoghq.eu"
# Use a Personal Access Token or Service Access Token
export DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>"
curl -X GET "https://api.${DD_SITE}/api/v2/spa/recommendations/${service}" \
-H "Accept: application/json" \
-H "Authorization: Bearer ${DD_BEARER_TOKEN}"# Path parameters
export service="CHANGE_ME"
# Set your Datadog site and credentials
export DD_SITE="ap1.datadoghq.com"
# Use a Personal Access Token or Service Access Token
export DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>"
curl -X GET "https://api.${DD_SITE}/api/v2/spa/recommendations/${service}" \
-H "Accept: application/json" \
-H "Authorization: Bearer ${DD_BEARER_TOKEN}"# Path parameters
export service="CHANGE_ME"
# Set your Datadog site and credentials
export DD_SITE="ap2.datadoghq.com"
# Use a Personal Access Token or Service Access Token
export DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>"
curl -X GET "https://api.${DD_SITE}/api/v2/spa/recommendations/${service}" \
-H "Accept: application/json" \
-H "Authorization: Bearer ${DD_BEARER_TOKEN}"# Path parameters
export service="CHANGE_ME"
# Set your Datadog site and credentials
export DD_SITE="uk1.datadoghq.com"
# Use a Personal Access Token or Service Access Token
export DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>"
curl -X GET "https://api.${DD_SITE}/api/v2/spa/recommendations/${service}" \
-H "Accept: application/json" \
-H "Authorization: Bearer ${DD_BEARER_TOKEN}"# Path parameters
export service="CHANGE_ME"
# Set your Datadog site and credentials
export DD_SITE="ddog-gov.com"
# Use a Personal Access Token or Service Access Token
export DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>"
curl -X GET "https://api.${DD_SITE}/api/v2/spa/recommendations/${service}" \
-H "Accept: application/json" \
-H "Authorization: Bearer ${DD_BEARER_TOKEN}"# Path parameters
export service="CHANGE_ME"
# Set your Datadog site and credentials
export DD_SITE="us2.ddog-gov.com"
# Use a Personal Access Token or Service Access Token
export DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>"
curl -X GET "https://api.${DD_SITE}/api/v2/spa/recommendations/${service}" \
-H "Accept: application/json" \
-H "Authorization: Bearer ${DD_BEARER_TOKEN}"// Get SPA Recommendations returns "OK" response
package main
import (
"context"
"encoding/json"
"fmt"
"os"
"github.com/DataDog/datadog-api-client-go/v2/api/datadog"
"github.com/DataDog/datadog-api-client-go/v2/api/datadogV2"
)
func main() {
ctx := datadog.NewDefaultContext(context.Background())
ctx = context.WithValue(ctx, datadog.ContextAccessToken, os.Getenv("DD_BEARER_TOKEN"))
configuration := datadog.NewConfiguration()
configuration.SetUnstableOperationEnabled("v2.GetSPARecommendations", true)
apiClient := datadog.NewAPIClient(configuration)
api := datadogV2.NewSpaApi(apiClient)
resp, r, err := api.GetSPARecommendations(ctx, "service", *datadogV2.NewGetSPARecommendationsOptionalParameters())
if err != nil {
fmt.Fprintf(os.Stderr, "Error when calling `SpaApi.GetSPARecommendations`: %v\n", err)
fmt.Fprintf(os.Stderr, "Full HTTP response: %v\n", r)
}
responseContent, _ := json.MarshalIndent(resp, "", " ")
fmt.Fprintf(os.Stdout, "Response from `SpaApi.GetSPARecommendations`:\n%s\n", responseContent)
}
Instructions
First install the library and its dependencies and then save the example to main.go and run following commands:
DD_SITE="datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" go run "main.go"DD_SITE="us3.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" go run "main.go"DD_SITE="us5.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" go run "main.go"DD_SITE="datadoghq.eu" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" go run "main.go"DD_SITE="ap1.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" go run "main.go"DD_SITE="ap2.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" go run "main.go"DD_SITE="uk1.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" go run "main.go"DD_SITE="ddog-gov.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" go run "main.go"DD_SITE="us2.ddog-gov.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" go run "main.go"// Get SPA Recommendations returns "OK" response
import com.datadog.api.client.ApiClient;
import com.datadog.api.client.ApiException;
import com.datadog.api.client.v2.api.SpaApi;
import com.datadog.api.client.v2.model.RecommendationDocument;
public class Example {
public static void main(String[] args) {
ApiClient defaultClient = ApiClient.getDefaultApiClient();
defaultClient.setAccessToken(System.getenv("DD_BEARER_TOKEN"));
defaultClient.setUnstableOperationEnabled("v2.getSPARecommendations", true);
SpaApi apiInstance = new SpaApi(defaultClient);
try {
RecommendationDocument result = apiInstance.getSPARecommendations("service");
System.out.println(result);
} catch (ApiException e) {
System.err.println("Exception when calling SpaApi#getSPARecommendations");
System.err.println("Status code: " + e.getCode());
System.err.println("Reason: " + e.getResponseBody());
System.err.println("Response headers: " + e.getResponseHeaders());
e.printStackTrace();
}
}
}
Instructions
First install the library and its dependencies and then save the example to Example.java and run following commands:
DD_SITE="datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" java "Example.java"DD_SITE="us3.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" java "Example.java"DD_SITE="us5.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" java "Example.java"DD_SITE="datadoghq.eu" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" java "Example.java"DD_SITE="ap1.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" java "Example.java"DD_SITE="ap2.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" java "Example.java"DD_SITE="uk1.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" java "Example.java"DD_SITE="ddog-gov.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" java "Example.java"DD_SITE="us2.ddog-gov.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" java "Example.java""""
Get SPA Recommendations returns "OK" response
"""
from os import environ
from datadog_api_client import ApiClient, Configuration
from datadog_api_client.v2.api.spa_api import SpaApi
configuration = Configuration()
configuration.access_token = environ["DD_BEARER_TOKEN"]
configuration.unstable_operations["get_spa_recommendations"] = True
with ApiClient(configuration) as api_client:
api_instance = SpaApi(api_client)
response = api_instance.get_spa_recommendations(
service="service",
)
print(response)
Instructions
First install the library and its dependencies and then save the example to example.py and run following commands:
DD_SITE="datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" python3 "example.py"DD_SITE="us3.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" python3 "example.py"DD_SITE="us5.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" python3 "example.py"DD_SITE="datadoghq.eu" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" python3 "example.py"DD_SITE="ap1.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" python3 "example.py"DD_SITE="ap2.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" python3 "example.py"DD_SITE="uk1.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" python3 "example.py"DD_SITE="ddog-gov.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" python3 "example.py"DD_SITE="us2.ddog-gov.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" python3 "example.py"# Get SPA Recommendations returns "OK" response
require "datadog_api_client"
DatadogAPIClient.configure do |config|
config.access_token = ENV["DD_BEARER_TOKEN"]
config.unstable_operations["v2.get_spa_recommendations".to_sym] = true
end
api_instance = DatadogAPIClient::V2::SpaAPI.new
p api_instance.get_spa_recommendations("service")
Instructions
First install the library and its dependencies and then save the example to example.rb and run following commands:
DD_SITE="datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" rb "example.rb"DD_SITE="us3.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" rb "example.rb"DD_SITE="us5.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" rb "example.rb"DD_SITE="datadoghq.eu" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" rb "example.rb"DD_SITE="ap1.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" rb "example.rb"DD_SITE="ap2.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" rb "example.rb"DD_SITE="uk1.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" rb "example.rb"DD_SITE="ddog-gov.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" rb "example.rb"DD_SITE="us2.ddog-gov.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" rb "example.rb"// Get SPA Recommendations returns "OK" response
use datadog_api_client::datadog;
use datadog_api_client::datadogV2::api_spa::GetSPARecommendationsOptionalParams;
use datadog_api_client::datadogV2::api_spa::SpaAPI;
#[tokio::main]
async fn main() {
let mut configuration = datadog::Configuration::new();
configuration.set_unstable_operation_enabled("v2.get_spa_recommendations", true);
let api = SpaAPI::with_config(configuration);
let resp = api
.get_spa_recommendations(
"service".to_string(),
GetSPARecommendationsOptionalParams::default(),
)
.await;
if let Ok(value) = resp {
println!("{:#?}", value);
} else {
println!("{:#?}", resp.unwrap_err());
}
}
Instructions
First install the library and its dependencies and then save the example to src/main.rs and run following commands:
DD_SITE="datadoghq.com" cargo runDD_SITE="us3.datadoghq.com" cargo runDD_SITE="us5.datadoghq.com" cargo runDD_SITE="datadoghq.eu" cargo runDD_SITE="ap1.datadoghq.com" cargo runDD_SITE="ap2.datadoghq.com" cargo runDD_SITE="uk1.datadoghq.com" cargo runDD_SITE="ddog-gov.com" cargo runDD_SITE="us2.ddog-gov.com" cargo run/**
* Get SPA Recommendations returns "OK" response
*/
import { client, v2 } from "@datadog/datadog-api-client";
const configuration = client.createConfiguration({
authMethods: {
AuthZ: {
accessToken: process.env.DD_BEARER_TOKEN as string,
},
},
});
configuration.unstableOperations["v2.getSPARecommendations"] = true;
const apiInstance = new v2.SpaApi(configuration);
const params: v2.SpaApiGetSPARecommendationsRequest = {
service: "service",
};
apiInstance
.getSPARecommendations(params)
.then((data: v2.RecommendationDocument) => {
console.log(
"API called successfully. Returned data: " + JSON.stringify(data)
);
})
.catch((error: any) => console.error(error));
Instructions
First install the library and its dependencies and then save the example to example.ts and run following commands:
DD_SITE="datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" tsc "example.ts"DD_SITE="us3.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" tsc "example.ts"DD_SITE="us5.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" tsc "example.ts"DD_SITE="datadoghq.eu" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" tsc "example.ts"DD_SITE="ap1.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" tsc "example.ts"DD_SITE="ap2.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" tsc "example.ts"DD_SITE="uk1.datadoghq.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" tsc "example.ts"DD_SITE="ddog-gov.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" tsc "example.ts"DD_SITE="us2.ddog-gov.com" DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>" tsc "example.ts"