Create Evaluation
Create an evaluation together with its items, optionally running test criteria against them.
Accepts three request shapes: standalone (inline data), from an existing dataset
(dataset_id with optional per-item references), or with a new reusable dataset created inline
from data. When the evaluation includes tasks that require execution (for example an LLM judge
or custom function), an async job and a Temporal workflow are started and the evaluation is
returned immediately with status running; task results and error_count populate
asynchronously. When it includes only contributor tasks, taxonomy-only input, or no tasks, no
workflow runs and it is returned with status completed. Optional tasks, metadata, tags,
and taxonomy_params are persisted alongside the evaluation and its items.
Body ParametersJSONExpand Collapse
evaluation: object { data, name, description, 6 more } or object { dataset_id, name, data, 6 more } or object { data, dataset, name, 7 more }
EvaluationStandaloneCreateRequest object { data, name, description, 6 more }
Tasks allow you to augment and evaluate your data
Tasks allow you to augment and evaluate your data
ChatCompletion object { configuration, alias, task_type }
configuration: object { messages, model, audio, 24 more }
Number between -2.0 and 2.0. Positive values penalize new tokens based on their existing frequency in the text so far.
Number between -2.0 and 2.0. Positive values penalize new tokens based on their existing frequency in the text so far.
Modify the likelihood of specified tokens appearing in the completion. Maps tokens to bias values from -100 to 100.
Modify the likelihood of specified tokens appearing in the completion. Maps tokens to bias values from -100 to 100.
An upper bound for the number of tokens that can be generated, including visible output tokens and reasoning tokens.
An upper bound for the number of tokens that can be generated, including visible output tokens and reasoning tokens.
Number between -2.0 and 2.0. Positive values penalize tokens based on whether they appear in the text so far.
Number between -2.0 and 2.0. Positive values penalize tokens based on whether they appear in the text so far.
For o1 models only. Constrains effort on reasoning. Values: low, medium, high.
stop: optional string or array of stringUp to 4 sequences where the API will stop generating further tokens.
Up to 4 sequences where the API will stop generating further tokens.
tool_choice: optional string or map[unknown]Controls which tool is called by the model. Values: none, auto, required, or specific tool.
Controls which tool is called by the model. Values: none, auto, required, or specific tool.
Inference object { configuration, alias, task_type }
configuration: object { model, args, inference_configuration }
inference_configuration: optional LaunchInferenceConfiguration { num_retries, timeout_seconds } or ItemLocatorVendor specific configuration
Vendor specific configuration
ApplicationVariant object { configuration, alias, task_type }
configuration: object { application_variant_id, inputs, history, 2 more }
Input data for the application. For agents service variants, you must provide inputs as a mapping from {input_name: input_value}. For V0 variants, you must specify the node your input should be passed to, structuring your input as {node_id: {input_name: input_value}}.
Input data for the application. For agents service variants, you must provide inputs as a mapping from {input_name: input_value}. For V0 variants, you must specify the node your input should be passed to, structuring your input as {node_id: {input_name: input_value}}.
Arbitrary user-defined metadata that can be attached to the process operations and will be registered in the interaction.
Arbitrary user-defined metadata that can be attached to the process operations and will be registered in the interaction.
overrides: optional object { concurrent, initial_state, partial_trace, 2 more } or map[object { artifact_ids_filter, artifact_name_regex, type } ] or ItemLocatorOptional overrides for the application
Optional overrides for the application
AgentexOutput object { configuration, alias, task_type }
configuration: object { agentex_agent_id, input_column, agent_task_params, 6 more }
input_column: string or map[unknown] or array of unknownThe dataset column to use as input for the agent
The dataset column to use as input for the agent
Extra params merged into the Agentex task/create call’s params object and forwarded verbatim to the agent. Required by agents that demand configuration at task creation — the golden agent, for example, rejects any task whose params omit config_id. SGP always pins is_eval: true; a caller-supplied description overrides the SGP default. Nested item.-prefixed strings and {{item.x}} templates are resolved per evaluation item, so a per-row config_id can come from a dataset column.
Extra params merged into the Agentex task/create call’s params object and forwarded verbatim to the agent. Required by agents that demand configuration at task creation — the golden agent, for example, rejects any task whose params omit config_id. SGP always pins is_eval: true; a caller-supplied description overrides the SGP default. Nested item.-prefixed strings and {{item.x}} templates are resolved per evaluation item, so a per-row config_id can come from a dataset column.
completion_mode: optional "first_message" or "turn_quiescence"How the agent’s first turn is judged finished. first_message (the default) grades the first non-empty agent text message after the input, which is cheap but grades a streaming harness on whatever text block streamed first. turn_quiescence keeps listening while the agent is still producing messages and grades once at least one agent text message exists and nothing new has arrived for quiescence_seconds — the right choice for tool-using agents. Neither mode requires the agent to mark the task complete; a terminal task status always ends the wait, and timeout_seconds always bounds it.
How the agent’s first turn is judged finished. first_message (the default) grades the first non-empty agent text message after the input, which is cheap but grades a streaming harness on whatever text block streamed first. turn_quiescence keeps listening while the agent is still producing messages and grades once at least one agent text message exists and nothing new has arrived for quiescence_seconds — the right choice for tool-using agents. Neither mode requires the agent to mark the task complete; a terminal task status always ends the wait, and timeout_seconds always bounds it.
Optional Agentex deployment ID to pin the eval to a specific deployment. When set, RPC traffic routes through /agents/{agent_id}/deployments/{deployment_id}/rpc. When unset, traffic uses the agent’s default RPC endpoint, which resolves through the agent’s current routing rules on the Agentex side.
Whether to include trace data in the evaluation results. Traces are read from SGP’s own span store for the agent’s trace, not from Agentex.
Whether to include trace data in the evaluation results. Traces are read from SGP’s own span store for the agent’s trace, not from Agentex.
input_mode: optional "text" or "data"How the resolved input_column is delivered to the agent. text (the default) sends a TextContent message with the value stringified. data sends a DataContent message whose data is the value as a JSON object; the resolved value must be an object, or a string that parses to one. Most agents accept text only and reject data.
How the resolved input_column is delivered to the agent. text (the default) sends a TextContent message with the value stringified. data sends a DataContent message whose data is the value as a JSON object; the resolved value must be an object, or a string that parses to one. Most agents accept text only and reject data.
Seconds of no new messages before completion_mode: turn_quiescence considers the turn finished. Ignored in first_message mode. Should exceed the agent’s longest expected gap between messages (a slow tool call), or the turn is graded early.
Seconds of no new messages before completion_mode: turn_quiescence considers the turn finished. Ignored in first_message mode. Should exceed the agent’s longest expected gap between messages (a slow tool call), or the turn is graded early.
Metric object { configuration, alias, task_type }
configuration: object { candidate, reference, type } or object { candidate, reference, type } or object { candidate, reference, type } or 4 more
AutoEvaluationQuestion object { configuration, alias, task_type }
AutoEvaluationGuidedDecoding object { configuration, alias, task_type }
configuration: object { model, prompt, response_format, 3 more } or object { choices, model, prompt, 3 more } or AutoEvaluationAgentTaskRequestWithItemLocator { definition, name, output_rules, 6 more }
AutoEvaluationStructuredOutputTaskRequestWithItemLocator object { model, prompt, response_format, 3 more }
run_condition: optional object { op, value } or object { path, op } or EqEvaluationRunCondition { left, right, op } or 12 more
AutoEvaluationGuidedDecodingTaskRequestWithItemLocator object { choices, model, prompt, 3 more }
run_condition: optional object { op, value } or object { path, op } or EqEvaluationRunCondition { left, right, op } or 12 more
AutoEvaluationAgentTaskRequestWithItemLocator object { definition, name, output_rules, 6 more }
designated_to: optional object { config, agent_name } or object { config, agent_name } or object { config, agent_name } or object { config, agent_name }
AutoEvaluationAgent object { configuration, alias, task_type }
configuration: AutoEvaluationAgentTaskRequestWithItemLocator { definition, name, output_rules, 6 more }
designated_to: optional object { config, agent_name } or object { config, agent_name } or object { config, agent_name } or object { config, agent_name }
ContributorEvaluationQuestion object { configuration, alias, task_type }
configuration: object { layout, question_id, prefill_from, 3 more }
direction: optional "row" or "column"The axis that children are placed in the container. Based on CSS flex-direction (see: https://developer.mozilla.org/en-US/docs/Web/CSS/flex-direction)
The axis that children are placed in the container. Based on CSS flex-direction (see: https://developer.mozilla.org/en-US/docs/Web/CSS/flex-direction)
CustomFunction object { configuration, alias, task_type }
configuration: object { function_source, arg_mapping, config_args, outputs } Configuration for a custom Python function evaluation task.
Configuration for a custom Python function evaluation task.
Mapping of function parameter names to item locators (e.g. item.field). Auto-derived from function signature if not provided.
Literal argument values for function parameters, such as thresholds or RNG seeds. Serialized JSON must be at most 10000 characters.
EvaluationFromDatasetCreateRequest object { dataset_id, name, data, 6 more }
data: optional array of object { dataset_item_id } Items to be evaluated, including references to the input dataset
Items to be evaluated, including references to the input dataset
Tasks allow you to augment and evaluate your data
Tasks allow you to augment and evaluate your data
ChatCompletion object { configuration, alias, task_type }
configuration: object { messages, model, audio, 24 more }
Number between -2.0 and 2.0. Positive values penalize new tokens based on their existing frequency in the text so far.
Number between -2.0 and 2.0. Positive values penalize new tokens based on their existing frequency in the text so far.
Modify the likelihood of specified tokens appearing in the completion. Maps tokens to bias values from -100 to 100.
Modify the likelihood of specified tokens appearing in the completion. Maps tokens to bias values from -100 to 100.
An upper bound for the number of tokens that can be generated, including visible output tokens and reasoning tokens.
An upper bound for the number of tokens that can be generated, including visible output tokens and reasoning tokens.
Number between -2.0 and 2.0. Positive values penalize tokens based on whether they appear in the text so far.
Number between -2.0 and 2.0. Positive values penalize tokens based on whether they appear in the text so far.
For o1 models only. Constrains effort on reasoning. Values: low, medium, high.
stop: optional string or array of stringUp to 4 sequences where the API will stop generating further tokens.
Up to 4 sequences where the API will stop generating further tokens.
tool_choice: optional string or map[unknown]Controls which tool is called by the model. Values: none, auto, required, or specific tool.
Controls which tool is called by the model. Values: none, auto, required, or specific tool.
Inference object { configuration, alias, task_type }
configuration: object { model, args, inference_configuration }
inference_configuration: optional LaunchInferenceConfiguration { num_retries, timeout_seconds } or ItemLocatorVendor specific configuration
Vendor specific configuration
ApplicationVariant object { configuration, alias, task_type }
configuration: object { application_variant_id, inputs, history, 2 more }
Input data for the application. For agents service variants, you must provide inputs as a mapping from {input_name: input_value}. For V0 variants, you must specify the node your input should be passed to, structuring your input as {node_id: {input_name: input_value}}.
Input data for the application. For agents service variants, you must provide inputs as a mapping from {input_name: input_value}. For V0 variants, you must specify the node your input should be passed to, structuring your input as {node_id: {input_name: input_value}}.
Arbitrary user-defined metadata that can be attached to the process operations and will be registered in the interaction.
Arbitrary user-defined metadata that can be attached to the process operations and will be registered in the interaction.
overrides: optional object { concurrent, initial_state, partial_trace, 2 more } or map[object { artifact_ids_filter, artifact_name_regex, type } ] or ItemLocatorOptional overrides for the application
Optional overrides for the application
AgentexOutput object { configuration, alias, task_type }
configuration: object { agentex_agent_id, input_column, agent_task_params, 6 more }
input_column: string or map[unknown] or array of unknownThe dataset column to use as input for the agent
The dataset column to use as input for the agent
Extra params merged into the Agentex task/create call’s params object and forwarded verbatim to the agent. Required by agents that demand configuration at task creation — the golden agent, for example, rejects any task whose params omit config_id. SGP always pins is_eval: true; a caller-supplied description overrides the SGP default. Nested item.-prefixed strings and {{item.x}} templates are resolved per evaluation item, so a per-row config_id can come from a dataset column.
Extra params merged into the Agentex task/create call’s params object and forwarded verbatim to the agent. Required by agents that demand configuration at task creation — the golden agent, for example, rejects any task whose params omit config_id. SGP always pins is_eval: true; a caller-supplied description overrides the SGP default. Nested item.-prefixed strings and {{item.x}} templates are resolved per evaluation item, so a per-row config_id can come from a dataset column.
completion_mode: optional "first_message" or "turn_quiescence"How the agent’s first turn is judged finished. first_message (the default) grades the first non-empty agent text message after the input, which is cheap but grades a streaming harness on whatever text block streamed first. turn_quiescence keeps listening while the agent is still producing messages and grades once at least one agent text message exists and nothing new has arrived for quiescence_seconds — the right choice for tool-using agents. Neither mode requires the agent to mark the task complete; a terminal task status always ends the wait, and timeout_seconds always bounds it.
How the agent’s first turn is judged finished. first_message (the default) grades the first non-empty agent text message after the input, which is cheap but grades a streaming harness on whatever text block streamed first. turn_quiescence keeps listening while the agent is still producing messages and grades once at least one agent text message exists and nothing new has arrived for quiescence_seconds — the right choice for tool-using agents. Neither mode requires the agent to mark the task complete; a terminal task status always ends the wait, and timeout_seconds always bounds it.
Optional Agentex deployment ID to pin the eval to a specific deployment. When set, RPC traffic routes through /agents/{agent_id}/deployments/{deployment_id}/rpc. When unset, traffic uses the agent’s default RPC endpoint, which resolves through the agent’s current routing rules on the Agentex side.
Whether to include trace data in the evaluation results. Traces are read from SGP’s own span store for the agent’s trace, not from Agentex.
Whether to include trace data in the evaluation results. Traces are read from SGP’s own span store for the agent’s trace, not from Agentex.
input_mode: optional "text" or "data"How the resolved input_column is delivered to the agent. text (the default) sends a TextContent message with the value stringified. data sends a DataContent message whose data is the value as a JSON object; the resolved value must be an object, or a string that parses to one. Most agents accept text only and reject data.
How the resolved input_column is delivered to the agent. text (the default) sends a TextContent message with the value stringified. data sends a DataContent message whose data is the value as a JSON object; the resolved value must be an object, or a string that parses to one. Most agents accept text only and reject data.
Seconds of no new messages before completion_mode: turn_quiescence considers the turn finished. Ignored in first_message mode. Should exceed the agent’s longest expected gap between messages (a slow tool call), or the turn is graded early.
Seconds of no new messages before completion_mode: turn_quiescence considers the turn finished. Ignored in first_message mode. Should exceed the agent’s longest expected gap between messages (a slow tool call), or the turn is graded early.
Metric object { configuration, alias, task_type }
configuration: object { candidate, reference, type } or object { candidate, reference, type } or object { candidate, reference, type } or 4 more
AutoEvaluationQuestion object { configuration, alias, task_type }
AutoEvaluationGuidedDecoding object { configuration, alias, task_type }
configuration: object { model, prompt, response_format, 3 more } or object { choices, model, prompt, 3 more } or AutoEvaluationAgentTaskRequestWithItemLocator { definition, name, output_rules, 6 more }
AutoEvaluationStructuredOutputTaskRequestWithItemLocator object { model, prompt, response_format, 3 more }
run_condition: optional object { op, value } or object { path, op } or EqEvaluationRunCondition { left, right, op } or 12 more
AutoEvaluationGuidedDecodingTaskRequestWithItemLocator object { choices, model, prompt, 3 more }
run_condition: optional object { op, value } or object { path, op } or EqEvaluationRunCondition { left, right, op } or 12 more
AutoEvaluationAgentTaskRequestWithItemLocator object { definition, name, output_rules, 6 more }
designated_to: optional object { config, agent_name } or object { config, agent_name } or object { config, agent_name } or object { config, agent_name }
AutoEvaluationAgent object { configuration, alias, task_type }
configuration: AutoEvaluationAgentTaskRequestWithItemLocator { definition, name, output_rules, 6 more }
designated_to: optional object { config, agent_name } or object { config, agent_name } or object { config, agent_name } or object { config, agent_name }
ContributorEvaluationQuestion object { configuration, alias, task_type }
configuration: object { layout, question_id, prefill_from, 3 more }
direction: optional "row" or "column"The axis that children are placed in the container. Based on CSS flex-direction (see: https://developer.mozilla.org/en-US/docs/Web/CSS/flex-direction)
The axis that children are placed in the container. Based on CSS flex-direction (see: https://developer.mozilla.org/en-US/docs/Web/CSS/flex-direction)
CustomFunction object { configuration, alias, task_type }
configuration: object { function_source, arg_mapping, config_args, outputs } Configuration for a custom Python function evaluation task.
Configuration for a custom Python function evaluation task.
Mapping of function parameter names to item locators (e.g. item.field). Auto-derived from function signature if not provided.
Literal argument values for function parameters, such as thresholds or RNG seeds. Serialized JSON must be at most 10000 characters.
EvaluationWithDatasetCreateRequest object { data, dataset, name, 7 more }
dataset: object { name, description, keys, tags } Create a reusable dataset from items in the data field
Create a reusable dataset from items in the data field
Tasks allow you to augment and evaluate your data
Tasks allow you to augment and evaluate your data
ChatCompletion object { configuration, alias, task_type }
configuration: object { messages, model, audio, 24 more }
Number between -2.0 and 2.0. Positive values penalize new tokens based on their existing frequency in the text so far.
Number between -2.0 and 2.0. Positive values penalize new tokens based on their existing frequency in the text so far.
Modify the likelihood of specified tokens appearing in the completion. Maps tokens to bias values from -100 to 100.
Modify the likelihood of specified tokens appearing in the completion. Maps tokens to bias values from -100 to 100.
An upper bound for the number of tokens that can be generated, including visible output tokens and reasoning tokens.
An upper bound for the number of tokens that can be generated, including visible output tokens and reasoning tokens.
Number between -2.0 and 2.0. Positive values penalize tokens based on whether they appear in the text so far.
Number between -2.0 and 2.0. Positive values penalize tokens based on whether they appear in the text so far.
For o1 models only. Constrains effort on reasoning. Values: low, medium, high.
stop: optional string or array of stringUp to 4 sequences where the API will stop generating further tokens.
Up to 4 sequences where the API will stop generating further tokens.
tool_choice: optional string or map[unknown]Controls which tool is called by the model. Values: none, auto, required, or specific tool.
Controls which tool is called by the model. Values: none, auto, required, or specific tool.
Inference object { configuration, alias, task_type }
configuration: object { model, args, inference_configuration }
inference_configuration: optional LaunchInferenceConfiguration { num_retries, timeout_seconds } or ItemLocatorVendor specific configuration
Vendor specific configuration
ApplicationVariant object { configuration, alias, task_type }
configuration: object { application_variant_id, inputs, history, 2 more }
Input data for the application. For agents service variants, you must provide inputs as a mapping from {input_name: input_value}. For V0 variants, you must specify the node your input should be passed to, structuring your input as {node_id: {input_name: input_value}}.
Input data for the application. For agents service variants, you must provide inputs as a mapping from {input_name: input_value}. For V0 variants, you must specify the node your input should be passed to, structuring your input as {node_id: {input_name: input_value}}.
Arbitrary user-defined metadata that can be attached to the process operations and will be registered in the interaction.
Arbitrary user-defined metadata that can be attached to the process operations and will be registered in the interaction.
overrides: optional object { concurrent, initial_state, partial_trace, 2 more } or map[object { artifact_ids_filter, artifact_name_regex, type } ] or ItemLocatorOptional overrides for the application
Optional overrides for the application
AgentexOutput object { configuration, alias, task_type }
configuration: object { agentex_agent_id, input_column, agent_task_params, 6 more }
input_column: string or map[unknown] or array of unknownThe dataset column to use as input for the agent
The dataset column to use as input for the agent
Extra params merged into the Agentex task/create call’s params object and forwarded verbatim to the agent. Required by agents that demand configuration at task creation — the golden agent, for example, rejects any task whose params omit config_id. SGP always pins is_eval: true; a caller-supplied description overrides the SGP default. Nested item.-prefixed strings and {{item.x}} templates are resolved per evaluation item, so a per-row config_id can come from a dataset column.
Extra params merged into the Agentex task/create call’s params object and forwarded verbatim to the agent. Required by agents that demand configuration at task creation — the golden agent, for example, rejects any task whose params omit config_id. SGP always pins is_eval: true; a caller-supplied description overrides the SGP default. Nested item.-prefixed strings and {{item.x}} templates are resolved per evaluation item, so a per-row config_id can come from a dataset column.
completion_mode: optional "first_message" or "turn_quiescence"How the agent’s first turn is judged finished. first_message (the default) grades the first non-empty agent text message after the input, which is cheap but grades a streaming harness on whatever text block streamed first. turn_quiescence keeps listening while the agent is still producing messages and grades once at least one agent text message exists and nothing new has arrived for quiescence_seconds — the right choice for tool-using agents. Neither mode requires the agent to mark the task complete; a terminal task status always ends the wait, and timeout_seconds always bounds it.
How the agent’s first turn is judged finished. first_message (the default) grades the first non-empty agent text message after the input, which is cheap but grades a streaming harness on whatever text block streamed first. turn_quiescence keeps listening while the agent is still producing messages and grades once at least one agent text message exists and nothing new has arrived for quiescence_seconds — the right choice for tool-using agents. Neither mode requires the agent to mark the task complete; a terminal task status always ends the wait, and timeout_seconds always bounds it.
Optional Agentex deployment ID to pin the eval to a specific deployment. When set, RPC traffic routes through /agents/{agent_id}/deployments/{deployment_id}/rpc. When unset, traffic uses the agent’s default RPC endpoint, which resolves through the agent’s current routing rules on the Agentex side.
Whether to include trace data in the evaluation results. Traces are read from SGP’s own span store for the agent’s trace, not from Agentex.
Whether to include trace data in the evaluation results. Traces are read from SGP’s own span store for the agent’s trace, not from Agentex.
input_mode: optional "text" or "data"How the resolved input_column is delivered to the agent. text (the default) sends a TextContent message with the value stringified. data sends a DataContent message whose data is the value as a JSON object; the resolved value must be an object, or a string that parses to one. Most agents accept text only and reject data.
How the resolved input_column is delivered to the agent. text (the default) sends a TextContent message with the value stringified. data sends a DataContent message whose data is the value as a JSON object; the resolved value must be an object, or a string that parses to one. Most agents accept text only and reject data.
Seconds of no new messages before completion_mode: turn_quiescence considers the turn finished. Ignored in first_message mode. Should exceed the agent’s longest expected gap between messages (a slow tool call), or the turn is graded early.
Seconds of no new messages before completion_mode: turn_quiescence considers the turn finished. Ignored in first_message mode. Should exceed the agent’s longest expected gap between messages (a slow tool call), or the turn is graded early.
Metric object { configuration, alias, task_type }
configuration: object { candidate, reference, type } or object { candidate, reference, type } or object { candidate, reference, type } or 4 more
AutoEvaluationQuestion object { configuration, alias, task_type }
AutoEvaluationGuidedDecoding object { configuration, alias, task_type }
configuration: object { model, prompt, response_format, 3 more } or object { choices, model, prompt, 3 more } or AutoEvaluationAgentTaskRequestWithItemLocator { definition, name, output_rules, 6 more }
AutoEvaluationStructuredOutputTaskRequestWithItemLocator object { model, prompt, response_format, 3 more }
run_condition: optional object { op, value } or object { path, op } or EqEvaluationRunCondition { left, right, op } or 12 more
AutoEvaluationGuidedDecodingTaskRequestWithItemLocator object { choices, model, prompt, 3 more }
run_condition: optional object { op, value } or object { path, op } or EqEvaluationRunCondition { left, right, op } or 12 more
AutoEvaluationAgentTaskRequestWithItemLocator object { definition, name, output_rules, 6 more }
designated_to: optional object { config, agent_name } or object { config, agent_name } or object { config, agent_name } or object { config, agent_name }
AutoEvaluationAgent object { configuration, alias, task_type }
configuration: AutoEvaluationAgentTaskRequestWithItemLocator { definition, name, output_rules, 6 more }
designated_to: optional object { config, agent_name } or object { config, agent_name } or object { config, agent_name } or object { config, agent_name }
ContributorEvaluationQuestion object { configuration, alias, task_type }
configuration: object { layout, question_id, prefill_from, 3 more }
direction: optional "row" or "column"The axis that children are placed in the container. Based on CSS flex-direction (see: https://developer.mozilla.org/en-US/docs/Web/CSS/flex-direction)
The axis that children are placed in the container. Based on CSS flex-direction (see: https://developer.mozilla.org/en-US/docs/Web/CSS/flex-direction)
CustomFunction object { configuration, alias, task_type }
configuration: object { function_source, arg_mapping, config_args, outputs } Configuration for a custom Python function evaluation task.
Configuration for a custom Python function evaluation task.
Mapping of function parameter names to item locators (e.g. item.field). Auto-derived from function signature if not provided.
Literal argument values for function parameters, such as thresholds or RNG seeds. Serialized JSON must be at most 10000 characters.
ReturnsExpand Collapse
Evaluation object { id, created_at, created_by, 12 more }
The date and time when the entity was archived in ISO format.
Progress of the evaluation’s underlying async job
Progress of the evaluation’s underlying async job
Tasks executed during evaluation. Populated with optional task view.
Tasks executed during evaluation. Populated with optional task view.
ChatCompletion object { configuration, alias, task_type }
configuration: object { messages, model, audio, 24 more }
Number between -2.0 and 2.0. Positive values penalize new tokens based on their existing frequency in the text so far.
Number between -2.0 and 2.0. Positive values penalize new tokens based on their existing frequency in the text so far.
Modify the likelihood of specified tokens appearing in the completion. Maps tokens to bias values from -100 to 100.
Modify the likelihood of specified tokens appearing in the completion. Maps tokens to bias values from -100 to 100.
An upper bound for the number of tokens that can be generated, including visible output tokens and reasoning tokens.
An upper bound for the number of tokens that can be generated, including visible output tokens and reasoning tokens.
Number between -2.0 and 2.0. Positive values penalize tokens based on whether they appear in the text so far.
Number between -2.0 and 2.0. Positive values penalize tokens based on whether they appear in the text so far.
For o1 models only. Constrains effort on reasoning. Values: low, medium, high.
stop: optional string or array of stringUp to 4 sequences where the API will stop generating further tokens.
Up to 4 sequences where the API will stop generating further tokens.
tool_choice: optional string or map[unknown]Controls which tool is called by the model. Values: none, auto, required, or specific tool.
Controls which tool is called by the model. Values: none, auto, required, or specific tool.
Inference object { configuration, alias, task_type }
configuration: object { model, args, inference_configuration }
inference_configuration: optional LaunchInferenceConfiguration { num_retries, timeout_seconds } or ItemLocatorVendor specific configuration
Vendor specific configuration
ApplicationVariant object { configuration, alias, task_type }
configuration: object { application_variant_id, inputs, history, 2 more }
Input data for the application. For agents service variants, you must provide inputs as a mapping from {input_name: input_value}. For V0 variants, you must specify the node your input should be passed to, structuring your input as {node_id: {input_name: input_value}}.
Input data for the application. For agents service variants, you must provide inputs as a mapping from {input_name: input_value}. For V0 variants, you must specify the node your input should be passed to, structuring your input as {node_id: {input_name: input_value}}.
Arbitrary user-defined metadata that can be attached to the process operations and will be registered in the interaction.
Arbitrary user-defined metadata that can be attached to the process operations and will be registered in the interaction.
overrides: optional object { concurrent, initial_state, partial_trace, 2 more } or map[object { artifact_ids_filter, artifact_name_regex, type } ] or ItemLocatorOptional overrides for the application
Optional overrides for the application
AgentexOutput object { configuration, alias, task_type }
configuration: object { agentex_agent_id, input_column, agent_task_params, 6 more }
input_column: string or map[unknown] or array of unknownThe dataset column to use as input for the agent
The dataset column to use as input for the agent
Extra params merged into the Agentex task/create call’s params object and forwarded verbatim to the agent. Required by agents that demand configuration at task creation — the golden agent, for example, rejects any task whose params omit config_id. SGP always pins is_eval: true; a caller-supplied description overrides the SGP default. Nested item.-prefixed strings and {{item.x}} templates are resolved per evaluation item, so a per-row config_id can come from a dataset column.
Extra params merged into the Agentex task/create call’s params object and forwarded verbatim to the agent. Required by agents that demand configuration at task creation — the golden agent, for example, rejects any task whose params omit config_id. SGP always pins is_eval: true; a caller-supplied description overrides the SGP default. Nested item.-prefixed strings and {{item.x}} templates are resolved per evaluation item, so a per-row config_id can come from a dataset column.
completion_mode: optional "first_message" or "turn_quiescence"How the agent’s first turn is judged finished. first_message (the default) grades the first non-empty agent text message after the input, which is cheap but grades a streaming harness on whatever text block streamed first. turn_quiescence keeps listening while the agent is still producing messages and grades once at least one agent text message exists and nothing new has arrived for quiescence_seconds — the right choice for tool-using agents. Neither mode requires the agent to mark the task complete; a terminal task status always ends the wait, and timeout_seconds always bounds it.
How the agent’s first turn is judged finished. first_message (the default) grades the first non-empty agent text message after the input, which is cheap but grades a streaming harness on whatever text block streamed first. turn_quiescence keeps listening while the agent is still producing messages and grades once at least one agent text message exists and nothing new has arrived for quiescence_seconds — the right choice for tool-using agents. Neither mode requires the agent to mark the task complete; a terminal task status always ends the wait, and timeout_seconds always bounds it.
Optional Agentex deployment ID to pin the eval to a specific deployment. When set, RPC traffic routes through /agents/{agent_id}/deployments/{deployment_id}/rpc. When unset, traffic uses the agent’s default RPC endpoint, which resolves through the agent’s current routing rules on the Agentex side.
Whether to include trace data in the evaluation results. Traces are read from SGP’s own span store for the agent’s trace, not from Agentex.
Whether to include trace data in the evaluation results. Traces are read from SGP’s own span store for the agent’s trace, not from Agentex.
input_mode: optional "text" or "data"How the resolved input_column is delivered to the agent. text (the default) sends a TextContent message with the value stringified. data sends a DataContent message whose data is the value as a JSON object; the resolved value must be an object, or a string that parses to one. Most agents accept text only and reject data.
How the resolved input_column is delivered to the agent. text (the default) sends a TextContent message with the value stringified. data sends a DataContent message whose data is the value as a JSON object; the resolved value must be an object, or a string that parses to one. Most agents accept text only and reject data.
Seconds of no new messages before completion_mode: turn_quiescence considers the turn finished. Ignored in first_message mode. Should exceed the agent’s longest expected gap between messages (a slow tool call), or the turn is graded early.
Seconds of no new messages before completion_mode: turn_quiescence considers the turn finished. Ignored in first_message mode. Should exceed the agent’s longest expected gap between messages (a slow tool call), or the turn is graded early.
Metric object { configuration, alias, task_type }
configuration: object { candidate, reference, type } or object { candidate, reference, type } or object { candidate, reference, type } or 4 more
AutoEvaluationQuestion object { configuration, alias, task_type }
AutoEvaluationGuidedDecoding object { configuration, alias, task_type }
configuration: object { model, prompt, response_format, 3 more } or object { choices, model, prompt, 3 more } or AutoEvaluationAgentTaskRequestWithItemLocator { definition, name, output_rules, 6 more }
AutoEvaluationStructuredOutputTaskRequestWithItemLocator object { model, prompt, response_format, 3 more }
run_condition: optional object { op, value } or object { path, op } or EqEvaluationRunCondition { left, right, op } or 12 more
AutoEvaluationGuidedDecodingTaskRequestWithItemLocator object { choices, model, prompt, 3 more }
run_condition: optional object { op, value } or object { path, op } or EqEvaluationRunCondition { left, right, op } or 12 more
AutoEvaluationAgentTaskRequestWithItemLocator object { definition, name, output_rules, 6 more }
designated_to: optional object { config, agent_name } or object { config, agent_name } or object { config, agent_name } or object { config, agent_name }
AutoEvaluationAgent object { configuration, alias, task_type }
configuration: AutoEvaluationAgentTaskRequestWithItemLocator { definition, name, output_rules, 6 more }
designated_to: optional object { config, agent_name } or object { config, agent_name } or object { config, agent_name } or object { config, agent_name }
ContributorEvaluationQuestion object { configuration, alias, task_type }
configuration: object { layout, question_id, prefill_from, 3 more }
direction: optional "row" or "column"The axis that children are placed in the container. Based on CSS flex-direction (see: https://developer.mozilla.org/en-US/docs/Web/CSS/flex-direction)
The axis that children are placed in the container. Based on CSS flex-direction (see: https://developer.mozilla.org/en-US/docs/Web/CSS/flex-direction)
CustomFunction object { configuration, alias, task_type }
configuration: object { function_source, arg_mapping, config_args, outputs } Configuration for a custom Python function evaluation task.
Configuration for a custom Python function evaluation task.
Mapping of function parameter names to item locators (e.g. item.field). Auto-derived from function signature if not provided.
Literal argument values for function parameters, such as thresholds or RNG seeds. Serialized JSON must be at most 10000 characters.
Create Evaluation
curl https://api.egp.scale.com/v5/evaluations \
-H 'Content-Type: application/json' \
-H "x-api-key: $SGP_API_KEY" \
-d '{
"data": [
{
"foo": "bar"
}
],
"name": "x",
"description": "description",
"files": [
{
"foo": "string"
}
],
"metadata": {
"foo": "bar"
},
"skip_prefilled_rows": true,
"tags": [
"x"
],
"tasks": [
{
"configuration": {
"messages": [
{
"foo": "bar"
}
],
"model": "model",
"audio": {
"foo": "bar"
},
"frequency_penalty": -2,
"function_call": {
"foo": "bar"
},
"functions": [
{
"foo": "bar"
}
],
"logit_bias": {
"foo": 0
},
"logprobs": true,
"max_completion_tokens": 0,
"max_tokens": 0,
"metadata": {
"foo": "string"
},
"modalities": [
"string"
],
"n": 0,
"parallel_tool_calls": true,
"prediction": {
"foo": "bar"
},
"presence_penalty": -2,
"reasoning_effort": "reasoning_effort",
"response_format": {
"foo": "bar"
},
"seed": 0,
"stop": "string",
"store": true,
"temperature": 0,
"tool_choice": "string",
"tools": [
{
"foo": "bar"
}
],
"top_k": 0,
"top_logprobs": 0,
"top_p": 0
},
"alias": "alias",
"task_type": "chat_completion"
}
],
"taxonomy_params": {
"foo": "bar"
}
}'{
"id": "id",
"created_at": "2019-12-27T18:11:19.117Z",
"created_by": {
"id": "id",
"type": "user",
"object": "identity"
},
"datasets": [
{
"id": "id",
"created_at": "2019-12-27T18:11:19.117Z",
"created_by": {
"id": "id",
"type": "user",
"object": "identity"
},
"current_version_num": 0,
"name": "name",
"tags": [
"string"
],
"archived_at": "2019-12-27T18:11:19.117Z",
"description": "description",
"object": "dataset"
}
],
"name": "name",
"status": "failed",
"tags": [
"string"
],
"archived_at": "2019-12-27T18:11:19.117Z",
"description": "description",
"error_count": 0,
"metadata": {
"foo": "bar"
},
"object": "evaluation",
"progress": {
"items": {
"failed": 0,
"pending": 0,
"successful": 0,
"total": 0,
"failed_items": [
{
"item_id": "item_id",
"error": "error",
"error_type": "error_type"
}
]
},
"workflows": {
"completed": 0,
"failed": 0,
"pending": 0,
"total": 0
}
},
"status_reason": "status_reason",
"tasks": [
{
"configuration": {
"messages": [
{
"foo": "bar"
}
],
"model": "model",
"audio": {
"foo": "bar"
},
"frequency_penalty": -2,
"function_call": {
"foo": "bar"
},
"functions": [
{
"foo": "bar"
}
],
"logit_bias": {
"foo": 0
},
"logprobs": true,
"max_completion_tokens": 0,
"max_tokens": 0,
"metadata": {
"foo": "string"
},
"modalities": [
"string"
],
"n": 0,
"parallel_tool_calls": true,
"prediction": {
"foo": "bar"
},
"presence_penalty": -2,
"reasoning_effort": "reasoning_effort",
"response_format": {
"foo": "bar"
},
"seed": 0,
"stop": "string",
"store": true,
"temperature": 0,
"tool_choice": "string",
"tools": [
{
"foo": "bar"
}
],
"top_k": 0,
"top_logprobs": 0,
"top_p": 0
},
"alias": "alias",
"task_type": "chat_completion"
}
]
}Returns Examples
{
"id": "id",
"created_at": "2019-12-27T18:11:19.117Z",
"created_by": {
"id": "id",
"type": "user",
"object": "identity"
},
"datasets": [
{
"id": "id",
"created_at": "2019-12-27T18:11:19.117Z",
"created_by": {
"id": "id",
"type": "user",
"object": "identity"
},
"current_version_num": 0,
"name": "name",
"tags": [
"string"
],
"archived_at": "2019-12-27T18:11:19.117Z",
"description": "description",
"object": "dataset"
}
],
"name": "name",
"status": "failed",
"tags": [
"string"
],
"archived_at": "2019-12-27T18:11:19.117Z",
"description": "description",
"error_count": 0,
"metadata": {
"foo": "bar"
},
"object": "evaluation",
"progress": {
"items": {
"failed": 0,
"pending": 0,
"successful": 0,
"total": 0,
"failed_items": [
{
"item_id": "item_id",
"error": "error",
"error_type": "error_type"
}
]
},
"workflows": {
"completed": 0,
"failed": 0,
"pending": 0,
"total": 0
}
},
"status_reason": "status_reason",
"tasks": [
{
"configuration": {
"messages": [
{
"foo": "bar"
}
],
"model": "model",
"audio": {
"foo": "bar"
},
"frequency_penalty": -2,
"function_call": {
"foo": "bar"
},
"functions": [
{
"foo": "bar"
}
],
"logit_bias": {
"foo": 0
},
"logprobs": true,
"max_completion_tokens": 0,
"max_tokens": 0,
"metadata": {
"foo": "string"
},
"modalities": [
"string"
],
"n": 0,
"parallel_tool_calls": true,
"prediction": {
"foo": "bar"
},
"presence_penalty": -2,
"reasoning_effort": "reasoning_effort",
"response_format": {
"foo": "bar"
},
"seed": 0,
"stop": "string",
"store": true,
"temperature": 0,
"tool_choice": "string",
"tools": [
{
"foo": "bar"
}
],
"top_k": 0,
"top_logprobs": 0,
"top_p": 0
},
"alias": "alias",
"task_type": "chat_completion"
}
]
}