## Create Evaluation **post** `/v5/evaluations` Create an evaluation together with its items, optionally running test criteria against them. Accepts three request shapes: standalone (inline `data`), from an existing dataset (`dataset_id` with optional per-item references), or with a new reusable dataset created inline from `data`. When the evaluation includes tasks that require execution (for example an LLM judge or custom function), an async job and a Temporal workflow are started and the evaluation is returned immediately with status `running`; task results and `error_count` populate asynchronously. When it includes only contributor tasks, taxonomy-only input, or no tasks, no workflow runs and it is returned with status `completed`. Optional `tasks`, `metadata`, `tags`, and `taxonomy_params` are persisted alongside the evaluation and its items. ### Body Parameters - `evaluation: object { data, name, description, 6 more } or object { dataset_id, name, data, 6 more } or object { data, dataset, name, 7 more }` - `EvaluationStandaloneCreateRequest object { data, name, description, 6 more }` - `data: array of map[unknown]` Items to be evaluated - `name: string` - `description: optional string` - `files: optional array of map[string]` Files to be associated to the evaluation - `metadata: optional map[unknown]` Optional metadata key-value pairs for the evaluation - `skip_prefilled_rows: optional boolean` Do not queue a contributor task for prefilled questions - `tags: optional array of string` The tags associated with the evaluation - `tasks: optional array of EvaluationTask` Tasks allow you to augment and evaluate your data - `ChatCompletion object { configuration, alias, task_type }` - `configuration: object { messages, model, audio, 24 more }` - `messages: array of map[unknown] or ItemLocator` openai standard message format - `array of map[unknown]` - `ItemLocator = string` - `model: string` model specified as `model_vendor/model`, for example `openai/gpt-4o` - `audio: optional map[unknown] or ItemLocator` Parameters for audio output. Required when audio output is requested with modalities: ['audio']. - `map[unknown]` - `ItemLocator = string` - `frequency_penalty: optional number or ItemLocator` Number between -2.0 and 2.0. Positive values penalize new tokens based on their existing frequency in the text so far. - `number` - `ItemLocator = string` - `function_call: optional map[unknown] or ItemLocator` Deprecated in favor of tool_choice. Controls which function is called by the model. - `map[unknown]` - `ItemLocator = string` - `functions: optional array of map[unknown] or ItemLocator` Deprecated in favor of tools. A list of functions the model may generate JSON inputs for. - `array of map[unknown]` - `ItemLocator = string` - `logit_bias: optional map[number] or ItemLocator` Modify the likelihood of specified tokens appearing in the completion. Maps tokens to bias values from -100 to 100. - `map[number]` - `ItemLocator = string` - `logprobs: optional boolean or ItemLocator` Whether to return log probabilities of the output tokens or not. - `boolean` - `ItemLocator = string` - `max_completion_tokens: optional number or ItemLocator` An upper bound for the number of tokens that can be generated, including visible output tokens and reasoning tokens. - `number` - `ItemLocator = string` - `max_tokens: optional number or ItemLocator` Deprecated in favor of max_completion_tokens. The maximum number of tokens to generate. - `number` - `ItemLocator = string` - `metadata: optional map[string] or ItemLocator` Developer-defined tags and values used for filtering completions in the dashboard. - `map[string]` - `ItemLocator = string` - `modalities: optional array of string or ItemLocator` Output types that you would like the model to generate for this request. - `array of string` - `ItemLocator = string` - `n: optional number or ItemLocator` How many chat completion choices to generate for each input message. - `number` - `ItemLocator = string` - `parallel_tool_calls: optional boolean or ItemLocator` Whether to enable parallel function calling during tool use. - `boolean` - `ItemLocator = string` - `prediction: optional map[unknown] or ItemLocator` Static predicted output content, such as the content of a text file being regenerated. - `map[unknown]` - `ItemLocator = string` - `presence_penalty: optional number or ItemLocator` Number between -2.0 and 2.0. Positive values penalize tokens based on whether they appear in the text so far. - `number` - `ItemLocator = string` - `reasoning_effort: optional string` For o1 models only. Constrains effort on reasoning. Values: low, medium, high. - `response_format: optional map[unknown] or ItemLocator` An object specifying the format that the model must output. - `map[unknown]` - `ItemLocator = string` - `seed: optional number or ItemLocator` If specified, system will attempt to sample deterministically for repeated requests with same seed. - `number` - `ItemLocator = string` - `stop: optional string or array of string` Up to 4 sequences where the API will stop generating further tokens. - `string` - `array of string` - `store: optional boolean or ItemLocator` Whether to store the output for use in model distillation or evals products. - `boolean` - `ItemLocator = string` - `temperature: optional number or ItemLocator` What sampling temperature to use. Higher values make output more random, lower more focused. - `number` - `ItemLocator = string` - `tool_choice: optional string or map[unknown]` Controls which tool is called by the model. Values: none, auto, required, or specific tool. - `string` - `map[unknown]` - `tools: optional array of map[unknown] or ItemLocator` A list of tools the model may call. Currently, only functions are supported. Max 128 functions. - `array of map[unknown]` - `ItemLocator = string` - `top_k: optional number or ItemLocator` Only sample from the top K options for each subsequent token - `number` - `ItemLocator = string` - `top_logprobs: optional number or ItemLocator` Number of most likely tokens to return at each position, with associated log probability. - `number` - `ItemLocator = string` - `top_p: optional number or ItemLocator` Alternative to temperature. Only tokens comprising top_p probability mass are considered. - `number` - `ItemLocator = string` - `alias: optional string` Alias to title the results column. Defaults to the `chat_completion` - `task_type: optional "chat_completion"` - `"chat_completion"` - `Inference object { configuration, alias, task_type }` - `configuration: object { model, args, inference_configuration }` - `model: string` model specified as `vendor/name` (ex. openai/gpt-5) - `args: optional map[unknown] or ItemLocator` Arguments passed into model - `map[unknown]` - `ItemLocator = string` - `inference_configuration: optional LaunchInferenceConfiguration or ItemLocator` Vendor specific configuration - `LaunchInferenceConfiguration object { num_retries, timeout_seconds }` - `num_retries: optional number` - `timeout_seconds: optional number` - `ItemLocator = string` - `alias: optional string` Alias to title the results column. Defaults to the `inference` - `task_type: optional "inference"` - `"inference"` - `ApplicationVariant object { configuration, alias, task_type }` - `configuration: object { application_variant_id, inputs, history, 2 more }` - `application_variant_id: string` - `inputs: map[unknown] or ItemLocator` Input data for the application. For agents service variants, you must provide inputs as a mapping from `{input_name: input_value}`. For V0 variants, you must specify the node your input should be passed to, structuring your input as `{node_id: {input_name: input_value}}`. - `map[unknown]` - `ItemLocator = string` - `history: optional array of object { request, response, session_data } or ItemLocator` History of the application - `ApplicationRequestResponsePairArray = array of object { request, response, session_data }` - `request: string` Request inputs - `response: string` Response outputs - `session_data: optional map[unknown]` Session data corresponding to the request response pair - `ItemLocator = string` - `operation_metadata: optional map[unknown] or ItemLocator` Arbitrary user-defined metadata that can be attached to the process operations and will be registered in the interaction. - `map[unknown]` - `ItemLocator = string` - `overrides: optional object { concurrent, initial_state, partial_trace, 2 more } or map[object { artifact_ids_filter, artifact_name_regex, type } ] or ItemLocator` Optional overrides for the application - `AgenticApplicationOverrides object { concurrent, initial_state, partial_trace, 2 more }` Execution override options for agentic applications - `concurrent: optional boolean` - `initial_state: optional object { current_node, state }` - `current_node: string` - `state: map[unknown]` - `partial_trace: optional array of object { duration_ms, node_id, operation_input, 5 more }` - `duration_ms: number` - `node_id: string` - `operation_input: string` - `operation_output: string` - `operation_type: string` - `start_timestamp: string` - `workflow_id: string` - `operation_metadata: optional map[unknown]` - `return_span: optional boolean` - `use_channels: optional boolean` - `map[object { artifact_ids_filter, artifact_name_regex, type } ]` - `artifact_ids_filter: optional array of string` - `artifact_name_regex: optional array of string` - `type: optional "knowledge_base_schema"` - `"knowledge_base_schema"` - `ItemLocator = string` - `alias: optional string` Alias to title the results column. Defaults to the `application_variant` - `task_type: optional "application_variant"` - `"application_variant"` - `AgentexOutput object { configuration, alias, task_type }` - `configuration: object { agentex_agent_id, input_column, agent_task_params, 6 more }` - `agentex_agent_id: string` The ID of the Agentex agent to use - `input_column: string or map[unknown] or array of unknown` The dataset column to use as input for the agent - `string` - `map[unknown]` - `array of unknown` - `agent_task_params: optional map[unknown] or ItemLocator` Extra params merged into the Agentex `task/create` call's `params` object and forwarded verbatim to the agent. Required by agents that demand configuration at task creation -- the golden agent, for example, rejects any task whose params omit `config_id`. SGP always pins `is_eval: true`; a caller-supplied `description` overrides the SGP default. Nested `item.`-prefixed strings and `{{item.x}}` templates are resolved per evaluation item, so a per-row `config_id` can come from a dataset column. - `map[unknown]` - `ItemLocator = string` - `completion_mode: optional "first_message" or "turn_quiescence"` How the agent's first turn is judged finished. `first_message` (the default) grades the first non-empty agent text message after the input, which is cheap but grades a streaming harness on whatever text block streamed first. `turn_quiescence` keeps listening while the agent is still producing messages and grades once at least one agent text message exists and nothing new has arrived for `quiescence_seconds` -- the right choice for tool-using agents. Neither mode requires the agent to mark the task complete; a terminal task status always ends the wait, and `timeout_seconds` always bounds it. - `"first_message"` - `"turn_quiescence"` - `deployment_id: optional string` Optional Agentex deployment ID to pin the eval to a specific deployment. When set, RPC traffic routes through /agents/{agent_id}/deployments/{deployment_id}/rpc. When unset, traffic uses the agent's default RPC endpoint, which resolves through the agent's current routing rules on the Agentex side. - `include_traces: optional boolean or ItemLocator` Whether to include trace data in the evaluation results. Traces are read from SGP's own span store for the agent's trace, not from Agentex. - `boolean` - `ItemLocator = string` - `input_mode: optional "text" or "data"` How the resolved `input_column` is delivered to the agent. `text` (the default) sends a TextContent message with the value stringified. `data` sends a DataContent message whose `data` is the value as a JSON object; the resolved value must be an object, or a string that parses to one. Most agents accept text only and reject `data`. - `"text"` - `"data"` - `quiescence_seconds: optional number or ItemLocator` Seconds of no new messages before `completion_mode: turn_quiescence` considers the turn finished. Ignored in `first_message` mode. Should exceed the agent's longest expected gap between messages (a slow tool call), or the turn is graded early. - `number` - `ItemLocator = string` - `timeout_seconds: optional number or ItemLocator` Maximum seconds to wait for the agent's first-turn response per item. If not set, the server-side default of 600s applies. Capped at 1500s to stay within the evaluation item activity's 1800s start-to-close budget. - `number` - `ItemLocator = string` - `alias: optional string` Alias to title the results column. Defaults to the `agentex_output` - `task_type: optional "agentex_output"` - `"agentex_output"` - `Metric object { configuration, alias, task_type }` - `configuration: object { candidate, reference, type } or object { candidate, reference, type } or object { candidate, reference, type } or 4 more` - `Bleu object { candidate, reference, type }` - `candidate: string` - `reference: string` - `type: "bleu"` - `"bleu"` - `Meteor object { candidate, reference, type }` - `candidate: string` - `reference: string` - `type: "meteor"` - `"meteor"` - `CosineSimilarity object { candidate, reference, type }` - `candidate: string` - `reference: string` - `type: "cosine_similarity"` - `"cosine_similarity"` - `F1 object { candidate, reference, type }` - `candidate: string` - `reference: string` - `type: "f1"` - `"f1"` - `Rouge1 object { candidate, reference, type }` - `candidate: string` - `reference: string` - `type: "rouge1"` - `"rouge1"` - `Rouge2 object { candidate, reference, type }` - `candidate: string` - `reference: string` - `type: "rouge2"` - `"rouge2"` - `RougeL object { candidate, reference, type }` - `candidate: string` - `reference: string` - `type: "rougeL"` - `"rougeL"` - `alias: optional string` Alias to title the results column. Defaults to the metric type specified in the configuration - `task_type: optional "metric"` - `"metric"` - `AutoEvaluationQuestion object { configuration, alias, task_type }` - `configuration: object { model, prompt, question_id }` - `model: string` model specified as `model_vendor/model_name` - `prompt: string` - `question_id: string` question to be evaluated - `alias: optional string` Alias to title the results column. Defaults to the `auto_evaluation_question` - `task_type: optional "auto_evaluation.question"` - `"auto_evaluation.question"` - `AutoEvaluationGuidedDecoding object { configuration, alias, task_type }` - `configuration: object { model, prompt, response_format, 3 more } or object { choices, model, prompt, 3 more } or AutoEvaluationAgentTaskRequestWithItemLocator` - `AutoEvaluationStructuredOutputTaskRequestWithItemLocator object { model, prompt, response_format, 3 more }` - `model: string` model specified as `model_vendor/model_name` - `prompt: string` - `response_format: map[unknown]` JSON schema used for structuring the model response - `inference_args: optional map[unknown]` Additional arguments to pass to the inference request - `run_condition: optional object { op, value } or object { path, op } or EqEvaluationRunCondition or 12 more` - `Const object { op, value }` - `op: optional "const"` - `"const"` - `value: optional string or number or boolean` - `string` - `number` - `boolean` - `Var object { path, op }` - `path: string` - `op: optional "var"` - `"var"` - `EqEvaluationRunCondition object { left, right, op }` - `left: unknown` - `right: unknown` - `op: optional "eq"` - `"eq"` - `NeEvaluationRunCondition object { left, right, op }` - `left: unknown` - `right: unknown` - `op: optional "ne"` - `"ne"` - `LtEvaluationRunCondition object { left, right, op }` - `left: unknown` - `right: unknown` - `op: optional "lt"` - `"lt"` - `LteEvaluationRunCondition object { left, right, op }` - `left: unknown` - `right: unknown` - `op: optional "lte"` - `"lte"` - `GtEvaluationRunCondition object { left, right, op }` - `left: unknown` - `right: unknown` - `op: optional "gt"` - `"gt"` - `GteEvaluationRunCondition object { left, right, op }` - `left: unknown` - `right: unknown` - `op: optional "gte"` - `"gte"` - `AndEvaluationRunCondition object { operands, op }` - `operands: array of unknown` - `op: optional "and"` - `"and"` - `OrEvaluationRunCondition object { operands, op }` - `operands: array of unknown` - `op: optional "or"` - `"or"` - `InEvaluationRunCondition object { left, operands, op }` - `left: unknown` - `operands: array of unknown` - `op: optional "in"` - `"in"` - `NotInEvaluationRunCondition object { left, operands, op }` - `left: unknown` - `operands: array of unknown` - `op: optional "not_in"` - `"not_in"` - `NotEvaluationRunCondition object { operands, op }` - `operands: array of unknown` - `op: optional "not"` - `"not"` - `IsNullEvaluationRunCondition object { operands, op }` - `operands: array of unknown` - `op: optional "is_null"` - `"is_null"` - `IsNotNullEvaluationRunCondition object { operands, op }` - `operands: array of unknown` - `op: optional "is_not_null"` - `"is_not_null"` - `system_prompt: optional string` - `AutoEvaluationGuidedDecodingTaskRequestWithItemLocator object { choices, model, prompt, 3 more }` - `choices: array of string` Choices array cannot be empty - `model: string` model specified as `model_vendor/model_name` - `prompt: string` - `inference_args: optional map[unknown]` Additional arguments to pass to the inference request - `run_condition: optional object { op, value } or object { path, op } or EqEvaluationRunCondition or 12 more` - `Const object { op, value }` - `op: optional "const"` - `"const"` - `value: optional string or number or boolean` - `string` - `number` - `boolean` - `Var object { path, op }` - `path: string` - `op: optional "var"` - `"var"` - `EqEvaluationRunCondition object { left, right, op }` - `NeEvaluationRunCondition object { left, right, op }` - `LtEvaluationRunCondition object { left, right, op }` - `LteEvaluationRunCondition object { left, right, op }` - `GtEvaluationRunCondition object { left, right, op }` - `GteEvaluationRunCondition object { left, right, op }` - `AndEvaluationRunCondition object { operands, op }` - `OrEvaluationRunCondition object { operands, op }` - `InEvaluationRunCondition object { left, operands, op }` - `NotInEvaluationRunCondition object { left, operands, op }` - `NotEvaluationRunCondition object { operands, op }` - `IsNullEvaluationRunCondition object { operands, op }` - `IsNotNullEvaluationRunCondition object { operands, op }` - `system_prompt: optional string` - `AutoEvaluationAgentTaskRequestWithItemLocator object { definition, name, output_rules, 6 more }` - `definition: string` - `name: string` - `output_rules: array of string` - `data_fields: optional array of string` - `designated_to: optional object { config, agent_name } or object { config, agent_name } or object { config, agent_name } or object { config, agent_name }` - `ApeAgent object { config, agent_name }` - `config: object { model, temperature }` - `model: optional string` - `temperature: optional number` - `agent_name: optional "APEAgent"` - `"APEAgent"` - `IfAgent object { config, agent_name }` - `config: object { model }` - `model: optional string` - `agent_name: optional "IFAgent"` - `"IFAgent"` - `TruthfulnessAgent object { config, agent_name }` - `config: object { model }` - `model: optional string` - `agent_name: optional "TruthfulnessAgent"` - `"TruthfulnessAgent"` - `BaseAgent object { config, agent_name }` - `config: object { model }` - `model: optional string` - `agent_name: optional "BaseAgent"` - `"BaseAgent"` - `output_type: optional "text" or "integer" or "float" or "boolean"` - `"text"` - `"integer"` - `"float"` - `"boolean"` - `output_values: optional array of string or number or boolean` - `string` - `number` - `boolean` - `rubric_id: optional string` - `rubric_version: optional number` - `alias: optional string` Alias to title the results column. Defaults to the `auto_evaluation_guided_decoding` - `task_type: optional "auto_evaluation.guided_decoding"` - `"auto_evaluation.guided_decoding"` - `AutoEvaluationAgent object { configuration, alias, task_type }` - `configuration: AutoEvaluationAgentTaskRequestWithItemLocator` - `alias: optional string` Alias to title the results column. Defaults to the `auto_evaluation_agent` - `task_type: optional "auto_evaluation.agent"` - `"auto_evaluation.agent"` - `ContributorEvaluationQuestion object { configuration, alias, task_type }` - `configuration: object { layout, question_id, prefill_from, 3 more }` - `layout: Container` - `children: array of Container or Component` The children to be displayed within the container - `Container object { children, direction }` - `Component object { data, label }` - `data: ItemLocator` A pointer to the data in each evaluation item to be displayed within the component - `label: optional string` - `direction: optional "row" or "column"` The axis that children are placed in the container. Based on CSS `flex-direction` (see: https://developer.mozilla.org/en-US/docs/Web/CSS/flex-direction) - `"row"` - `"column"` - `question_id: string` - `prefill_from: optional string` Dataset column to prefill contributor question task result - `queue_id: optional string` The contributor annotation queue to include this task in. Defaults to `default` - `required: optional boolean` Whether the question is required to be answered - `rubric_id: optional string` ID of the rubric to use for scoring this evaluation question - `alias: optional string` Alias to title the results column. Defaults to the `contributor_evaluation_question` - `task_type: optional "contributor_evaluation.question"` - `"contributor_evaluation.question"` - `CustomFunction object { configuration, alias, task_type }` - `configuration: object { function_source, arg_mapping, config_args, outputs }` Configuration for a custom Python function evaluation task. - `function_source: string` Python function source code - `arg_mapping: optional map[string]` Mapping of function parameter names to item locators (e.g. item.field). Auto-derived from function signature if not provided. - `config_args: optional map[unknown]` Literal argument values for function parameters, such as thresholds or RNG seeds. Serialized JSON must be at most 10000 characters. - `outputs: optional array of object { path, alias }` Optional output paths to materialize as separate result columns. If omitted, the function return value is stored only under the task alias/data key. - `path: string` Dot path in the custom function return value to materialize. - `alias: optional string` Result column alias. Defaults to path with dots replaced by underscores. - `alias: optional string` Alias to title the results column. Defaults to the function name. - `task_type: optional "custom_function"` - `"custom_function"` - `taxonomy_params: optional map[unknown]` Taxonomy params from the task builder. When provided, stores directly as evaluation taxonomy. - `EvaluationFromDatasetCreateRequest object { dataset_id, name, data, 6 more }` - `dataset_id: string` The ID of the dataset containing the items referenced by the `data` field - `name: string` - `data: optional array of object { dataset_item_id }` Items to be evaluated, including references to the input dataset - `dataset_item_id: string` - `description: optional string` - `metadata: optional map[unknown]` Optional metadata key-value pairs for the evaluation - `skip_prefilled_rows: optional boolean` Do not queue a contributor task for prefilled questions - `tags: optional array of string` The tags associated with the evaluation - `tasks: optional array of EvaluationTask` Tasks allow you to augment and evaluate your data - `ChatCompletion object { configuration, alias, task_type }` - `Inference object { configuration, alias, task_type }` - `ApplicationVariant object { configuration, alias, task_type }` - `AgentexOutput object { configuration, alias, task_type }` - `Metric object { configuration, alias, task_type }` - `AutoEvaluationQuestion object { configuration, alias, task_type }` - `AutoEvaluationGuidedDecoding object { configuration, alias, task_type }` - `AutoEvaluationAgent object { configuration, alias, task_type }` - `ContributorEvaluationQuestion object { configuration, alias, task_type }` - `CustomFunction object { configuration, alias, task_type }` - `taxonomy_params: optional map[unknown]` Taxonomy params from the task builder. When provided, stores directly as evaluation taxonomy. - `EvaluationWithDatasetCreateRequest object { data, dataset, name, 7 more }` - `data: array of map[unknown]` Items to be evaluated - `dataset: object { name, description, keys, tags }` Create a reusable dataset from items in the `data` field - `name: string` - `description: optional string` - `keys: optional array of string` Keys from items in the `data` field that should be included in the dataset. If not provided, all keys will be included. - `tags: optional array of string` The tags associated with the entity - `name: string` - `description: optional string` - `files: optional array of map[string]` Files to be associated to the evaluation - `metadata: optional map[unknown]` Optional metadata key-value pairs for the evaluation - `skip_prefilled_rows: optional boolean` Do not queue a contributor task for prefilled questions - `tags: optional array of string` The tags associated with the evaluation - `tasks: optional array of EvaluationTask` Tasks allow you to augment and evaluate your data - `ChatCompletion object { configuration, alias, task_type }` - `Inference object { configuration, alias, task_type }` - `ApplicationVariant object { configuration, alias, task_type }` - `AgentexOutput object { configuration, alias, task_type }` - `Metric object { configuration, alias, task_type }` - `AutoEvaluationQuestion object { configuration, alias, task_type }` - `AutoEvaluationGuidedDecoding object { configuration, alias, task_type }` - `AutoEvaluationAgent object { configuration, alias, task_type }` - `ContributorEvaluationQuestion object { configuration, alias, task_type }` - `CustomFunction object { configuration, alias, task_type }` - `taxonomy_params: optional map[unknown]` Taxonomy params from the task builder. When provided, stores directly as evaluation taxonomy. ### Returns - `Evaluation object { id, created_at, created_by, 12 more }` - `id: string` The unique identifier of the entity. - `created_at: string` The date and time when the entity was created in ISO format. - `created_by: Identity` The identity that created the entity. - `id: string` - `type: "user" or "service_account"` - `"user"` - `"service_account"` - `object: optional "identity"` - `"identity"` - `datasets: array of Dataset` - `id: string` The unique identifier of the entity. - `created_at: string` The date and time when the entity was created in ISO format. - `created_by: Identity` The identity that created the entity. - `current_version_num: number` - `name: string` - `tags: array of string` The tags associated with the entity - `archived_at: optional string` The date and time when the entity was archived in ISO format. - `description: optional string` - `object: optional "dataset"` - `"dataset"` - `name: string` - `status: "failed" or "completed" or "running"` - `"failed"` - `"completed"` - `"running"` - `tags: array of string` The tags associated with the entity - `archived_at: optional string` The date and time when the entity was archived in ISO format. - `description: optional string` - `error_count: optional number` Number of task errors across all items in this evaluation. - `metadata: optional map[unknown]` Metadata key-value pairs for the evaluation - `object: optional "evaluation"` - `"evaluation"` - `progress: optional EvaluationTasksProgressSchema` Progress of the evaluation's underlying async job - `items: optional object { failed, pending, successful, 2 more }` - `failed: number` - `pending: number` - `successful: number` - `total: number` - `failed_items: optional array of object { item_id, error, error_type }` - `item_id: string` - `error: optional string` - `error_type: optional string` - `workflows: optional object { completed, failed, pending, total }` - `completed: number` - `failed: number` - `pending: number` - `total: number` - `status_reason: optional string` Reason for evaluation status - `tasks: optional array of EvaluationTask` Tasks executed during evaluation. Populated with optional `task` view. - `ChatCompletion object { configuration, alias, task_type }` - `configuration: object { messages, model, audio, 24 more }` - `messages: array of map[unknown] or ItemLocator` openai standard message format - `array of map[unknown]` - `ItemLocator = string` - `model: string` model specified as `model_vendor/model`, for example `openai/gpt-4o` - `audio: optional map[unknown] or ItemLocator` Parameters for audio output. Required when audio output is requested with modalities: ['audio']. - `map[unknown]` - `ItemLocator = string` - `frequency_penalty: optional number or ItemLocator` Number between -2.0 and 2.0. Positive values penalize new tokens based on their existing frequency in the text so far. - `number` - `ItemLocator = string` - `function_call: optional map[unknown] or ItemLocator` Deprecated in favor of tool_choice. Controls which function is called by the model. - `map[unknown]` - `ItemLocator = string` - `functions: optional array of map[unknown] or ItemLocator` Deprecated in favor of tools. A list of functions the model may generate JSON inputs for. - `array of map[unknown]` - `ItemLocator = string` - `logit_bias: optional map[number] or ItemLocator` Modify the likelihood of specified tokens appearing in the completion. Maps tokens to bias values from -100 to 100. - `map[number]` - `ItemLocator = string` - `logprobs: optional boolean or ItemLocator` Whether to return log probabilities of the output tokens or not. - `boolean` - `ItemLocator = string` - `max_completion_tokens: optional number or ItemLocator` An upper bound for the number of tokens that can be generated, including visible output tokens and reasoning tokens. - `number` - `ItemLocator = string` - `max_tokens: optional number or ItemLocator` Deprecated in favor of max_completion_tokens. The maximum number of tokens to generate. - `number` - `ItemLocator = string` - `metadata: optional map[string] or ItemLocator` Developer-defined tags and values used for filtering completions in the dashboard. - `map[string]` - `ItemLocator = string` - `modalities: optional array of string or ItemLocator` Output types that you would like the model to generate for this request. - `array of string` - `ItemLocator = string` - `n: optional number or ItemLocator` How many chat completion choices to generate for each input message. - `number` - `ItemLocator = string` - `parallel_tool_calls: optional boolean or ItemLocator` Whether to enable parallel function calling during tool use. - `boolean` - `ItemLocator = string` - `prediction: optional map[unknown] or ItemLocator` Static predicted output content, such as the content of a text file being regenerated. - `map[unknown]` - `ItemLocator = string` - `presence_penalty: optional number or ItemLocator` Number between -2.0 and 2.0. Positive values penalize tokens based on whether they appear in the text so far. - `number` - `ItemLocator = string` - `reasoning_effort: optional string` For o1 models only. Constrains effort on reasoning. Values: low, medium, high. - `response_format: optional map[unknown] or ItemLocator` An object specifying the format that the model must output. - `map[unknown]` - `ItemLocator = string` - `seed: optional number or ItemLocator` If specified, system will attempt to sample deterministically for repeated requests with same seed. - `number` - `ItemLocator = string` - `stop: optional string or array of string` Up to 4 sequences where the API will stop generating further tokens. - `string` - `array of string` - `store: optional boolean or ItemLocator` Whether to store the output for use in model distillation or evals products. - `boolean` - `ItemLocator = string` - `temperature: optional number or ItemLocator` What sampling temperature to use. Higher values make output more random, lower more focused. - `number` - `ItemLocator = string` - `tool_choice: optional string or map[unknown]` Controls which tool is called by the model. Values: none, auto, required, or specific tool. - `string` - `map[unknown]` - `tools: optional array of map[unknown] or ItemLocator` A list of tools the model may call. Currently, only functions are supported. Max 128 functions. - `array of map[unknown]` - `ItemLocator = string` - `top_k: optional number or ItemLocator` Only sample from the top K options for each subsequent token - `number` - `ItemLocator = string` - `top_logprobs: optional number or ItemLocator` Number of most likely tokens to return at each position, with associated log probability. - `number` - `ItemLocator = string` - `top_p: optional number or ItemLocator` Alternative to temperature. Only tokens comprising top_p probability mass are considered. - `number` - `ItemLocator = string` - `alias: optional string` Alias to title the results column. Defaults to the `chat_completion` - `task_type: optional "chat_completion"` - `"chat_completion"` - `Inference object { configuration, alias, task_type }` - `configuration: object { model, args, inference_configuration }` - `model: string` model specified as `vendor/name` (ex. openai/gpt-5) - `args: optional map[unknown] or ItemLocator` Arguments passed into model - `map[unknown]` - `ItemLocator = string` - `inference_configuration: optional LaunchInferenceConfiguration or ItemLocator` Vendor specific configuration - `LaunchInferenceConfiguration object { num_retries, timeout_seconds }` - `num_retries: optional number` - `timeout_seconds: optional number` - `ItemLocator = string` - `alias: optional string` Alias to title the results column. Defaults to the `inference` - `task_type: optional "inference"` - `"inference"` - `ApplicationVariant object { configuration, alias, task_type }` - `configuration: object { application_variant_id, inputs, history, 2 more }` - `application_variant_id: string` - `inputs: map[unknown] or ItemLocator` Input data for the application. For agents service variants, you must provide inputs as a mapping from `{input_name: input_value}`. For V0 variants, you must specify the node your input should be passed to, structuring your input as `{node_id: {input_name: input_value}}`. - `map[unknown]` - `ItemLocator = string` - `history: optional array of object { request, response, session_data } or ItemLocator` History of the application - `ApplicationRequestResponsePairArray = array of object { request, response, session_data }` - `request: string` Request inputs - `response: string` Response outputs - `session_data: optional map[unknown]` Session data corresponding to the request response pair - `ItemLocator = string` - `operation_metadata: optional map[unknown] or ItemLocator` Arbitrary user-defined metadata that can be attached to the process operations and will be registered in the interaction. - `map[unknown]` - `ItemLocator = string` - `overrides: optional object { concurrent, initial_state, partial_trace, 2 more } or map[object { artifact_ids_filter, artifact_name_regex, type } ] or ItemLocator` Optional overrides for the application - `AgenticApplicationOverrides object { concurrent, initial_state, partial_trace, 2 more }` Execution override options for agentic applications - `concurrent: optional boolean` - `initial_state: optional object { current_node, state }` - `current_node: string` - `state: map[unknown]` - `partial_trace: optional array of object { duration_ms, node_id, operation_input, 5 more }` - `duration_ms: number` - `node_id: string` - `operation_input: string` - `operation_output: string` - `operation_type: string` - `start_timestamp: string` - `workflow_id: string` - `operation_metadata: optional map[unknown]` - `return_span: optional boolean` - `use_channels: optional boolean` - `map[object { artifact_ids_filter, artifact_name_regex, type } ]` - `artifact_ids_filter: optional array of string` - `artifact_name_regex: optional array of string` - `type: optional "knowledge_base_schema"` - `"knowledge_base_schema"` - `ItemLocator = string` - `alias: optional string` Alias to title the results column. Defaults to the `application_variant` - `task_type: optional "application_variant"` - `"application_variant"` - `AgentexOutput object { configuration, alias, task_type }` - `configuration: object { agentex_agent_id, input_column, agent_task_params, 6 more }` - `agentex_agent_id: string` The ID of the Agentex agent to use - `input_column: string or map[unknown] or array of unknown` The dataset column to use as input for the agent - `string` - `map[unknown]` - `array of unknown` - `agent_task_params: optional map[unknown] or ItemLocator` Extra params merged into the Agentex `task/create` call's `params` object and forwarded verbatim to the agent. Required by agents that demand configuration at task creation -- the golden agent, for example, rejects any task whose params omit `config_id`. SGP always pins `is_eval: true`; a caller-supplied `description` overrides the SGP default. Nested `item.`-prefixed strings and `{{item.x}}` templates are resolved per evaluation item, so a per-row `config_id` can come from a dataset column. - `map[unknown]` - `ItemLocator = string` - `completion_mode: optional "first_message" or "turn_quiescence"` How the agent's first turn is judged finished. `first_message` (the default) grades the first non-empty agent text message after the input, which is cheap but grades a streaming harness on whatever text block streamed first. `turn_quiescence` keeps listening while the agent is still producing messages and grades once at least one agent text message exists and nothing new has arrived for `quiescence_seconds` -- the right choice for tool-using agents. Neither mode requires the agent to mark the task complete; a terminal task status always ends the wait, and `timeout_seconds` always bounds it. - `"first_message"` - `"turn_quiescence"` - `deployment_id: optional string` Optional Agentex deployment ID to pin the eval to a specific deployment. When set, RPC traffic routes through /agents/{agent_id}/deployments/{deployment_id}/rpc. When unset, traffic uses the agent's default RPC endpoint, which resolves through the agent's current routing rules on the Agentex side. - `include_traces: optional boolean or ItemLocator` Whether to include trace data in the evaluation results. Traces are read from SGP's own span store for the agent's trace, not from Agentex. - `boolean` - `ItemLocator = string` - `input_mode: optional "text" or "data"` How the resolved `input_column` is delivered to the agent. `text` (the default) sends a TextContent message with the value stringified. `data` sends a DataContent message whose `data` is the value as a JSON object; the resolved value must be an object, or a string that parses to one. Most agents accept text only and reject `data`. - `"text"` - `"data"` - `quiescence_seconds: optional number or ItemLocator` Seconds of no new messages before `completion_mode: turn_quiescence` considers the turn finished. Ignored in `first_message` mode. Should exceed the agent's longest expected gap between messages (a slow tool call), or the turn is graded early. - `number` - `ItemLocator = string` - `timeout_seconds: optional number or ItemLocator` Maximum seconds to wait for the agent's first-turn response per item. If not set, the server-side default of 600s applies. Capped at 1500s to stay within the evaluation item activity's 1800s start-to-close budget. - `number` - `ItemLocator = string` - `alias: optional string` Alias to title the results column. Defaults to the `agentex_output` - `task_type: optional "agentex_output"` - `"agentex_output"` - `Metric object { configuration, alias, task_type }` - `configuration: object { candidate, reference, type } or object { candidate, reference, type } or object { candidate, reference, type } or 4 more` - `Bleu object { candidate, reference, type }` - `candidate: string` - `reference: string` - `type: "bleu"` - `"bleu"` - `Meteor object { candidate, reference, type }` - `candidate: string` - `reference: string` - `type: "meteor"` - `"meteor"` - `CosineSimilarity object { candidate, reference, type }` - `candidate: string` - `reference: string` - `type: "cosine_similarity"` - `"cosine_similarity"` - `F1 object { candidate, reference, type }` - `candidate: string` - `reference: string` - `type: "f1"` - `"f1"` - `Rouge1 object { candidate, reference, type }` - `candidate: string` - `reference: string` - `type: "rouge1"` - `"rouge1"` - `Rouge2 object { candidate, reference, type }` - `candidate: string` - `reference: string` - `type: "rouge2"` - `"rouge2"` - `RougeL object { candidate, reference, type }` - `candidate: string` - `reference: string` - `type: "rougeL"` - `"rougeL"` - `alias: optional string` Alias to title the results column. Defaults to the metric type specified in the configuration - `task_type: optional "metric"` - `"metric"` - `AutoEvaluationQuestion object { configuration, alias, task_type }` - `configuration: object { model, prompt, question_id }` - `model: string` model specified as `model_vendor/model_name` - `prompt: string` - `question_id: string` question to be evaluated - `alias: optional string` Alias to title the results column. Defaults to the `auto_evaluation_question` - `task_type: optional "auto_evaluation.question"` - `"auto_evaluation.question"` - `AutoEvaluationGuidedDecoding object { configuration, alias, task_type }` - `configuration: object { model, prompt, response_format, 3 more } or object { choices, model, prompt, 3 more } or AutoEvaluationAgentTaskRequestWithItemLocator` - `AutoEvaluationStructuredOutputTaskRequestWithItemLocator object { model, prompt, response_format, 3 more }` - `model: string` model specified as `model_vendor/model_name` - `prompt: string` - `response_format: map[unknown]` JSON schema used for structuring the model response - `inference_args: optional map[unknown]` Additional arguments to pass to the inference request - `run_condition: optional object { op, value } or object { path, op } or EqEvaluationRunCondition or 12 more` - `Const object { op, value }` - `op: optional "const"` - `"const"` - `value: optional string or number or boolean` - `string` - `number` - `boolean` - `Var object { path, op }` - `path: string` - `op: optional "var"` - `"var"` - `EqEvaluationRunCondition object { left, right, op }` - `left: unknown` - `right: unknown` - `op: optional "eq"` - `"eq"` - `NeEvaluationRunCondition object { left, right, op }` - `left: unknown` - `right: unknown` - `op: optional "ne"` - `"ne"` - `LtEvaluationRunCondition object { left, right, op }` - `left: unknown` - `right: unknown` - `op: optional "lt"` - `"lt"` - `LteEvaluationRunCondition object { left, right, op }` - `left: unknown` - `right: unknown` - `op: optional "lte"` - `"lte"` - `GtEvaluationRunCondition object { left, right, op }` - `left: unknown` - `right: unknown` - `op: optional "gt"` - `"gt"` - `GteEvaluationRunCondition object { left, right, op }` - `left: unknown` - `right: unknown` - `op: optional "gte"` - `"gte"` - `AndEvaluationRunCondition object { operands, op }` - `operands: array of unknown` - `op: optional "and"` - `"and"` - `OrEvaluationRunCondition object { operands, op }` - `operands: array of unknown` - `op: optional "or"` - `"or"` - `InEvaluationRunCondition object { left, operands, op }` - `left: unknown` - `operands: array of unknown` - `op: optional "in"` - `"in"` - `NotInEvaluationRunCondition object { left, operands, op }` - `left: unknown` - `operands: array of unknown` - `op: optional "not_in"` - `"not_in"` - `NotEvaluationRunCondition object { operands, op }` - `operands: array of unknown` - `op: optional "not"` - `"not"` - `IsNullEvaluationRunCondition object { operands, op }` - `operands: array of unknown` - `op: optional "is_null"` - `"is_null"` - `IsNotNullEvaluationRunCondition object { operands, op }` - `operands: array of unknown` - `op: optional "is_not_null"` - `"is_not_null"` - `system_prompt: optional string` - `AutoEvaluationGuidedDecodingTaskRequestWithItemLocator object { choices, model, prompt, 3 more }` - `choices: array of string` Choices array cannot be empty - `model: string` model specified as `model_vendor/model_name` - `prompt: string` - `inference_args: optional map[unknown]` Additional arguments to pass to the inference request - `run_condition: optional object { op, value } or object { path, op } or EqEvaluationRunCondition or 12 more` - `Const object { op, value }` - `op: optional "const"` - `"const"` - `value: optional string or number or boolean` - `string` - `number` - `boolean` - `Var object { path, op }` - `path: string` - `op: optional "var"` - `"var"` - `EqEvaluationRunCondition object { left, right, op }` - `NeEvaluationRunCondition object { left, right, op }` - `LtEvaluationRunCondition object { left, right, op }` - `LteEvaluationRunCondition object { left, right, op }` - `GtEvaluationRunCondition object { left, right, op }` - `GteEvaluationRunCondition object { left, right, op }` - `AndEvaluationRunCondition object { operands, op }` - `OrEvaluationRunCondition object { operands, op }` - `InEvaluationRunCondition object { left, operands, op }` - `NotInEvaluationRunCondition object { left, operands, op }` - `NotEvaluationRunCondition object { operands, op }` - `IsNullEvaluationRunCondition object { operands, op }` - `IsNotNullEvaluationRunCondition object { operands, op }` - `system_prompt: optional string` - `AutoEvaluationAgentTaskRequestWithItemLocator object { definition, name, output_rules, 6 more }` - `definition: string` - `name: string` - `output_rules: array of string` - `data_fields: optional array of string` - `designated_to: optional object { config, agent_name } or object { config, agent_name } or object { config, agent_name } or object { config, agent_name }` - `ApeAgent object { config, agent_name }` - `config: object { model, temperature }` - `model: optional string` - `temperature: optional number` - `agent_name: optional "APEAgent"` - `"APEAgent"` - `IfAgent object { config, agent_name }` - `config: object { model }` - `model: optional string` - `agent_name: optional "IFAgent"` - `"IFAgent"` - `TruthfulnessAgent object { config, agent_name }` - `config: object { model }` - `model: optional string` - `agent_name: optional "TruthfulnessAgent"` - `"TruthfulnessAgent"` - `BaseAgent object { config, agent_name }` - `config: object { model }` - `model: optional string` - `agent_name: optional "BaseAgent"` - `"BaseAgent"` - `output_type: optional "text" or "integer" or "float" or "boolean"` - `"text"` - `"integer"` - `"float"` - `"boolean"` - `output_values: optional array of string or number or boolean` - `string` - `number` - `boolean` - `rubric_id: optional string` - `rubric_version: optional number` - `alias: optional string` Alias to title the results column. Defaults to the `auto_evaluation_guided_decoding` - `task_type: optional "auto_evaluation.guided_decoding"` - `"auto_evaluation.guided_decoding"` - `AutoEvaluationAgent object { configuration, alias, task_type }` - `configuration: AutoEvaluationAgentTaskRequestWithItemLocator` - `alias: optional string` Alias to title the results column. Defaults to the `auto_evaluation_agent` - `task_type: optional "auto_evaluation.agent"` - `"auto_evaluation.agent"` - `ContributorEvaluationQuestion object { configuration, alias, task_type }` - `configuration: object { layout, question_id, prefill_from, 3 more }` - `layout: Container` - `children: array of Container or Component` The children to be displayed within the container - `Container object { children, direction }` - `Component object { data, label }` - `data: ItemLocator` A pointer to the data in each evaluation item to be displayed within the component - `label: optional string` - `direction: optional "row" or "column"` The axis that children are placed in the container. Based on CSS `flex-direction` (see: https://developer.mozilla.org/en-US/docs/Web/CSS/flex-direction) - `"row"` - `"column"` - `question_id: string` - `prefill_from: optional string` Dataset column to prefill contributor question task result - `queue_id: optional string` The contributor annotation queue to include this task in. Defaults to `default` - `required: optional boolean` Whether the question is required to be answered - `rubric_id: optional string` ID of the rubric to use for scoring this evaluation question - `alias: optional string` Alias to title the results column. Defaults to the `contributor_evaluation_question` - `task_type: optional "contributor_evaluation.question"` - `"contributor_evaluation.question"` - `CustomFunction object { configuration, alias, task_type }` - `configuration: object { function_source, arg_mapping, config_args, outputs }` Configuration for a custom Python function evaluation task. - `function_source: string` Python function source code - `arg_mapping: optional map[string]` Mapping of function parameter names to item locators (e.g. item.field). Auto-derived from function signature if not provided. - `config_args: optional map[unknown]` Literal argument values for function parameters, such as thresholds or RNG seeds. Serialized JSON must be at most 10000 characters. - `outputs: optional array of object { path, alias }` Optional output paths to materialize as separate result columns. If omitted, the function return value is stored only under the task alias/data key. - `path: string` Dot path in the custom function return value to materialize. - `alias: optional string` Result column alias. Defaults to path with dots replaced by underscores. - `alias: optional string` Alias to title the results column. Defaults to the function name. - `task_type: optional "custom_function"` - `"custom_function"` ### Example ```http curl https://api.egp.scale.com/v5/evaluations \ -H 'Content-Type: application/json' \ -H "x-api-key: $SGP_API_KEY" \ -d '{ "data": [ { "foo": "bar" } ], "name": "x", "description": "description", "files": [ { "foo": "string" } ], "metadata": { "foo": "bar" }, "skip_prefilled_rows": true, "tags": [ "x" ], "tasks": [ { "configuration": { "messages": [ { "foo": "bar" } ], "model": "model", "audio": { "foo": "bar" }, "frequency_penalty": -2, "function_call": { "foo": "bar" }, "functions": [ { "foo": "bar" } ], "logit_bias": { "foo": 0 }, "logprobs": true, "max_completion_tokens": 0, "max_tokens": 0, "metadata": { "foo": "string" }, "modalities": [ "string" ], "n": 0, "parallel_tool_calls": true, "prediction": { "foo": "bar" }, "presence_penalty": -2, "reasoning_effort": "reasoning_effort", "response_format": { "foo": "bar" }, "seed": 0, "stop": "string", "store": true, "temperature": 0, "tool_choice": "string", "tools": [ { "foo": "bar" } ], "top_k": 0, "top_logprobs": 0, "top_p": 0 }, "alias": "alias", "task_type": "chat_completion" } ], "taxonomy_params": { "foo": "bar" } }' ``` #### Response ```json { "id": "id", "created_at": "2019-12-27T18:11:19.117Z", "created_by": { "id": "id", "type": "user", "object": "identity" }, "datasets": [ { "id": "id", "created_at": "2019-12-27T18:11:19.117Z", "created_by": { "id": "id", "type": "user", "object": "identity" }, "current_version_num": 0, "name": "name", "tags": [ "string" ], "archived_at": "2019-12-27T18:11:19.117Z", "description": "description", "object": "dataset" } ], "name": "name", "status": "failed", "tags": [ "string" ], "archived_at": "2019-12-27T18:11:19.117Z", "description": "description", "error_count": 0, "metadata": { "foo": "bar" }, "object": "evaluation", "progress": { "items": { "failed": 0, "pending": 0, "successful": 0, "total": 0, "failed_items": [ { "item_id": "item_id", "error": "error", "error_type": "error_type" } ] }, "workflows": { "completed": 0, "failed": 0, "pending": 0, "total": 0 } }, "status_reason": "status_reason", "tasks": [ { "configuration": { "messages": [ { "foo": "bar" } ], "model": "model", "audio": { "foo": "bar" }, "frequency_penalty": -2, "function_call": { "foo": "bar" }, "functions": [ { "foo": "bar" } ], "logit_bias": { "foo": 0 }, "logprobs": true, "max_completion_tokens": 0, "max_tokens": 0, "metadata": { "foo": "string" }, "modalities": [ "string" ], "n": 0, "parallel_tool_calls": true, "prediction": { "foo": "bar" }, "presence_penalty": -2, "reasoning_effort": "reasoning_effort", "response_format": { "foo": "bar" }, "seed": 0, "stop": "string", "store": true, "temperature": 0, "tool_choice": "string", "tools": [ { "foo": "bar" } ], "top_k": 0, "top_logprobs": 0, "top_p": 0 }, "alias": "alias", "task_type": "chat_completion" } ] } ```