# Tasks ## Add Test Criteria to Evaluation **post** `/v5/evaluations/{evaluation_id}/tasks` Add a new test criteria to an existing evaluation. Narrowed to contributor question tasks (`contributor_evaluation.question`); other task types must be configured when the evaluation is first created and are rejected here. The request is also rejected if the evaluation is archived, if a test criteria with the same alias already exists, or if any contributor annotation task for the evaluation has already been claimed or completed. Because only contributor question tasks are accepted, the added criteria is applied synchronously and contributors answer it against the evaluation's existing items — no async job or Temporal workflow is started. ### Path Parameters - `evaluation_id: string` ### Body Parameters - `task: EvaluationTask` New test criteria to add to the evaluation. Rejected when contributor annotation tasks for this evaluation have already been claimed or completed. Triggers a rerun so the new task executes against existing items. - `ChatCompletion object { configuration, alias, task_type }` - `configuration: object { messages, model, audio, 24 more }` - `messages: array of map[unknown] or ItemLocator` openai standard message format - `array of map[unknown]` - `ItemLocator = string` - `model: string` model specified as `model_vendor/model`, for example `openai/gpt-4o` - `audio: optional map[unknown] or ItemLocator` Parameters for audio output. Required when audio output is requested with modalities: ['audio']. - `map[unknown]` - `ItemLocator = string` - `frequency_penalty: optional number or ItemLocator` Number between -2.0 and 2.0. Positive values penalize new tokens based on their existing frequency in the text so far. - `number` - `ItemLocator = string` - `function_call: optional map[unknown] or ItemLocator` Deprecated in favor of tool_choice. Controls which function is called by the model. - `map[unknown]` - `ItemLocator = string` - `functions: optional array of map[unknown] or ItemLocator` Deprecated in favor of tools. A list of functions the model may generate JSON inputs for. - `array of map[unknown]` - `ItemLocator = string` - `logit_bias: optional map[number] or ItemLocator` Modify the likelihood of specified tokens appearing in the completion. Maps tokens to bias values from -100 to 100. - `map[number]` - `ItemLocator = string` - `logprobs: optional boolean or ItemLocator` Whether to return log probabilities of the output tokens or not. - `boolean` - `ItemLocator = string` - `max_completion_tokens: optional number or ItemLocator` An upper bound for the number of tokens that can be generated, including visible output tokens and reasoning tokens. - `number` - `ItemLocator = string` - `max_tokens: optional number or ItemLocator` Deprecated in favor of max_completion_tokens. The maximum number of tokens to generate. - `number` - `ItemLocator = string` - `metadata: optional map[string] or ItemLocator` Developer-defined tags and values used for filtering completions in the dashboard. - `map[string]` - `ItemLocator = string` - `modalities: optional array of string or ItemLocator` Output types that you would like the model to generate for this request. - `array of string` - `ItemLocator = string` - `n: optional number or ItemLocator` How many chat completion choices to generate for each input message. - `number` - `ItemLocator = string` - `parallel_tool_calls: optional boolean or ItemLocator` Whether to enable parallel function calling during tool use. - `boolean` - `ItemLocator = string` - `prediction: optional map[unknown] or ItemLocator` Static predicted output content, such as the content of a text file being regenerated. - `map[unknown]` - `ItemLocator = string` - `presence_penalty: optional number or ItemLocator` Number between -2.0 and 2.0. Positive values penalize tokens based on whether they appear in the text so far. - `number` - `ItemLocator = string` - `reasoning_effort: optional string` For o1 models only. Constrains effort on reasoning. Values: low, medium, high. - `response_format: optional map[unknown] or ItemLocator` An object specifying the format that the model must output. - `map[unknown]` - `ItemLocator = string` - `seed: optional number or ItemLocator` If specified, system will attempt to sample deterministically for repeated requests with same seed. - `number` - `ItemLocator = string` - `stop: optional string or array of string` Up to 4 sequences where the API will stop generating further tokens. - `string` - `array of string` - `store: optional boolean or ItemLocator` Whether to store the output for use in model distillation or evals products. - `boolean` - `ItemLocator = string` - `temperature: optional number or ItemLocator` What sampling temperature to use. Higher values make output more random, lower more focused. - `number` - `ItemLocator = string` - `tool_choice: optional string or map[unknown]` Controls which tool is called by the model. Values: none, auto, required, or specific tool. - `string` - `map[unknown]` - `tools: optional array of map[unknown] or ItemLocator` A list of tools the model may call. Currently, only functions are supported. Max 128 functions. - `array of map[unknown]` - `ItemLocator = string` - `top_k: optional number or ItemLocator` Only sample from the top K options for each subsequent token - `number` - `ItemLocator = string` - `top_logprobs: optional number or ItemLocator` Number of most likely tokens to return at each position, with associated log probability. - `number` - `ItemLocator = string` - `top_p: optional number or ItemLocator` Alternative to temperature. Only tokens comprising top_p probability mass are considered. - `number` - `ItemLocator = string` - `alias: optional string` Alias to title the results column. Defaults to the `chat_completion` - `task_type: optional "chat_completion"` - `"chat_completion"` - `Inference object { configuration, alias, task_type }` - `configuration: object { model, args, inference_configuration }` - `model: string` model specified as `vendor/name` (ex. openai/gpt-5) - `args: optional map[unknown] or ItemLocator` Arguments passed into model - `map[unknown]` - `ItemLocator = string` - `inference_configuration: optional LaunchInferenceConfiguration or ItemLocator` Vendor specific configuration - `LaunchInferenceConfiguration object { num_retries, timeout_seconds }` - `num_retries: optional number` - `timeout_seconds: optional number` - `ItemLocator = string` - `alias: optional string` Alias to title the results column. Defaults to the `inference` - `task_type: optional "inference"` - `"inference"` - `ApplicationVariant object { configuration, alias, task_type }` - `configuration: object { application_variant_id, inputs, history, 2 more }` - `application_variant_id: string` - `inputs: map[unknown] or ItemLocator` Input data for the application. For agents service variants, you must provide inputs as a mapping from `{input_name: input_value}`. For V0 variants, you must specify the node your input should be passed to, structuring your input as `{node_id: {input_name: input_value}}`. - `map[unknown]` - `ItemLocator = string` - `history: optional array of object { request, response, session_data } or ItemLocator` History of the application - `ApplicationRequestResponsePairArray = array of object { request, response, session_data }` - `request: string` Request inputs - `response: string` Response outputs - `session_data: optional map[unknown]` Session data corresponding to the request response pair - `ItemLocator = string` - `operation_metadata: optional map[unknown] or ItemLocator` Arbitrary user-defined metadata that can be attached to the process operations and will be registered in the interaction. - `map[unknown]` - `ItemLocator = string` - `overrides: optional object { concurrent, initial_state, partial_trace, 2 more } or map[object { artifact_ids_filter, artifact_name_regex, type } ] or ItemLocator` Optional overrides for the application - `AgenticApplicationOverrides object { concurrent, initial_state, partial_trace, 2 more }` Execution override options for agentic applications - `concurrent: optional boolean` - `initial_state: optional object { current_node, state }` - `current_node: string` - `state: map[unknown]` - `partial_trace: optional array of object { duration_ms, node_id, operation_input, 5 more }` - `duration_ms: number` - `node_id: string` - `operation_input: string` - `operation_output: string` - `operation_type: string` - `start_timestamp: string` - `workflow_id: string` - `operation_metadata: optional map[unknown]` - `return_span: optional boolean` - `use_channels: optional boolean` - `map[object { artifact_ids_filter, artifact_name_regex, type } ]` - `artifact_ids_filter: optional array of string` - `artifact_name_regex: optional array of string` - `type: optional "knowledge_base_schema"` - `"knowledge_base_schema"` - `ItemLocator = string` - `alias: optional string` Alias to title the results column. Defaults to the `application_variant` - `task_type: optional "application_variant"` - `"application_variant"` - `AgentexOutput object { configuration, alias, task_type }` - `configuration: object { agentex_agent_id, input_column, agent_task_params, 6 more }` - `agentex_agent_id: string` The ID of the Agentex agent to use - `input_column: string or map[unknown] or array of unknown` The dataset column to use as input for the agent - `string` - `map[unknown]` - `array of unknown` - `agent_task_params: optional map[unknown] or ItemLocator` Extra params merged into the Agentex `task/create` call's `params` object and forwarded verbatim to the agent. Required by agents that demand configuration at task creation -- the golden agent, for example, rejects any task whose params omit `config_id`. SGP always pins `is_eval: true`; a caller-supplied `description` overrides the SGP default. Nested `item.`-prefixed strings and `{{item.x}}` templates are resolved per evaluation item, so a per-row `config_id` can come from a dataset column. - `map[unknown]` - `ItemLocator = string` - `completion_mode: optional "first_message" or "turn_quiescence"` How the agent's first turn is judged finished. `first_message` (the default) grades the first non-empty agent text message after the input, which is cheap but grades a streaming harness on whatever text block streamed first. `turn_quiescence` keeps listening while the agent is still producing messages and grades once at least one agent text message exists and nothing new has arrived for `quiescence_seconds` -- the right choice for tool-using agents. Neither mode requires the agent to mark the task complete; a terminal task status always ends the wait, and `timeout_seconds` always bounds it. - `"first_message"` - `"turn_quiescence"` - `deployment_id: optional string` Optional Agentex deployment ID to pin the eval to a specific deployment. When set, RPC traffic routes through /agents/{agent_id}/deployments/{deployment_id}/rpc. When unset, traffic uses the agent's default RPC endpoint, which resolves through the agent's current routing rules on the Agentex side. - `include_traces: optional boolean or ItemLocator` Whether to include trace data in the evaluation results. Traces are read from SGP's own span store for the agent's trace, not from Agentex. - `boolean` - `ItemLocator = string` - `input_mode: optional "text" or "data"` How the resolved `input_column` is delivered to the agent. `text` (the default) sends a TextContent message with the value stringified. `data` sends a DataContent message whose `data` is the value as a JSON object; the resolved value must be an object, or a string that parses to one. Most agents accept text only and reject `data`. - `"text"` - `"data"` - `quiescence_seconds: optional number or ItemLocator` Seconds of no new messages before `completion_mode: turn_quiescence` considers the turn finished. Ignored in `first_message` mode. Should exceed the agent's longest expected gap between messages (a slow tool call), or the turn is graded early. - `number` - `ItemLocator = string` - `timeout_seconds: optional number or ItemLocator` Maximum seconds to wait for the agent's first-turn response per item. If not set, the server-side default of 600s applies. Capped at 1500s to stay within the evaluation item activity's 1800s start-to-close budget. - `number` - `ItemLocator = string` - `alias: optional string` Alias to title the results column. Defaults to the `agentex_output` - `task_type: optional "agentex_output"` - `"agentex_output"` - `Metric object { configuration, alias, task_type }` - `configuration: object { candidate, reference, type } or object { candidate, reference, type } or object { candidate, reference, type } or 4 more` - `Bleu object { candidate, reference, type }` - `candidate: string` - `reference: string` - `type: "bleu"` - `"bleu"` - `Meteor object { candidate, reference, type }` - `candidate: string` - `reference: string` - `type: "meteor"` - `"meteor"` - `CosineSimilarity object { candidate, reference, type }` - `candidate: string` - `reference: string` - `type: "cosine_similarity"` - `"cosine_similarity"` - `F1 object { candidate, reference, type }` - `candidate: string` - `reference: string` - `type: "f1"` - `"f1"` - `Rouge1 object { candidate, reference, type }` - `candidate: string` - `reference: string` - `type: "rouge1"` - `"rouge1"` - `Rouge2 object { candidate, reference, type }` - `candidate: string` - `reference: string` - `type: "rouge2"` - `"rouge2"` - `RougeL object { candidate, reference, type }` - `candidate: string` - `reference: string` - `type: "rougeL"` - `"rougeL"` - `alias: optional string` Alias to title the results column. Defaults to the metric type specified in the configuration - `task_type: optional "metric"` - `"metric"` - `AutoEvaluationQuestion object { configuration, alias, task_type }` - `configuration: object { model, prompt, question_id }` - `model: string` model specified as `model_vendor/model_name` - `prompt: string` - `question_id: string` question to be evaluated - `alias: optional string` Alias to title the results column. Defaults to the `auto_evaluation_question` - `task_type: optional "auto_evaluation.question"` - `"auto_evaluation.question"` - `AutoEvaluationGuidedDecoding object { configuration, alias, task_type }` - `configuration: object { model, prompt, response_format, 3 more } or object { choices, model, prompt, 3 more } or AutoEvaluationAgentTaskRequestWithItemLocator` - `AutoEvaluationStructuredOutputTaskRequestWithItemLocator object { model, prompt, response_format, 3 more }` - `model: string` model specified as `model_vendor/model_name` - `prompt: string` - `response_format: map[unknown]` JSON schema used for structuring the model response - `inference_args: optional map[unknown]` Additional arguments to pass to the inference request - `run_condition: optional object { op, value } or object { path, op } or EqEvaluationRunCondition or 12 more` - `Const object { op, value }` - `op: optional "const"` - `"const"` - `value: optional string or number or boolean` - `string` - `number` - `boolean` - `Var object { path, op }` - `path: string` - `op: optional "var"` - `"var"` - `EqEvaluationRunCondition object { left, right, op }` - `left: unknown` - `right: unknown` - `op: optional "eq"` - `"eq"` - `NeEvaluationRunCondition object { left, right, op }` - `left: unknown` - `right: unknown` - `op: optional "ne"` - `"ne"` - `LtEvaluationRunCondition object { left, right, op }` - `left: unknown` - `right: unknown` - `op: optional "lt"` - `"lt"` - `LteEvaluationRunCondition object { left, right, op }` - `left: unknown` - `right: unknown` - `op: optional "lte"` - `"lte"` - `GtEvaluationRunCondition object { left, right, op }` - `left: unknown` - `right: unknown` - `op: optional "gt"` - `"gt"` - `GteEvaluationRunCondition object { left, right, op }` - `left: unknown` - `right: unknown` - `op: optional "gte"` - `"gte"` - `AndEvaluationRunCondition object { operands, op }` - `operands: array of unknown` - `op: optional "and"` - `"and"` - `OrEvaluationRunCondition object { operands, op }` - `operands: array of unknown` - `op: optional "or"` - `"or"` - `InEvaluationRunCondition object { left, operands, op }` - `left: unknown` - `operands: array of unknown` - `op: optional "in"` - `"in"` - `NotInEvaluationRunCondition object { left, operands, op }` - `left: unknown` - `operands: array of unknown` - `op: optional "not_in"` - `"not_in"` - `NotEvaluationRunCondition object { operands, op }` - `operands: array of unknown` - `op: optional "not"` - `"not"` - `IsNullEvaluationRunCondition object { operands, op }` - `operands: array of unknown` - `op: optional "is_null"` - `"is_null"` - `IsNotNullEvaluationRunCondition object { operands, op }` - `operands: array of unknown` - `op: optional "is_not_null"` - `"is_not_null"` - `system_prompt: optional string` - `AutoEvaluationGuidedDecodingTaskRequestWithItemLocator object { choices, model, prompt, 3 more }` - `choices: array of string` Choices array cannot be empty - `model: string` model specified as `model_vendor/model_name` - `prompt: string` - `inference_args: optional map[unknown]` Additional arguments to pass to the inference request - `run_condition: optional object { op, value } or object { path, op } or EqEvaluationRunCondition or 12 more` - `Const object { op, value }` - `op: optional "const"` - `"const"` - `value: optional string or number or boolean` - `string` - `number` - `boolean` - `Var object { path, op }` - `path: string` - `op: optional "var"` - `"var"` - `EqEvaluationRunCondition object { left, right, op }` - `NeEvaluationRunCondition object { left, right, op }` - `LtEvaluationRunCondition object { left, right, op }` - `LteEvaluationRunCondition object { left, right, op }` - `GtEvaluationRunCondition object { left, right, op }` - `GteEvaluationRunCondition object { left, right, op }` - `AndEvaluationRunCondition object { operands, op }` - `OrEvaluationRunCondition object { operands, op }` - `InEvaluationRunCondition object { left, operands, op }` - `NotInEvaluationRunCondition object { left, operands, op }` - `NotEvaluationRunCondition object { operands, op }` - `IsNullEvaluationRunCondition object { operands, op }` - `IsNotNullEvaluationRunCondition object { operands, op }` - `system_prompt: optional string` - `AutoEvaluationAgentTaskRequestWithItemLocator object { definition, name, output_rules, 6 more }` - `definition: string` - `name: string` - `output_rules: array of string` - `data_fields: optional array of string` - `designated_to: optional object { config, agent_name } or object { config, agent_name } or object { config, agent_name } or object { config, agent_name }` - `ApeAgent object { config, agent_name }` - `config: object { model, temperature }` - `model: optional string` - `temperature: optional number` - `agent_name: optional "APEAgent"` - `"APEAgent"` - `IfAgent object { config, agent_name }` - `config: object { model }` - `model: optional string` - `agent_name: optional "IFAgent"` - `"IFAgent"` - `TruthfulnessAgent object { config, agent_name }` - `config: object { model }` - `model: optional string` - `agent_name: optional "TruthfulnessAgent"` - `"TruthfulnessAgent"` - `BaseAgent object { config, agent_name }` - `config: object { model }` - `model: optional string` - `agent_name: optional "BaseAgent"` - `"BaseAgent"` - `output_type: optional "text" or "integer" or "float" or "boolean"` - `"text"` - `"integer"` - `"float"` - `"boolean"` - `output_values: optional array of string or number or boolean` - `string` - `number` - `boolean` - `rubric_id: optional string` - `rubric_version: optional number` - `alias: optional string` Alias to title the results column. Defaults to the `auto_evaluation_guided_decoding` - `task_type: optional "auto_evaluation.guided_decoding"` - `"auto_evaluation.guided_decoding"` - `AutoEvaluationAgent object { configuration, alias, task_type }` - `configuration: AutoEvaluationAgentTaskRequestWithItemLocator` - `alias: optional string` Alias to title the results column. Defaults to the `auto_evaluation_agent` - `task_type: optional "auto_evaluation.agent"` - `"auto_evaluation.agent"` - `ContributorEvaluationQuestion object { configuration, alias, task_type }` - `configuration: object { layout, question_id, prefill_from, 3 more }` - `layout: Container` - `children: array of Container or Component` The children to be displayed within the container - `Container object { children, direction }` - `Component object { data, label }` - `data: ItemLocator` A pointer to the data in each evaluation item to be displayed within the component - `label: optional string` - `direction: optional "row" or "column"` The axis that children are placed in the container. Based on CSS `flex-direction` (see: https://developer.mozilla.org/en-US/docs/Web/CSS/flex-direction) - `"row"` - `"column"` - `question_id: string` - `prefill_from: optional string` Dataset column to prefill contributor question task result - `queue_id: optional string` The contributor annotation queue to include this task in. Defaults to `default` - `required: optional boolean` Whether the question is required to be answered - `rubric_id: optional string` ID of the rubric to use for scoring this evaluation question - `alias: optional string` Alias to title the results column. Defaults to the `contributor_evaluation_question` - `task_type: optional "contributor_evaluation.question"` - `"contributor_evaluation.question"` - `CustomFunction object { configuration, alias, task_type }` - `configuration: object { function_source, arg_mapping, config_args, outputs }` Configuration for a custom Python function evaluation task. - `function_source: string` Python function source code - `arg_mapping: optional map[string]` Mapping of function parameter names to item locators (e.g. item.field). Auto-derived from function signature if not provided. - `config_args: optional map[unknown]` Literal argument values for function parameters, such as thresholds or RNG seeds. Serialized JSON must be at most 10000 characters. - `outputs: optional array of object { path, alias }` Optional output paths to materialize as separate result columns. If omitted, the function return value is stored only under the task alias/data key. - `path: string` Dot path in the custom function return value to materialize. - `alias: optional string` Result column alias. Defaults to path with dots replaced by underscores. - `alias: optional string` Alias to title the results column. Defaults to the function name. - `task_type: optional "custom_function"` - `"custom_function"` ### Returns - `Evaluation object { id, created_at, created_by, 12 more }` - `id: string` The unique identifier of the entity. - `created_at: string` The date and time when the entity was created in ISO format. - `created_by: Identity` The identity that created the entity. - `id: string` - `type: "user" or "service_account"` - `"user"` - `"service_account"` - `object: optional "identity"` - `"identity"` - `datasets: array of Dataset` - `id: string` The unique identifier of the entity. - `created_at: string` The date and time when the entity was created in ISO format. - `created_by: Identity` The identity that created the entity. - `current_version_num: number` - `name: string` - `tags: array of string` The tags associated with the entity - `archived_at: optional string` The date and time when the entity was archived in ISO format. - `description: optional string` - `object: optional "dataset"` - `"dataset"` - `name: string` - `status: "failed" or "completed" or "running"` - `"failed"` - `"completed"` - `"running"` - `tags: array of string` The tags associated with the entity - `archived_at: optional string` The date and time when the entity was archived in ISO format. - `description: optional string` - `error_count: optional number` Number of task errors across all items in this evaluation. - `metadata: optional map[unknown]` Metadata key-value pairs for the evaluation - `object: optional "evaluation"` - `"evaluation"` - `progress: optional EvaluationTasksProgressSchema` Progress of the evaluation's underlying async job - `items: optional object { failed, pending, successful, 2 more }` - `failed: number` - `pending: number` - `successful: number` - `total: number` - `failed_items: optional array of object { item_id, error, error_type }` - `item_id: string` - `error: optional string` - `error_type: optional string` - `workflows: optional object { completed, failed, pending, total }` - `completed: number` - `failed: number` - `pending: number` - `total: number` - `status_reason: optional string` Reason for evaluation status - `tasks: optional array of EvaluationTask` Tasks executed during evaluation. Populated with optional `task` view. - `ChatCompletion object { configuration, alias, task_type }` - `configuration: object { messages, model, audio, 24 more }` - `messages: array of map[unknown] or ItemLocator` openai standard message format - `array of map[unknown]` - `ItemLocator = string` - `model: string` model specified as `model_vendor/model`, for example `openai/gpt-4o` - `audio: optional map[unknown] or ItemLocator` Parameters for audio output. Required when audio output is requested with modalities: ['audio']. - `map[unknown]` - `ItemLocator = string` - `frequency_penalty: optional number or ItemLocator` Number between -2.0 and 2.0. Positive values penalize new tokens based on their existing frequency in the text so far. - `number` - `ItemLocator = string` - `function_call: optional map[unknown] or ItemLocator` Deprecated in favor of tool_choice. Controls which function is called by the model. - `map[unknown]` - `ItemLocator = string` - `functions: optional array of map[unknown] or ItemLocator` Deprecated in favor of tools. A list of functions the model may generate JSON inputs for. - `array of map[unknown]` - `ItemLocator = string` - `logit_bias: optional map[number] or ItemLocator` Modify the likelihood of specified tokens appearing in the completion. Maps tokens to bias values from -100 to 100. - `map[number]` - `ItemLocator = string` - `logprobs: optional boolean or ItemLocator` Whether to return log probabilities of the output tokens or not. - `boolean` - `ItemLocator = string` - `max_completion_tokens: optional number or ItemLocator` An upper bound for the number of tokens that can be generated, including visible output tokens and reasoning tokens. - `number` - `ItemLocator = string` - `max_tokens: optional number or ItemLocator` Deprecated in favor of max_completion_tokens. The maximum number of tokens to generate. - `number` - `ItemLocator = string` - `metadata: optional map[string] or ItemLocator` Developer-defined tags and values used for filtering completions in the dashboard. - `map[string]` - `ItemLocator = string` - `modalities: optional array of string or ItemLocator` Output types that you would like the model to generate for this request. - `array of string` - `ItemLocator = string` - `n: optional number or ItemLocator` How many chat completion choices to generate for each input message. - `number` - `ItemLocator = string` - `parallel_tool_calls: optional boolean or ItemLocator` Whether to enable parallel function calling during tool use. - `boolean` - `ItemLocator = string` - `prediction: optional map[unknown] or ItemLocator` Static predicted output content, such as the content of a text file being regenerated. - `map[unknown]` - `ItemLocator = string` - `presence_penalty: optional number or ItemLocator` Number between -2.0 and 2.0. Positive values penalize tokens based on whether they appear in the text so far. - `number` - `ItemLocator = string` - `reasoning_effort: optional string` For o1 models only. Constrains effort on reasoning. Values: low, medium, high. - `response_format: optional map[unknown] or ItemLocator` An object specifying the format that the model must output. - `map[unknown]` - `ItemLocator = string` - `seed: optional number or ItemLocator` If specified, system will attempt to sample deterministically for repeated requests with same seed. - `number` - `ItemLocator = string` - `stop: optional string or array of string` Up to 4 sequences where the API will stop generating further tokens. - `string` - `array of string` - `store: optional boolean or ItemLocator` Whether to store the output for use in model distillation or evals products. - `boolean` - `ItemLocator = string` - `temperature: optional number or ItemLocator` What sampling temperature to use. Higher values make output more random, lower more focused. - `number` - `ItemLocator = string` - `tool_choice: optional string or map[unknown]` Controls which tool is called by the model. Values: none, auto, required, or specific tool. - `string` - `map[unknown]` - `tools: optional array of map[unknown] or ItemLocator` A list of tools the model may call. Currently, only functions are supported. Max 128 functions. - `array of map[unknown]` - `ItemLocator = string` - `top_k: optional number or ItemLocator` Only sample from the top K options for each subsequent token - `number` - `ItemLocator = string` - `top_logprobs: optional number or ItemLocator` Number of most likely tokens to return at each position, with associated log probability. - `number` - `ItemLocator = string` - `top_p: optional number or ItemLocator` Alternative to temperature. Only tokens comprising top_p probability mass are considered. - `number` - `ItemLocator = string` - `alias: optional string` Alias to title the results column. Defaults to the `chat_completion` - `task_type: optional "chat_completion"` - `"chat_completion"` - `Inference object { configuration, alias, task_type }` - `configuration: object { model, args, inference_configuration }` - `model: string` model specified as `vendor/name` (ex. openai/gpt-5) - `args: optional map[unknown] or ItemLocator` Arguments passed into model - `map[unknown]` - `ItemLocator = string` - `inference_configuration: optional LaunchInferenceConfiguration or ItemLocator` Vendor specific configuration - `LaunchInferenceConfiguration object { num_retries, timeout_seconds }` - `num_retries: optional number` - `timeout_seconds: optional number` - `ItemLocator = string` - `alias: optional string` Alias to title the results column. Defaults to the `inference` - `task_type: optional "inference"` - `"inference"` - `ApplicationVariant object { configuration, alias, task_type }` - `configuration: object { application_variant_id, inputs, history, 2 more }` - `application_variant_id: string` - `inputs: map[unknown] or ItemLocator` Input data for the application. For agents service variants, you must provide inputs as a mapping from `{input_name: input_value}`. For V0 variants, you must specify the node your input should be passed to, structuring your input as `{node_id: {input_name: input_value}}`. - `map[unknown]` - `ItemLocator = string` - `history: optional array of object { request, response, session_data } or ItemLocator` History of the application - `ApplicationRequestResponsePairArray = array of object { request, response, session_data }` - `request: string` Request inputs - `response: string` Response outputs - `session_data: optional map[unknown]` Session data corresponding to the request response pair - `ItemLocator = string` - `operation_metadata: optional map[unknown] or ItemLocator` Arbitrary user-defined metadata that can be attached to the process operations and will be registered in the interaction. - `map[unknown]` - `ItemLocator = string` - `overrides: optional object { concurrent, initial_state, partial_trace, 2 more } or map[object { artifact_ids_filter, artifact_name_regex, type } ] or ItemLocator` Optional overrides for the application - `AgenticApplicationOverrides object { concurrent, initial_state, partial_trace, 2 more }` Execution override options for agentic applications - `concurrent: optional boolean` - `initial_state: optional object { current_node, state }` - `current_node: string` - `state: map[unknown]` - `partial_trace: optional array of object { duration_ms, node_id, operation_input, 5 more }` - `duration_ms: number` - `node_id: string` - `operation_input: string` - `operation_output: string` - `operation_type: string` - `start_timestamp: string` - `workflow_id: string` - `operation_metadata: optional map[unknown]` - `return_span: optional boolean` - `use_channels: optional boolean` - `map[object { artifact_ids_filter, artifact_name_regex, type } ]` - `artifact_ids_filter: optional array of string` - `artifact_name_regex: optional array of string` - `type: optional "knowledge_base_schema"` - `"knowledge_base_schema"` - `ItemLocator = string` - `alias: optional string` Alias to title the results column. Defaults to the `application_variant` - `task_type: optional "application_variant"` - `"application_variant"` - `AgentexOutput object { configuration, alias, task_type }` - `configuration: object { agentex_agent_id, input_column, agent_task_params, 6 more }` - `agentex_agent_id: string` The ID of the Agentex agent to use - `input_column: string or map[unknown] or array of unknown` The dataset column to use as input for the agent - `string` - `map[unknown]` - `array of unknown` - `agent_task_params: optional map[unknown] or ItemLocator` Extra params merged into the Agentex `task/create` call's `params` object and forwarded verbatim to the agent. Required by agents that demand configuration at task creation -- the golden agent, for example, rejects any task whose params omit `config_id`. SGP always pins `is_eval: true`; a caller-supplied `description` overrides the SGP default. Nested `item.`-prefixed strings and `{{item.x}}` templates are resolved per evaluation item, so a per-row `config_id` can come from a dataset column. - `map[unknown]` - `ItemLocator = string` - `completion_mode: optional "first_message" or "turn_quiescence"` How the agent's first turn is judged finished. `first_message` (the default) grades the first non-empty agent text message after the input, which is cheap but grades a streaming harness on whatever text block streamed first. `turn_quiescence` keeps listening while the agent is still producing messages and grades once at least one agent text message exists and nothing new has arrived for `quiescence_seconds` -- the right choice for tool-using agents. Neither mode requires the agent to mark the task complete; a terminal task status always ends the wait, and `timeout_seconds` always bounds it. - `"first_message"` - `"turn_quiescence"` - `deployment_id: optional string` Optional Agentex deployment ID to pin the eval to a specific deployment. When set, RPC traffic routes through /agents/{agent_id}/deployments/{deployment_id}/rpc. When unset, traffic uses the agent's default RPC endpoint, which resolves through the agent's current routing rules on the Agentex side. - `include_traces: optional boolean or ItemLocator` Whether to include trace data in the evaluation results. Traces are read from SGP's own span store for the agent's trace, not from Agentex. - `boolean` - `ItemLocator = string` - `input_mode: optional "text" or "data"` How the resolved `input_column` is delivered to the agent. `text` (the default) sends a TextContent message with the value stringified. `data` sends a DataContent message whose `data` is the value as a JSON object; the resolved value must be an object, or a string that parses to one. Most agents accept text only and reject `data`. - `"text"` - `"data"` - `quiescence_seconds: optional number or ItemLocator` Seconds of no new messages before `completion_mode: turn_quiescence` considers the turn finished. Ignored in `first_message` mode. Should exceed the agent's longest expected gap between messages (a slow tool call), or the turn is graded early. - `number` - `ItemLocator = string` - `timeout_seconds: optional number or ItemLocator` Maximum seconds to wait for the agent's first-turn response per item. If not set, the server-side default of 600s applies. Capped at 1500s to stay within the evaluation item activity's 1800s start-to-close budget. - `number` - `ItemLocator = string` - `alias: optional string` Alias to title the results column. Defaults to the `agentex_output` - `task_type: optional "agentex_output"` - `"agentex_output"` - `Metric object { configuration, alias, task_type }` - `configuration: object { candidate, reference, type } or object { candidate, reference, type } or object { candidate, reference, type } or 4 more` - `Bleu object { candidate, reference, type }` - `candidate: string` - `reference: string` - `type: "bleu"` - `"bleu"` - `Meteor object { candidate, reference, type }` - `candidate: string` - `reference: string` - `type: "meteor"` - `"meteor"` - `CosineSimilarity object { candidate, reference, type }` - `candidate: string` - `reference: string` - `type: "cosine_similarity"` - `"cosine_similarity"` - `F1 object { candidate, reference, type }` - `candidate: string` - `reference: string` - `type: "f1"` - `"f1"` - `Rouge1 object { candidate, reference, type }` - `candidate: string` - `reference: string` - `type: "rouge1"` - `"rouge1"` - `Rouge2 object { candidate, reference, type }` - `candidate: string` - `reference: string` - `type: "rouge2"` - `"rouge2"` - `RougeL object { candidate, reference, type }` - `candidate: string` - `reference: string` - `type: "rougeL"` - `"rougeL"` - `alias: optional string` Alias to title the results column. Defaults to the metric type specified in the configuration - `task_type: optional "metric"` - `"metric"` - `AutoEvaluationQuestion object { configuration, alias, task_type }` - `configuration: object { model, prompt, question_id }` - `model: string` model specified as `model_vendor/model_name` - `prompt: string` - `question_id: string` question to be evaluated - `alias: optional string` Alias to title the results column. Defaults to the `auto_evaluation_question` - `task_type: optional "auto_evaluation.question"` - `"auto_evaluation.question"` - `AutoEvaluationGuidedDecoding object { configuration, alias, task_type }` - `configuration: object { model, prompt, response_format, 3 more } or object { choices, model, prompt, 3 more } or AutoEvaluationAgentTaskRequestWithItemLocator` - `AutoEvaluationStructuredOutputTaskRequestWithItemLocator object { model, prompt, response_format, 3 more }` - `model: string` model specified as `model_vendor/model_name` - `prompt: string` - `response_format: map[unknown]` JSON schema used for structuring the model response - `inference_args: optional map[unknown]` Additional arguments to pass to the inference request - `run_condition: optional object { op, value } or object { path, op } or EqEvaluationRunCondition or 12 more` - `Const object { op, value }` - `op: optional "const"` - `"const"` - `value: optional string or number or boolean` - `string` - `number` - `boolean` - `Var object { path, op }` - `path: string` - `op: optional "var"` - `"var"` - `EqEvaluationRunCondition object { left, right, op }` - `left: unknown` - `right: unknown` - `op: optional "eq"` - `"eq"` - `NeEvaluationRunCondition object { left, right, op }` - `left: unknown` - `right: unknown` - `op: optional "ne"` - `"ne"` - `LtEvaluationRunCondition object { left, right, op }` - `left: unknown` - `right: unknown` - `op: optional "lt"` - `"lt"` - `LteEvaluationRunCondition object { left, right, op }` - `left: unknown` - `right: unknown` - `op: optional "lte"` - `"lte"` - `GtEvaluationRunCondition object { left, right, op }` - `left: unknown` - `right: unknown` - `op: optional "gt"` - `"gt"` - `GteEvaluationRunCondition object { left, right, op }` - `left: unknown` - `right: unknown` - `op: optional "gte"` - `"gte"` - `AndEvaluationRunCondition object { operands, op }` - `operands: array of unknown` - `op: optional "and"` - `"and"` - `OrEvaluationRunCondition object { operands, op }` - `operands: array of unknown` - `op: optional "or"` - `"or"` - `InEvaluationRunCondition object { left, operands, op }` - `left: unknown` - `operands: array of unknown` - `op: optional "in"` - `"in"` - `NotInEvaluationRunCondition object { left, operands, op }` - `left: unknown` - `operands: array of unknown` - `op: optional "not_in"` - `"not_in"` - `NotEvaluationRunCondition object { operands, op }` - `operands: array of unknown` - `op: optional "not"` - `"not"` - `IsNullEvaluationRunCondition object { operands, op }` - `operands: array of unknown` - `op: optional "is_null"` - `"is_null"` - `IsNotNullEvaluationRunCondition object { operands, op }` - `operands: array of unknown` - `op: optional "is_not_null"` - `"is_not_null"` - `system_prompt: optional string` - `AutoEvaluationGuidedDecodingTaskRequestWithItemLocator object { choices, model, prompt, 3 more }` - `choices: array of string` Choices array cannot be empty - `model: string` model specified as `model_vendor/model_name` - `prompt: string` - `inference_args: optional map[unknown]` Additional arguments to pass to the inference request - `run_condition: optional object { op, value } or object { path, op } or EqEvaluationRunCondition or 12 more` - `Const object { op, value }` - `op: optional "const"` - `"const"` - `value: optional string or number or boolean` - `string` - `number` - `boolean` - `Var object { path, op }` - `path: string` - `op: optional "var"` - `"var"` - `EqEvaluationRunCondition object { left, right, op }` - `NeEvaluationRunCondition object { left, right, op }` - `LtEvaluationRunCondition object { left, right, op }` - `LteEvaluationRunCondition object { left, right, op }` - `GtEvaluationRunCondition object { left, right, op }` - `GteEvaluationRunCondition object { left, right, op }` - `AndEvaluationRunCondition object { operands, op }` - `OrEvaluationRunCondition object { operands, op }` - `InEvaluationRunCondition object { left, operands, op }` - `NotInEvaluationRunCondition object { left, operands, op }` - `NotEvaluationRunCondition object { operands, op }` - `IsNullEvaluationRunCondition object { operands, op }` - `IsNotNullEvaluationRunCondition object { operands, op }` - `system_prompt: optional string` - `AutoEvaluationAgentTaskRequestWithItemLocator object { definition, name, output_rules, 6 more }` - `definition: string` - `name: string` - `output_rules: array of string` - `data_fields: optional array of string` - `designated_to: optional object { config, agent_name } or object { config, agent_name } or object { config, agent_name } or object { config, agent_name }` - `ApeAgent object { config, agent_name }` - `config: object { model, temperature }` - `model: optional string` - `temperature: optional number` - `agent_name: optional "APEAgent"` - `"APEAgent"` - `IfAgent object { config, agent_name }` - `config: object { model }` - `model: optional string` - `agent_name: optional "IFAgent"` - `"IFAgent"` - `TruthfulnessAgent object { config, agent_name }` - `config: object { model }` - `model: optional string` - `agent_name: optional "TruthfulnessAgent"` - `"TruthfulnessAgent"` - `BaseAgent object { config, agent_name }` - `config: object { model }` - `model: optional string` - `agent_name: optional "BaseAgent"` - `"BaseAgent"` - `output_type: optional "text" or "integer" or "float" or "boolean"` - `"text"` - `"integer"` - `"float"` - `"boolean"` - `output_values: optional array of string or number or boolean` - `string` - `number` - `boolean` - `rubric_id: optional string` - `rubric_version: optional number` - `alias: optional string` Alias to title the results column. Defaults to the `auto_evaluation_guided_decoding` - `task_type: optional "auto_evaluation.guided_decoding"` - `"auto_evaluation.guided_decoding"` - `AutoEvaluationAgent object { configuration, alias, task_type }` - `configuration: AutoEvaluationAgentTaskRequestWithItemLocator` - `alias: optional string` Alias to title the results column. Defaults to the `auto_evaluation_agent` - `task_type: optional "auto_evaluation.agent"` - `"auto_evaluation.agent"` - `ContributorEvaluationQuestion object { configuration, alias, task_type }` - `configuration: object { layout, question_id, prefill_from, 3 more }` - `layout: Container` - `children: array of Container or Component` The children to be displayed within the container - `Container object { children, direction }` - `Component object { data, label }` - `data: ItemLocator` A pointer to the data in each evaluation item to be displayed within the component - `label: optional string` - `direction: optional "row" or "column"` The axis that children are placed in the container. Based on CSS `flex-direction` (see: https://developer.mozilla.org/en-US/docs/Web/CSS/flex-direction) - `"row"` - `"column"` - `question_id: string` - `prefill_from: optional string` Dataset column to prefill contributor question task result - `queue_id: optional string` The contributor annotation queue to include this task in. Defaults to `default` - `required: optional boolean` Whether the question is required to be answered - `rubric_id: optional string` ID of the rubric to use for scoring this evaluation question - `alias: optional string` Alias to title the results column. Defaults to the `contributor_evaluation_question` - `task_type: optional "contributor_evaluation.question"` - `"contributor_evaluation.question"` - `CustomFunction object { configuration, alias, task_type }` - `configuration: object { function_source, arg_mapping, config_args, outputs }` Configuration for a custom Python function evaluation task. - `function_source: string` Python function source code - `arg_mapping: optional map[string]` Mapping of function parameter names to item locators (e.g. item.field). Auto-derived from function signature if not provided. - `config_args: optional map[unknown]` Literal argument values for function parameters, such as thresholds or RNG seeds. Serialized JSON must be at most 10000 characters. - `outputs: optional array of object { path, alias }` Optional output paths to materialize as separate result columns. If omitted, the function return value is stored only under the task alias/data key. - `path: string` Dot path in the custom function return value to materialize. - `alias: optional string` Result column alias. Defaults to path with dots replaced by underscores. - `alias: optional string` Alias to title the results column. Defaults to the function name. - `task_type: optional "custom_function"` - `"custom_function"` ### Example ```http curl https://api.egp.scale.com/v5/evaluations/$EVALUATION_ID/tasks \ -H 'Content-Type: application/json' \ -H "x-api-key: $SGP_API_KEY" \ -d '{ "task": { "configuration": { "messages": [ { "foo": "bar" } ], "model": "model" }, "task_type": "chat_completion" } }' ``` #### Response ```json { "id": "id", "created_at": "2019-12-27T18:11:19.117Z", "created_by": { "id": "id", "type": "user", "object": "identity" }, "datasets": [ { "id": "id", "created_at": "2019-12-27T18:11:19.117Z", "created_by": { "id": "id", "type": "user", "object": "identity" }, "current_version_num": 0, "name": "name", "tags": [ "string" ], "archived_at": "2019-12-27T18:11:19.117Z", "description": "description", "object": "dataset" } ], "name": "name", "status": "failed", "tags": [ "string" ], "archived_at": "2019-12-27T18:11:19.117Z", "description": "description", "error_count": 0, "metadata": { "foo": "bar" }, "object": "evaluation", "progress": { "items": { "failed": 0, "pending": 0, "successful": 0, "total": 0, "failed_items": [ { "item_id": "item_id", "error": "error", "error_type": "error_type" } ] }, "workflows": { "completed": 0, "failed": 0, "pending": 0, "total": 0 } }, "status_reason": "status_reason", "tasks": [ { "configuration": { "messages": [ { "foo": "bar" } ], "model": "model", "audio": { "foo": "bar" }, "frequency_penalty": -2, "function_call": { "foo": "bar" }, "functions": [ { "foo": "bar" } ], "logit_bias": { "foo": 0 }, "logprobs": true, "max_completion_tokens": 0, "max_tokens": 0, "metadata": { "foo": "string" }, "modalities": [ "string" ], "n": 0, "parallel_tool_calls": true, "prediction": { "foo": "bar" }, "presence_penalty": -2, "reasoning_effort": "reasoning_effort", "response_format": { "foo": "bar" }, "seed": 0, "stop": "string", "store": true, "temperature": 0, "tool_choice": "string", "tools": [ { "foo": "bar" } ], "top_k": 0, "top_logprobs": 0, "top_p": 0 }, "alias": "alias", "task_type": "chat_completion" } ] } ``` ## Update Test Criteria Configuration **patch** `/v5/evaluations/{evaluation_id}/tasks/{alias}` Replace the full configuration of a single test criteria, identified by its alias. The alias must match an existing test criteria on the evaluation, and the replacement configuration is validated against the evaluation's current items before being applied. The request is rejected if the evaluation is archived, if no test criteria matches the alias, or if any contributor annotation task for the evaluation has already been claimed or completed — at that point labelers are in-flight and mutating the task definition would corrupt their work. ### Path Parameters - `evaluation_id: string` - `alias: string` ### Body Parameters - `configuration: map[unknown]` Full replacement for the test criteria's configuration JSON. Only allowed when no contributor annotation tasks for this evaluation have been claimed or completed. ### Returns - `Evaluation object { id, created_at, created_by, 12 more }` - `id: string` The unique identifier of the entity. - `created_at: string` The date and time when the entity was created in ISO format. - `created_by: Identity` The identity that created the entity. - `id: string` - `type: "user" or "service_account"` - `"user"` - `"service_account"` - `object: optional "identity"` - `"identity"` - `datasets: array of Dataset` - `id: string` The unique identifier of the entity. - `created_at: string` The date and time when the entity was created in ISO format. - `created_by: Identity` The identity that created the entity. - `current_version_num: number` - `name: string` - `tags: array of string` The tags associated with the entity - `archived_at: optional string` The date and time when the entity was archived in ISO format. - `description: optional string` - `object: optional "dataset"` - `"dataset"` - `name: string` - `status: "failed" or "completed" or "running"` - `"failed"` - `"completed"` - `"running"` - `tags: array of string` The tags associated with the entity - `archived_at: optional string` The date and time when the entity was archived in ISO format. - `description: optional string` - `error_count: optional number` Number of task errors across all items in this evaluation. - `metadata: optional map[unknown]` Metadata key-value pairs for the evaluation - `object: optional "evaluation"` - `"evaluation"` - `progress: optional EvaluationTasksProgressSchema` Progress of the evaluation's underlying async job - `items: optional object { failed, pending, successful, 2 more }` - `failed: number` - `pending: number` - `successful: number` - `total: number` - `failed_items: optional array of object { item_id, error, error_type }` - `item_id: string` - `error: optional string` - `error_type: optional string` - `workflows: optional object { completed, failed, pending, total }` - `completed: number` - `failed: number` - `pending: number` - `total: number` - `status_reason: optional string` Reason for evaluation status - `tasks: optional array of EvaluationTask` Tasks executed during evaluation. Populated with optional `task` view. - `ChatCompletion object { configuration, alias, task_type }` - `configuration: object { messages, model, audio, 24 more }` - `messages: array of map[unknown] or ItemLocator` openai standard message format - `array of map[unknown]` - `ItemLocator = string` - `model: string` model specified as `model_vendor/model`, for example `openai/gpt-4o` - `audio: optional map[unknown] or ItemLocator` Parameters for audio output. Required when audio output is requested with modalities: ['audio']. - `map[unknown]` - `ItemLocator = string` - `frequency_penalty: optional number or ItemLocator` Number between -2.0 and 2.0. Positive values penalize new tokens based on their existing frequency in the text so far. - `number` - `ItemLocator = string` - `function_call: optional map[unknown] or ItemLocator` Deprecated in favor of tool_choice. Controls which function is called by the model. - `map[unknown]` - `ItemLocator = string` - `functions: optional array of map[unknown] or ItemLocator` Deprecated in favor of tools. A list of functions the model may generate JSON inputs for. - `array of map[unknown]` - `ItemLocator = string` - `logit_bias: optional map[number] or ItemLocator` Modify the likelihood of specified tokens appearing in the completion. Maps tokens to bias values from -100 to 100. - `map[number]` - `ItemLocator = string` - `logprobs: optional boolean or ItemLocator` Whether to return log probabilities of the output tokens or not. - `boolean` - `ItemLocator = string` - `max_completion_tokens: optional number or ItemLocator` An upper bound for the number of tokens that can be generated, including visible output tokens and reasoning tokens. - `number` - `ItemLocator = string` - `max_tokens: optional number or ItemLocator` Deprecated in favor of max_completion_tokens. The maximum number of tokens to generate. - `number` - `ItemLocator = string` - `metadata: optional map[string] or ItemLocator` Developer-defined tags and values used for filtering completions in the dashboard. - `map[string]` - `ItemLocator = string` - `modalities: optional array of string or ItemLocator` Output types that you would like the model to generate for this request. - `array of string` - `ItemLocator = string` - `n: optional number or ItemLocator` How many chat completion choices to generate for each input message. - `number` - `ItemLocator = string` - `parallel_tool_calls: optional boolean or ItemLocator` Whether to enable parallel function calling during tool use. - `boolean` - `ItemLocator = string` - `prediction: optional map[unknown] or ItemLocator` Static predicted output content, such as the content of a text file being regenerated. - `map[unknown]` - `ItemLocator = string` - `presence_penalty: optional number or ItemLocator` Number between -2.0 and 2.0. Positive values penalize tokens based on whether they appear in the text so far. - `number` - `ItemLocator = string` - `reasoning_effort: optional string` For o1 models only. Constrains effort on reasoning. Values: low, medium, high. - `response_format: optional map[unknown] or ItemLocator` An object specifying the format that the model must output. - `map[unknown]` - `ItemLocator = string` - `seed: optional number or ItemLocator` If specified, system will attempt to sample deterministically for repeated requests with same seed. - `number` - `ItemLocator = string` - `stop: optional string or array of string` Up to 4 sequences where the API will stop generating further tokens. - `string` - `array of string` - `store: optional boolean or ItemLocator` Whether to store the output for use in model distillation or evals products. - `boolean` - `ItemLocator = string` - `temperature: optional number or ItemLocator` What sampling temperature to use. Higher values make output more random, lower more focused. - `number` - `ItemLocator = string` - `tool_choice: optional string or map[unknown]` Controls which tool is called by the model. Values: none, auto, required, or specific tool. - `string` - `map[unknown]` - `tools: optional array of map[unknown] or ItemLocator` A list of tools the model may call. Currently, only functions are supported. Max 128 functions. - `array of map[unknown]` - `ItemLocator = string` - `top_k: optional number or ItemLocator` Only sample from the top K options for each subsequent token - `number` - `ItemLocator = string` - `top_logprobs: optional number or ItemLocator` Number of most likely tokens to return at each position, with associated log probability. - `number` - `ItemLocator = string` - `top_p: optional number or ItemLocator` Alternative to temperature. Only tokens comprising top_p probability mass are considered. - `number` - `ItemLocator = string` - `alias: optional string` Alias to title the results column. Defaults to the `chat_completion` - `task_type: optional "chat_completion"` - `"chat_completion"` - `Inference object { configuration, alias, task_type }` - `configuration: object { model, args, inference_configuration }` - `model: string` model specified as `vendor/name` (ex. openai/gpt-5) - `args: optional map[unknown] or ItemLocator` Arguments passed into model - `map[unknown]` - `ItemLocator = string` - `inference_configuration: optional LaunchInferenceConfiguration or ItemLocator` Vendor specific configuration - `LaunchInferenceConfiguration object { num_retries, timeout_seconds }` - `num_retries: optional number` - `timeout_seconds: optional number` - `ItemLocator = string` - `alias: optional string` Alias to title the results column. Defaults to the `inference` - `task_type: optional "inference"` - `"inference"` - `ApplicationVariant object { configuration, alias, task_type }` - `configuration: object { application_variant_id, inputs, history, 2 more }` - `application_variant_id: string` - `inputs: map[unknown] or ItemLocator` Input data for the application. For agents service variants, you must provide inputs as a mapping from `{input_name: input_value}`. For V0 variants, you must specify the node your input should be passed to, structuring your input as `{node_id: {input_name: input_value}}`. - `map[unknown]` - `ItemLocator = string` - `history: optional array of object { request, response, session_data } or ItemLocator` History of the application - `ApplicationRequestResponsePairArray = array of object { request, response, session_data }` - `request: string` Request inputs - `response: string` Response outputs - `session_data: optional map[unknown]` Session data corresponding to the request response pair - `ItemLocator = string` - `operation_metadata: optional map[unknown] or ItemLocator` Arbitrary user-defined metadata that can be attached to the process operations and will be registered in the interaction. - `map[unknown]` - `ItemLocator = string` - `overrides: optional object { concurrent, initial_state, partial_trace, 2 more } or map[object { artifact_ids_filter, artifact_name_regex, type } ] or ItemLocator` Optional overrides for the application - `AgenticApplicationOverrides object { concurrent, initial_state, partial_trace, 2 more }` Execution override options for agentic applications - `concurrent: optional boolean` - `initial_state: optional object { current_node, state }` - `current_node: string` - `state: map[unknown]` - `partial_trace: optional array of object { duration_ms, node_id, operation_input, 5 more }` - `duration_ms: number` - `node_id: string` - `operation_input: string` - `operation_output: string` - `operation_type: string` - `start_timestamp: string` - `workflow_id: string` - `operation_metadata: optional map[unknown]` - `return_span: optional boolean` - `use_channels: optional boolean` - `map[object { artifact_ids_filter, artifact_name_regex, type } ]` - `artifact_ids_filter: optional array of string` - `artifact_name_regex: optional array of string` - `type: optional "knowledge_base_schema"` - `"knowledge_base_schema"` - `ItemLocator = string` - `alias: optional string` Alias to title the results column. Defaults to the `application_variant` - `task_type: optional "application_variant"` - `"application_variant"` - `AgentexOutput object { configuration, alias, task_type }` - `configuration: object { agentex_agent_id, input_column, agent_task_params, 6 more }` - `agentex_agent_id: string` The ID of the Agentex agent to use - `input_column: string or map[unknown] or array of unknown` The dataset column to use as input for the agent - `string` - `map[unknown]` - `array of unknown` - `agent_task_params: optional map[unknown] or ItemLocator` Extra params merged into the Agentex `task/create` call's `params` object and forwarded verbatim to the agent. Required by agents that demand configuration at task creation -- the golden agent, for example, rejects any task whose params omit `config_id`. SGP always pins `is_eval: true`; a caller-supplied `description` overrides the SGP default. Nested `item.`-prefixed strings and `{{item.x}}` templates are resolved per evaluation item, so a per-row `config_id` can come from a dataset column. - `map[unknown]` - `ItemLocator = string` - `completion_mode: optional "first_message" or "turn_quiescence"` How the agent's first turn is judged finished. `first_message` (the default) grades the first non-empty agent text message after the input, which is cheap but grades a streaming harness on whatever text block streamed first. `turn_quiescence` keeps listening while the agent is still producing messages and grades once at least one agent text message exists and nothing new has arrived for `quiescence_seconds` -- the right choice for tool-using agents. Neither mode requires the agent to mark the task complete; a terminal task status always ends the wait, and `timeout_seconds` always bounds it. - `"first_message"` - `"turn_quiescence"` - `deployment_id: optional string` Optional Agentex deployment ID to pin the eval to a specific deployment. When set, RPC traffic routes through /agents/{agent_id}/deployments/{deployment_id}/rpc. When unset, traffic uses the agent's default RPC endpoint, which resolves through the agent's current routing rules on the Agentex side. - `include_traces: optional boolean or ItemLocator` Whether to include trace data in the evaluation results. Traces are read from SGP's own span store for the agent's trace, not from Agentex. - `boolean` - `ItemLocator = string` - `input_mode: optional "text" or "data"` How the resolved `input_column` is delivered to the agent. `text` (the default) sends a TextContent message with the value stringified. `data` sends a DataContent message whose `data` is the value as a JSON object; the resolved value must be an object, or a string that parses to one. Most agents accept text only and reject `data`. - `"text"` - `"data"` - `quiescence_seconds: optional number or ItemLocator` Seconds of no new messages before `completion_mode: turn_quiescence` considers the turn finished. Ignored in `first_message` mode. Should exceed the agent's longest expected gap between messages (a slow tool call), or the turn is graded early. - `number` - `ItemLocator = string` - `timeout_seconds: optional number or ItemLocator` Maximum seconds to wait for the agent's first-turn response per item. If not set, the server-side default of 600s applies. Capped at 1500s to stay within the evaluation item activity's 1800s start-to-close budget. - `number` - `ItemLocator = string` - `alias: optional string` Alias to title the results column. Defaults to the `agentex_output` - `task_type: optional "agentex_output"` - `"agentex_output"` - `Metric object { configuration, alias, task_type }` - `configuration: object { candidate, reference, type } or object { candidate, reference, type } or object { candidate, reference, type } or 4 more` - `Bleu object { candidate, reference, type }` - `candidate: string` - `reference: string` - `type: "bleu"` - `"bleu"` - `Meteor object { candidate, reference, type }` - `candidate: string` - `reference: string` - `type: "meteor"` - `"meteor"` - `CosineSimilarity object { candidate, reference, type }` - `candidate: string` - `reference: string` - `type: "cosine_similarity"` - `"cosine_similarity"` - `F1 object { candidate, reference, type }` - `candidate: string` - `reference: string` - `type: "f1"` - `"f1"` - `Rouge1 object { candidate, reference, type }` - `candidate: string` - `reference: string` - `type: "rouge1"` - `"rouge1"` - `Rouge2 object { candidate, reference, type }` - `candidate: string` - `reference: string` - `type: "rouge2"` - `"rouge2"` - `RougeL object { candidate, reference, type }` - `candidate: string` - `reference: string` - `type: "rougeL"` - `"rougeL"` - `alias: optional string` Alias to title the results column. Defaults to the metric type specified in the configuration - `task_type: optional "metric"` - `"metric"` - `AutoEvaluationQuestion object { configuration, alias, task_type }` - `configuration: object { model, prompt, question_id }` - `model: string` model specified as `model_vendor/model_name` - `prompt: string` - `question_id: string` question to be evaluated - `alias: optional string` Alias to title the results column. Defaults to the `auto_evaluation_question` - `task_type: optional "auto_evaluation.question"` - `"auto_evaluation.question"` - `AutoEvaluationGuidedDecoding object { configuration, alias, task_type }` - `configuration: object { model, prompt, response_format, 3 more } or object { choices, model, prompt, 3 more } or AutoEvaluationAgentTaskRequestWithItemLocator` - `AutoEvaluationStructuredOutputTaskRequestWithItemLocator object { model, prompt, response_format, 3 more }` - `model: string` model specified as `model_vendor/model_name` - `prompt: string` - `response_format: map[unknown]` JSON schema used for structuring the model response - `inference_args: optional map[unknown]` Additional arguments to pass to the inference request - `run_condition: optional object { op, value } or object { path, op } or EqEvaluationRunCondition or 12 more` - `Const object { op, value }` - `op: optional "const"` - `"const"` - `value: optional string or number or boolean` - `string` - `number` - `boolean` - `Var object { path, op }` - `path: string` - `op: optional "var"` - `"var"` - `EqEvaluationRunCondition object { left, right, op }` - `left: unknown` - `right: unknown` - `op: optional "eq"` - `"eq"` - `NeEvaluationRunCondition object { left, right, op }` - `left: unknown` - `right: unknown` - `op: optional "ne"` - `"ne"` - `LtEvaluationRunCondition object { left, right, op }` - `left: unknown` - `right: unknown` - `op: optional "lt"` - `"lt"` - `LteEvaluationRunCondition object { left, right, op }` - `left: unknown` - `right: unknown` - `op: optional "lte"` - `"lte"` - `GtEvaluationRunCondition object { left, right, op }` - `left: unknown` - `right: unknown` - `op: optional "gt"` - `"gt"` - `GteEvaluationRunCondition object { left, right, op }` - `left: unknown` - `right: unknown` - `op: optional "gte"` - `"gte"` - `AndEvaluationRunCondition object { operands, op }` - `operands: array of unknown` - `op: optional "and"` - `"and"` - `OrEvaluationRunCondition object { operands, op }` - `operands: array of unknown` - `op: optional "or"` - `"or"` - `InEvaluationRunCondition object { left, operands, op }` - `left: unknown` - `operands: array of unknown` - `op: optional "in"` - `"in"` - `NotInEvaluationRunCondition object { left, operands, op }` - `left: unknown` - `operands: array of unknown` - `op: optional "not_in"` - `"not_in"` - `NotEvaluationRunCondition object { operands, op }` - `operands: array of unknown` - `op: optional "not"` - `"not"` - `IsNullEvaluationRunCondition object { operands, op }` - `operands: array of unknown` - `op: optional "is_null"` - `"is_null"` - `IsNotNullEvaluationRunCondition object { operands, op }` - `operands: array of unknown` - `op: optional "is_not_null"` - `"is_not_null"` - `system_prompt: optional string` - `AutoEvaluationGuidedDecodingTaskRequestWithItemLocator object { choices, model, prompt, 3 more }` - `choices: array of string` Choices array cannot be empty - `model: string` model specified as `model_vendor/model_name` - `prompt: string` - `inference_args: optional map[unknown]` Additional arguments to pass to the inference request - `run_condition: optional object { op, value } or object { path, op } or EqEvaluationRunCondition or 12 more` - `Const object { op, value }` - `op: optional "const"` - `"const"` - `value: optional string or number or boolean` - `string` - `number` - `boolean` - `Var object { path, op }` - `path: string` - `op: optional "var"` - `"var"` - `EqEvaluationRunCondition object { left, right, op }` - `NeEvaluationRunCondition object { left, right, op }` - `LtEvaluationRunCondition object { left, right, op }` - `LteEvaluationRunCondition object { left, right, op }` - `GtEvaluationRunCondition object { left, right, op }` - `GteEvaluationRunCondition object { left, right, op }` - `AndEvaluationRunCondition object { operands, op }` - `OrEvaluationRunCondition object { operands, op }` - `InEvaluationRunCondition object { left, operands, op }` - `NotInEvaluationRunCondition object { left, operands, op }` - `NotEvaluationRunCondition object { operands, op }` - `IsNullEvaluationRunCondition object { operands, op }` - `IsNotNullEvaluationRunCondition object { operands, op }` - `system_prompt: optional string` - `AutoEvaluationAgentTaskRequestWithItemLocator object { definition, name, output_rules, 6 more }` - `definition: string` - `name: string` - `output_rules: array of string` - `data_fields: optional array of string` - `designated_to: optional object { config, agent_name } or object { config, agent_name } or object { config, agent_name } or object { config, agent_name }` - `ApeAgent object { config, agent_name }` - `config: object { model, temperature }` - `model: optional string` - `temperature: optional number` - `agent_name: optional "APEAgent"` - `"APEAgent"` - `IfAgent object { config, agent_name }` - `config: object { model }` - `model: optional string` - `agent_name: optional "IFAgent"` - `"IFAgent"` - `TruthfulnessAgent object { config, agent_name }` - `config: object { model }` - `model: optional string` - `agent_name: optional "TruthfulnessAgent"` - `"TruthfulnessAgent"` - `BaseAgent object { config, agent_name }` - `config: object { model }` - `model: optional string` - `agent_name: optional "BaseAgent"` - `"BaseAgent"` - `output_type: optional "text" or "integer" or "float" or "boolean"` - `"text"` - `"integer"` - `"float"` - `"boolean"` - `output_values: optional array of string or number or boolean` - `string` - `number` - `boolean` - `rubric_id: optional string` - `rubric_version: optional number` - `alias: optional string` Alias to title the results column. Defaults to the `auto_evaluation_guided_decoding` - `task_type: optional "auto_evaluation.guided_decoding"` - `"auto_evaluation.guided_decoding"` - `AutoEvaluationAgent object { configuration, alias, task_type }` - `configuration: AutoEvaluationAgentTaskRequestWithItemLocator` - `alias: optional string` Alias to title the results column. Defaults to the `auto_evaluation_agent` - `task_type: optional "auto_evaluation.agent"` - `"auto_evaluation.agent"` - `ContributorEvaluationQuestion object { configuration, alias, task_type }` - `configuration: object { layout, question_id, prefill_from, 3 more }` - `layout: Container` - `children: array of Container or Component` The children to be displayed within the container - `Container object { children, direction }` - `Component object { data, label }` - `data: ItemLocator` A pointer to the data in each evaluation item to be displayed within the component - `label: optional string` - `direction: optional "row" or "column"` The axis that children are placed in the container. Based on CSS `flex-direction` (see: https://developer.mozilla.org/en-US/docs/Web/CSS/flex-direction) - `"row"` - `"column"` - `question_id: string` - `prefill_from: optional string` Dataset column to prefill contributor question task result - `queue_id: optional string` The contributor annotation queue to include this task in. Defaults to `default` - `required: optional boolean` Whether the question is required to be answered - `rubric_id: optional string` ID of the rubric to use for scoring this evaluation question - `alias: optional string` Alias to title the results column. Defaults to the `contributor_evaluation_question` - `task_type: optional "contributor_evaluation.question"` - `"contributor_evaluation.question"` - `CustomFunction object { configuration, alias, task_type }` - `configuration: object { function_source, arg_mapping, config_args, outputs }` Configuration for a custom Python function evaluation task. - `function_source: string` Python function source code - `arg_mapping: optional map[string]` Mapping of function parameter names to item locators (e.g. item.field). Auto-derived from function signature if not provided. - `config_args: optional map[unknown]` Literal argument values for function parameters, such as thresholds or RNG seeds. Serialized JSON must be at most 10000 characters. - `outputs: optional array of object { path, alias }` Optional output paths to materialize as separate result columns. If omitted, the function return value is stored only under the task alias/data key. - `path: string` Dot path in the custom function return value to materialize. - `alias: optional string` Result column alias. Defaults to path with dots replaced by underscores. - `alias: optional string` Alias to title the results column. Defaults to the function name. - `task_type: optional "custom_function"` - `"custom_function"` ### Example ```http curl https://api.egp.scale.com/v5/evaluations/$EVALUATION_ID/tasks/$ALIAS \ -X PATCH \ -H 'Content-Type: application/json' \ -H "x-api-key: $SGP_API_KEY" \ -d '{ "configuration": { "foo": "bar" } }' ``` #### Response ```json { "id": "id", "created_at": "2019-12-27T18:11:19.117Z", "created_by": { "id": "id", "type": "user", "object": "identity" }, "datasets": [ { "id": "id", "created_at": "2019-12-27T18:11:19.117Z", "created_by": { "id": "id", "type": "user", "object": "identity" }, "current_version_num": 0, "name": "name", "tags": [ "string" ], "archived_at": "2019-12-27T18:11:19.117Z", "description": "description", "object": "dataset" } ], "name": "name", "status": "failed", "tags": [ "string" ], "archived_at": "2019-12-27T18:11:19.117Z", "description": "description", "error_count": 0, "metadata": { "foo": "bar" }, "object": "evaluation", "progress": { "items": { "failed": 0, "pending": 0, "successful": 0, "total": 0, "failed_items": [ { "item_id": "item_id", "error": "error", "error_type": "error_type" } ] }, "workflows": { "completed": 0, "failed": 0, "pending": 0, "total": 0 } }, "status_reason": "status_reason", "tasks": [ { "configuration": { "messages": [ { "foo": "bar" } ], "model": "model", "audio": { "foo": "bar" }, "frequency_penalty": -2, "function_call": { "foo": "bar" }, "functions": [ { "foo": "bar" } ], "logit_bias": { "foo": 0 }, "logprobs": true, "max_completion_tokens": 0, "max_tokens": 0, "metadata": { "foo": "string" }, "modalities": [ "string" ], "n": 0, "parallel_tool_calls": true, "prediction": { "foo": "bar" }, "presence_penalty": -2, "reasoning_effort": "reasoning_effort", "response_format": { "foo": "bar" }, "seed": 0, "stop": "string", "store": true, "temperature": 0, "tool_choice": "string", "tools": [ { "foo": "bar" } ], "top_k": 0, "top_logprobs": 0, "top_p": 0 }, "alias": "alias", "task_type": "chat_completion" } ] } ```