Update a custom model

PATCH/v5/models/{model_id}

Update a custom model record; vendor-configuration changes are applied asynchronously by redeploying the model.

This supports three kinds of update: changing model metadata only, renaming the model, and changing the vendor configuration. A vendor-configuration change is asynchronous — it puts the model back into a deploying status, records an update job, and starts a Temporal workflow to redeploy, so the new configuration is not live when this returns; metadata-only and rename changes take effect immediately. The vendor configuration supplied must match the model’s own vendor (launch or llmengine), and only those two vendors are supported. A model that is currently deploying cannot be modified and the request fails until deployment finishes. When renaming with on_conflict set to swap, the name is exchanged with an existing model of the same name and vendor instead of failing on the uniqueness constraint.

Path ParametersExpand Collapse

model_id: string

Body ParametersJSONExpand Collapse

model: object { model_metadata } or object { vendor_configuration, model_metadata } or object { name, on_conflict }

One of the following:

DefaultModelPatchRequest object { model_metadata }

model_metadata: optional map[unknown]

ModelConfigurationPatchRequest object { vendor_configuration, model_metadata }

vendor_configuration: object { model_image, model_infra } or object { chat_template_override, checkpoint_path, cpus, 20 more }

One of the following:

PartialLaunchVendorConfiguration object { model_image, model_infra }

model_image: optional object { command, env_vars, healthcheck_route, 9 more }

command: optional array of string

env_vars: optional map[unknown]

healthcheck_route: optional string

predict_route: optional string

readiness_delay: optional number

registry: optional string

repository: optional string

request_schema: optional map[unknown]

response_schema: optional map[unknown]

streaming_command: optional array of string

streaming_predict_route: optional string

tag: optional string

model_infra: optional object { cpus, endpoint_type, gpu_type, 9 more }

cpus: optional string or number

One of the following:

string

number

endpoint_type: optional "async" or "sync" or "streaming"

One of the following:

"async"

"sync"

"streaming"

gpu_type: optional "nvidia-tesla-t4" or "nvidia-ampere-a10" or "nvidia-ampere-a100" or 4 more

One of the following:

"nvidia-tesla-t4"

"nvidia-ampere-a10"

"nvidia-ampere-a100"

"nvidia-ampere-a100e"

"nvidia-hopper-h100"

"nvidia-hopper-h100-1g20gb"

"nvidia-hopper-h100-3g40gb"

gpus: optional number

high_priority: optional boolean

labels: optional map[string]

max_workers: optional number

memory: optional string

min_workers: optional number

per_worker: optional number

public_inference: optional boolean

storage: optional string

PartialLlmEngineVendorConfiguration object { chat_template_override, checkpoint_path, cpus, 20 more }

chat_template_override: optional string

checkpoint_path: optional string

cpus: optional number

default_callback_url: optional string

endpoint_type: optional string

gpu_type: optional string

gpus: optional number

high_priority: optional boolean

inference_framework: optional string

inference_framework_image_tag: optional string

labels: optional map[string]

max_workers: optional number

memory: optional string

min_workers: optional number

model: optional string

nodes_per_worker: optional number

num_shards: optional number

per_worker: optional number

post_inference_hooks: optional array of string

public_inference: optional boolean

quantize: optional string

source: optional string

storage: optional string

model_metadata: optional map[unknown]

SwapNamesModelPatchRequest object { name, on_conflict }

name: string

on_conflict: optional "error" or "swap"

One of the following:

"error"

"swap"

ReturnsExpand Collapse

InferenceModel object { id, created_at, created_by_identity_type, 10 more }

id: string

The unique identifier of the entity.

created_at: string

The date and time when the entity was created in ISO format.

formatdate-time

created_by_identity_type: "user" or "service_account"

The type of identity that created the entity.

One of the following:

"user"

"service_account"

created_by_user_id: string

The user who originally created the entity.

model_type: InferenceModelType

One of the following:

"generic"

"completion"

"chat_completion"

model_vendor: InferenceModelVendor

One of the following:

"openai"

"cohere"

"vertex_ai"

"anthropic"

"azure"

"gemini"

"launch"

"llmengine"

"model_zoo"

"bedrock"

"xai"

"fireworks_ai"

name: string

status: "failed" or "ready" or "deploying" or "deployment_timeout"

One of the following:

"failed"

"ready"

"deploying"

"deployment_timeout"

model_availability: optional InferenceModelAvailability

One of the following:

"unknown"

"available"

"unavailable"

model_metadata: optional map[unknown]

object: optional "model"

status_reason: optional string

vendor_configuration: optional LaunchVendorConfiguration { model_image, model_infra } or LlmEngineVendorConfiguration { model, chat_template_override, checkpoint_path, 20 more }

One of the following:

LaunchVendorConfiguration object { model_image, model_infra }

model_image: object { command, registry, repository, 9 more }

command: array of string

registry: string

repository: string

tag: string

env_vars: optional map[unknown]

healthcheck_route: optional string

predict_route: optional string

readiness_delay: optional number

request_schema: optional map[unknown]

response_schema: optional map[unknown]

streaming_command: optional array of string

streaming_predict_route: optional string

model_infra: object { cpus, endpoint_type, gpu_type, 9 more }

cpus: optional string or number

One of the following:

string

number

endpoint_type: optional "async" or "sync" or "streaming"

One of the following:

"async"

"sync"

"streaming"

gpu_type: optional "nvidia-tesla-t4" or "nvidia-ampere-a10" or "nvidia-ampere-a100" or 4 more

One of the following:

"nvidia-tesla-t4"

"nvidia-ampere-a10"

"nvidia-ampere-a100"

"nvidia-ampere-a100e"

"nvidia-hopper-h100"

"nvidia-hopper-h100-1g20gb"

"nvidia-hopper-h100-3g40gb"

gpus: optional number

high_priority: optional boolean

labels: optional map[string]

max_workers: optional number

memory: optional string

min_workers: optional number

per_worker: optional number

public_inference: optional boolean

storage: optional string

LlmEngineVendorConfiguration object { model, chat_template_override, checkpoint_path, 20 more }

model: string

chat_template_override: optional string

checkpoint_path: optional string

cpus: optional number

default_callback_url: optional string

endpoint_type: optional string

gpu_type: optional string

gpus: optional number

high_priority: optional boolean

inference_framework: optional string

inference_framework_image_tag: optional string

labels: optional map[string]

max_workers: optional number

memory: optional string

min_workers: optional number

nodes_per_worker: optional number

num_shards: optional number

per_worker: optional number

post_inference_hooks: optional array of string

public_inference: optional boolean

quantize: optional string

source: optional string

storage: optional string

Update a custom model

curl https://api.egp.scale.com/v5/models/$MODEL_ID \
    -X PATCH \
    -H 'Content-Type: application/json' \
    -H "x-api-key: $SGP_API_KEY" \
    -d '{
          "model_metadata": {
            "foo": "bar"
          }
        }'

{
  "id": "id",
  "created_at": "2019-12-27T18:11:19.117Z",
  "created_by_identity_type": "user",
  "created_by_user_id": "created_by_user_id",
  "model_type": "generic",
  "model_vendor": "openai",
  "name": "name",
  "status": "failed",
  "model_availability": "unknown",
  "model_metadata": {
    "foo": "bar"
  },
  "object": "model",
  "status_reason": "status_reason",
  "vendor_configuration": {
    "model_image": {
      "command": [
        "string"
      ],
      "registry": "registry",
      "repository": "repository",
      "tag": "tag",
      "env_vars": {
        "foo": "bar"
      },
      "healthcheck_route": "healthcheck_route",
      "predict_route": "predict_route",
      "readiness_delay": 0,
      "request_schema": {
        "foo": "bar"
      },
      "response_schema": {
        "foo": "bar"
      },
      "streaming_command": [
        "string"
      ],
      "streaming_predict_route": "streaming_predict_route"
    },
    "model_infra": {
      "cpus": "string",
      "endpoint_type": "async",
      "gpu_type": "nvidia-tesla-t4",
      "gpus": 0,
      "high_priority": true,
      "labels": {
        "foo": "string"
      },
      "max_workers": 0,
      "memory": "memory",
      "min_workers": 0,
      "per_worker": 0,
      "public_inference": true,
      "storage": "storage"
    }
  }
}

Returns Examples

{
  "id": "id",
  "created_at": "2019-12-27T18:11:19.117Z",
  "created_by_identity_type": "user",
  "created_by_user_id": "created_by_user_id",
  "model_type": "generic",
  "model_vendor": "openai",
  "name": "name",
  "status": "failed",
  "model_availability": "unknown",
  "model_metadata": {
    "foo": "bar"
  },
  "object": "model",
  "status_reason": "status_reason",
  "vendor_configuration": {
    "model_image": {
      "command": [
        "string"
      ],
      "registry": "registry",
      "repository": "repository",
      "tag": "tag",
      "env_vars": {
        "foo": "bar"
      },
      "healthcheck_route": "healthcheck_route",
      "predict_route": "predict_route",
      "readiness_delay": 0,
      "request_schema": {
        "foo": "bar"
      },
      "response_schema": {
        "foo": "bar"
      },
      "streaming_command": [
        "string"
      ],
      "streaming_predict_route": "streaming_predict_route"
    },
    "model_infra": {
      "cpus": "string",
      "endpoint_type": "async",
      "gpu_type": "nvidia-tesla-t4",
      "gpus": 0,
      "high_priority": true,
      "labels": {
        "foo": "string"
      },
      "max_workers": 0,
      "memory": "memory",
      "min_workers": 0,
      "per_worker": 0,
      "public_inference": true,
      "storage": "storage"
    }
  }
}