Class: Aws::SageMaker::Types::ProductionVariantRoutingConfig

Inherits:

Struct

Object
Struct
Aws::SageMaker::Types::ProductionVariantRoutingConfig

show all

Includes:: Aws::Structure

Defined in:: lib/aws-sdk-sagemaker/types.rb

Overview

Settings that control how the endpoint routes incoming traffic to the instances that the endpoint hosts.

Constant Summary collapse

SENSITIVE =

[]

Instance Attribute Summary collapse

#prefix_aware_routing_config ⇒ Types::PrefixAwareRoutingConfig
The configuration for prefix-aware routing.
#routing_strategy ⇒ String
Sets how the endpoint routes incoming traffic:.

Instance Attribute Details

#prefix_aware_routing_config ⇒ `Types::PrefixAwareRoutingConfig`

The configuration for prefix-aware routing. Specify this parameter only when you set RoutingStrategy to PREFIX_AWARE.

Returns:

# File 'lib/aws-sdk-sagemaker/types.rb', line 47642

class ProductionVariantRoutingConfig < Struct.new(
  :routing_strategy,
  :prefix_aware_routing_config)
  SENSITIVE = []
  include Aws::Structure
end

#routing_strategy ⇒ `String`

Sets how the endpoint routes incoming traffic:

LEAST_OUTSTANDING_REQUESTS: The endpoint routes requests to the specific instances that have more capacity to process them.
RANDOM: The endpoint routes each request to a randomly chosen instance.
PREFIX_AWARE: The endpoint routes requests that share the same prompt prefix to the same instance. When the number of in-flight requests on the selected instance reaches the configured threshold, the endpoint routes the request to an instance with more available capacity.