Day 2 - Operating Real-Time Endpoints: Configuration, Auto Scaling, and Multi-Model Endpoints

📌 Key Takeaways

• Real-time inference is split into three resources, Model → EndpointConfig → Endpoint, and this separation is the foundation of zero-downtime replacement and traffic splitting.
• Zero-downtime deployment means creating a new EndpointConfig and swapping it in with UpdateEndpoint. Start the new fleet → shift traffic → shut down the old fleet.
• Auto Scaling is applied per variant through Application Auto Scaling, and the default metric is SageMakerVariantInvocationsPerInstance + Target Tracking.
• MinCapacity cannot be 0. If you need true scale-to-zero, use Serverless or Async.
• The distinction between MME (same container, thousands of dynamically loaded models), MCE (different containers), and an inference pipeline (preprocessing → prediction in series) is in scope for the exam.

The Three-Layer Structure of an Endpoint

As we saw yesterday, the real-time endpoint carries the heaviest operational burden of the four inference options. To run a stable, low-latency service on always-on instances, you need to understand configuration, scaling, and cost control together. The starting point is the fact that SageMaker splits resources into three layers.

Model              : Training output (S3 model artifact) + inference container image
EndpointConfig     : Defines which Model to run, on which instances / at what ratio (ProductionVariant)
Endpoint           : The actual HTTPS URL. It points to an EndpointConfig

from sagemaker.model import Model
 
model = Model(image_uri=image, model_data=s3_artifact, role=role)
model.deploy(
    initial_instance_count=2,
    instance_type="ml.m5.xlarge",
    endpoint_name="prod-endpoint",
)

The Path an Inference Request Takes

Client
   │  InvokeEndpoint (HTTPS, payload)
   ▼
[Endpoint  prod-endpoint]  ← Name is fixed. The client only knows this URL
   │  Look up the EndpointConfig it currently points to
   ▼
[EndpointConfig  v3]
   ├─ ProductionVariant A (ml.m5.xlarge × 2, weight 90)
   └─ ProductionVariant B (ml.m5.xlarge × 1, weight 10)
                │
                ▼
        [Inference container] ─ Load model artifact → predict
                │
                ▼
        Response + CloudWatch metrics recorded

The key point is that the Endpoint and the EndpointConfig are separate. When you create a new EndpointConfig and swap it in with UpdateEndpoint, SageMaker starts a new fleet, shifts traffic, and then cleans up the old fleet — there is no downtime.

A single EndpointConfig can hold multiple ProductionVariants, and you specify the traffic ratio for each variant with InitialVariantWeight. This weight-based split is the foundation of A/B testing and canary deployments, which we cover tomorrow.

⚠️ Trap: In a question about "zero-downtime deployment," an answer choice that deletes the endpoint and recreates it with the same name is wrong. Downtime occurs during the recreation. Choices that overwrite the S3 artifact or SSH into an instance to change files do not work either.

Auto Scaling: Matching the Instance Count to Traffic

To reduce the trap of always-on cost, the instance count must move automatically with traffic. SageMaker supports scaling per variant through Application Auto Scaling.

import boto3
client = boto3.client("application-autoscaling")
 
client.register_scalable_target(
    ServiceNamespace="sagemaker",
    ResourceId="endpoint/prod-endpoint/variant/AllTraffic",
    ScalableDimension="sagemaker:variant:DesiredInstanceCount",
    MinCapacity=2, MaxCapacity=10,
)
client.put_scaling_policy(
    PolicyName="invocations-per-instance",
    PolicyType="TargetTrackingScaling",
    TargetTrackingScalingPolicyConfiguration={
        "TargetValue": 1000.0,
        "PredefinedMetricSpecification": {
            "PredefinedMetricType": "SageMakerVariantInvocationsPerInstance"
        },
        "ScaleInCooldown": 300, "ScaleOutCooldown": 60,
    },
)

Comparing Policy Types

• Recommended default metric: SageMakerVariantInvocationsPerInstance — tracks the number of invocations per second per instance against the target value.
• Cooldown: Keep ScaleOut short and ScaleIn long. This prevents flapping, where capacity is repeatedly reduced and increased as traffic fluctuates. The example above uses ScaleOut 60 seconds / ScaleIn 300 seconds.
• MinCapacity ≥ 1: It never drops to 0.

Criteria for Choosing a Scaling Metric

⚠️ Trap: For a question saying "I want cost to be 0 when there is no traffic," you must not choose a real-time endpoint + Auto Scaling. Since MinCapacity cannot be 0, Serverless Inference or Asynchronous Inference is the answer.

Multi-Model Endpoint (MME): Thousands of Models on One Endpoint

When models balloon into the hundreds or thousands per customer or per region, running each on a separate endpoint is unmanageable in both cost and operations. An MME loads multiple models onto shared instances and loads them dynamically at invocation time.

from sagemaker.multidatamodel import MultiDataModel
 
mme = MultiDataModel(
    name="mme-endpoint",
    model_data_prefix="s3://my-bucket/models/",  # Prefix where model artifacts are placed
    image_uri=image, role=role,
)
predictor = mme.deploy(initial_instance_count=2, instance_type="ml.m5.xlarge")
# At invocation time, specify which model with TargetModel
predictor.predict(data, target_model="customer-123.tar.gz")

Request (TargetModel=customer-123.tar.gz)
        │
        ▼
   Already loaded in memory?
   ├─ Yes → Infer immediately (low latency)
   └─ No  → Load from S3 → infer (cold latency on first call)
                  │
                  └─ When memory runs short, unload the least recently used model (LRU)

• Behavior: When a request arrives, the model is loaded into memory (reused if already there), and when memory runs short, the least recently used models are unloaded first (LRU).
• Benefit: A handful of instances host thousands of models → cost drops significantly.
• Trade-off: A model that has not been invoked for a long time incurs cold-start latency on its first call, and all models must share the same framework/container.

Distinguishing MME, MCE, and Inference Pipelines

To put models from different frameworks on one endpoint, you use multi-container, not multi-model.

Multi-Model Endpoint (MME)    : Same container, N different model artifacts, dynamic loading
Multi-Container Endpoint (MCE): N different containers. Direct (individual invocation) or Serial mode
Inference Pipeline            : Containers chained in series (preprocessing → prediction → postprocessing), handled in one request

"There are many models" alone does not make MME the answer. It comes with the premise that all models share the same framework/container. If the passage hints that the frameworks differ, it points to MCE. Also, an MME's instances are not 0 and are not GPU-only.

A typical inference pipeline example: chain a Scikit-learn preprocessing container → an XGBoost prediction container in series, guaranteeing that the same preprocessing as training is applied at inference time (preventing training-serving skew). Having the client call two separate endpoints in order leaves network round trips and consistency problems.

Which hosting structure should I choose?
├─ Cost 0 when there is no traffic? ─→ Serverless / Async (not a real-time endpoint)
├─ Must preprocessing and prediction be bundled in one request?
│    └─ Yes → Inference pipeline (containers chained in series)
├─ Are the models in the same framework/container?
│    ├─ Yes + hundreds to thousands of models → MME (specify with TargetModel, LRU loading)
│    └─ No → MCE Direct mode (invoke each container individually)
└─ Few models, each with a different scale/SLA → individual real-time endpoints

"Thousands of similar models + same framework + cost reduction" → MME. "Bundle preprocessing and prediction into one request" → inference pipeline. "Models from different frameworks on one endpoint, called separately" → MCE Direct mode. You must be able to tell these three apart.

Key Operational Monitoring Metrics

ModelLatency           : Time the model container spent on inference
OverheadLatency        : SageMaker overhead (request/response handling)
Invocations            : Number of invocations
Invocation4XX/5XX      : Client/server errors
CPU/GPU/Memory Utilization : Resource utilization → basis for deciding scaling and instance type

If latency is large in ModelLatency, optimize the model or move up an instance. If it is large in OverheadLatency, suspect payload and serialization problems.

Passage Clue → Answer Mapping

In the next post, we cover inference optimization with Neo, Inferentia, and inference pipelines to make endpoints faster and cheaper.

📖 Glossary

• ProductionVariant : One unit inside an EndpointConfig that defines "which model, on how many of which instances, with what percent of traffic."
• InitialVariantWeight : The weight that splits traffic among multiple variants. It is the basis of A/B testing and canary deployments.
• UpdateEndpoint : The API that keeps the endpoint name and swaps only the EndpointConfig it points to. The standard method for zero-downtime replacement.
• Application Auto Scaling : The shared AWS scaling service that automatically adjusts the instance count of SageMaker variants.
• Target Tracking : A scaling policy where you just state "keep this metric at this value" and it scales out and in on its own.
• SageMakerVariantInvocationsPerInstance : The number of invocations one instance handles per second. The default metric for scaling real-time endpoints.
• Cooldown : The time to wait after a scaling adjustment before the next one. Keeping ScaleIn long prevents flapping.
• Flapping : The instability of repeatedly reducing and increasing instances as traffic fluctuates.
• Multi-model endpoint (MME) : A way of hosting many models that share the same container on one endpoint and loading them into memory at invocation time.
• LRU unload : A way of freeing space when memory runs short by taking down the model that has gone unused the longest.