Day 1 - Inference Options: Real-time vs Serverless vs Asynchronous vs Batch Transform

📌 Key Takeaways

• SageMaker does not have a single way to deploy. You pick one of four options based on traffic pattern · payload size · latency requirement · cost structure.
• Real-time: Sustained traffic + millisecond latency. 6MB payload, 60-second response. You are billed continuously even with no traffic.
• Serverless: Intermittent, unpredictable traffic. Zero idle cost, but there is a cold start, a 4MB payload, and no GPU support.
• Asynchronous: Large payloads (up to 1GB) + long processing (up to 1 hour). S3 queuing + SNS notification, can scale to zero.
• Batch Transform: Large offline batch inference that needs no real-time response. No endpoint, you are billed only for job run time.

Overview of the Four Inference Options

The MLS-C01 "which inference method should you choose" type of question is really a question of memorizing these four lines of selection criteria.

Option            Latency     Traffic pattern          Payload/time limit        Billing
─────────────────────────────────────────────────────────────────────────
Real-time         ms          Sustained, steady        6MB, 60s                 Instance uptime (always on)
Serverless        seconds     Intermittent, variable   4MB, 60s                 Requests + duration (idle $0)
Asynchronous      near-RT     Large payloads, queuing  1GB, up to 1 hour        Instance hours (can scale to 0)
Batch Transform   offline     Bulk batch processing    100MB/record, large sets Job run time only

This single table is the heart of today's study, and nearly every scenario you will meet on the exam reduces to these four lines.

Choosing an inference option first splits "online or offline," and if online, branches on "is traffic sustained or intermittent" and "is the payload large or small." The exam always drops these three axes (latency requirement, traffic pattern, payload size) in as keywords.

Real-time Endpoint: Sustained Traffic + Low Latency

A real-time endpoint puts the model on always-on (provisioned) instances and responds to HTTPS requests in milliseconds.

predictor = model.deploy(
    initial_instance_count=2,
    instance_type="ml.m5.xlarge",
    endpoint_name="fraud-rt-endpoint",
)
response = predictor.predict(payload)  # synchronous call, immediate response

• When to use: Online services where a user waits for the response, such as recommendation systems, fraud detection, and chatbots
• Constraints: 6MB payload, response within 60 seconds
• Cost trap: If the instances are on, you keep getting billed even with no traffic. It is inefficient for intermittent traffic.

Serverless Inference: Intermittent/Unpredictable Traffic

Serverless inference does not make you manage instances. Compute scales out automatically when requests arrive and shrinks to 0 when there are none, so idle cost is zero.

from sagemaker.serverless import ServerlessInferenceConfig
 
serverless_config = ServerlessInferenceConfig(
    memory_size_in_mb=2048,
    max_concurrency=10,
)
predictor = model.deploy(serverless_inference_config=serverless_config)

• When to use: Internal tools called only a few times a day, new services with erratic traffic
• Trade-off: There is a cold start. The first request is delayed because of container startup.
• Constraints: 4MB payload, no GPU support (CPU/memory based)

⚠️ Pitfall: If the question has a strict latency SLO such as "guarantee p99 latency of 50ms," Serverless is the wrong answer. The cold start creates tail latency.

Asynchronous Inference: Large Payloads + Long Processing Time

Asynchronous inference puts requests in an S3-based queue, and when processing finishes it writes the result to S3 and sends a notification (SNS). The caller does not wait for an immediate response.

from sagemaker.async_inference import AsyncInferenceConfig
 
async_config = AsyncInferenceConfig(
    output_path="s3://my-bucket/async-output/",
    notification_config={"SuccessTopic": "arn:aws:sns:...:success"},
)
predictor = model.deploy(async_inference_config=async_config)
response = predictor.predict_async(input_path="s3://my-bucket/input/large.json")

• When to use: Jobs with large payloads (up to 1GB) and long processing (up to 1 hour), such as high-resolution image/video analysis and large NLP document inference
• Key advantage: It can scale instances to zero (scale to zero) when there is no traffic, saving cost
• Near-real-time: Not immediate, but it also does not hold requests for a long time and process them in bulk like a batch.

Batch Transform: Large Offline Batch Inference

Batch transform does not bring up an endpoint. It runs batch inference over an entire large dataset in S3, writes the result to S3, and automatically releases the resources when the job finishes.

transformer = model.transformer(
    instance_count=4,
    instance_type="ml.m5.xlarge",
    output_path="s3://my-bucket/batch-output/",
)
transformer.transform(
    data="s3://my-bucket/input-dataset/",
    content_type="text/csv",
    split_type="Line",
)

• When to use: Large-scale processing that needs no real-time response, such as refreshing churn scores for all customers every night or batch predictions for a monthly report
• Cost: Billed only while the job runs. There is no always-on endpoint cost.
• Not a constraint: Latency does not matter at all. There is no user waiting for a response.

When you see the keywords "no real-time response needed + large data," it is almost always Batch Transform. Conversely, "the user sees the result immediately + steady traffic" means Real-time. The gray zone between the two (large payload, long processing, queuing acceptable) falls to Asynchronous, and if traffic is rare and you can tolerate a cold start, it falls to Serverless.

Comparing Operational Characteristics

Selection Table: When to Use / When Not to Use

⚠️ Pitfall: Asynchronous and Batch Transform are both "not an immediate answer + S3 results," so they get confused. The fork in the road is the unit of request. If you queue individual requests arriving online, it is Asynchronous, and if you run the entire dataset already accumulated in S3 in one go, it is Batch Transform.

Selection Decision Flow

The decision tree to run in your head in the exam room is as follows.

1. Is a real-time response needed?
   No → bulk batch? → Batch Transform
2. Yes (online inference)
   ├ Large payload (>6MB) / long processing (>60s)? → Asynchronous
   ├ Intermittent/unpredictable traffic + cold start OK? → Serverless
   └ Sustained traffic + low latency essential? → Real-time

This tree filters from the strongest constraint down, in the order payload size → traffic pattern → latency requirement. If the payload is 1GB, Real-time and Serverless are physically impossible, so there is no need to even weigh the traffic pattern.

[Keywords to pull from the question]
        │
 ① "Is the user waiting for the result?"
        ├─ No, and the whole S3 dataset ──────→ Batch Transform
        └─ Yes / per individual request ↓
        │
 ② "Do payload and processing time exceed 6MB and 60 seconds?"
        ├─ Yes (up to 1GB · 1 hour, completion notification needed) ─→ Asynchronous
        └─ No ↓
        │
 ③ "Is traffic intermittent and do you want idle cost at zero?"
        ├─ Yes (accept cold start, within 4MB, CPU) ─→ Serverless
        └─ No ↓
        │
 ④ "Sustained traffic + strict low latency (p99)?" ───→ Real-time

Cost Perspective Summary

Zero idle cost       : Serverless, Asynchronous (scale to 0), Batch Transform
Always-on billing    : Real-time (as long as the instances are on)
Predictable fixed cost: Real-time (capacity is reserved in advance, so it is stable)

In a cost optimization question, if the situation "traffic is intermittent but Real-time is in use" appears, switching to Serverless or Asynchronous is almost always the answer.

A choice saying "attaching Auto Scaling to Real-time makes idle cost zero" is wrong. Even with Auto Scaling, the minimum number of instances must stay on, so a complete $0 is hard. If you need a true $0, it is Serverless.

Symptoms and Corrections of a Wrongly Chosen Option

Question Clue → Answer Mapping

The four inference options are not competing alternatives; they are tools for different traffic, payload, and latency scenarios. Catch the keywords laid out in the question (sustained/intermittent, large payload, immediate response, bulk offline, idle cost) and plug them into the decision tree above.

Tomorrow we cover the configuration, Auto Scaling, and multi-model patterns of the Real-time endpoint, which carries the heaviest operational burden.

📖 Glossary

• Endpoint: An always-on service address where a model is brought up so it can be invoked over HTTPS. Batch Transform does not create one.
• Payload: The input data carried in a single inference request. Its size limit is the strongest constraint in choosing an option.
• Synchronous call / asynchronous call: Waiting until the response arrives / submitting only the request and receiving the result later via S3 or a notification.
• Cold start: The phenomenon where the first request is slow because a container has to be brought up from an idle state. Serverless's signature trade-off.
• Scale to zero: Shrinking instances to 0 when there is no traffic to bring cost close to 0.
• Idle cost: Money spent on instances that are on even though there are no requests. Real-time's signature trap.
• Queuing: A method of putting requests in a queue and processing them in order. The working principle of Asynchronous.
• SNS notification: AWS messaging that announces processing completion or failure through a topic. Used for completion notices in asynchronous inference.
• Batch Transform job: A unit of execution that runs batch inference over a whole S3 dataset without an endpoint and releases resources when finished.
• p99 latency: The response time at the level of the slowest 1 of 100 requests. Used when talking about a tail latency SLO.