Day 4 - Batch & Serverless Inference Deep Dive: Throughput Tuning & Cost Tradeoffs

📌 Key Takeaways

• Batch and serverless share zero always-on cost, but their uses are opposite — bulk batch vs intermittent individual requests.
• The batch throughput levers are InstanceCount (parallelism) · SplitType (splitting) · BatchStrategy/MaxPayload/MaxConcurrentTransforms (mini-batch).
• The input must be splittable for adding instances to have an effect. A single large file will not get faster no matter how many you add.
• To match predictions with the original IDs, use input_filter/output_filter/join_source.
• For serverless you specify only MemorySize + MaxConcurrency. Cold starts are mitigated with Provisioned Concurrency (which incurs standing cost).

Batch Transform — Tuning for Bulk Processing

We dig into the two options on the opposite side from synchronous endpoints: batch transform and serverless inference. First, batch transform reads bulk input from S3, runs inference in bulk, and writes the results to S3. There are three levers for raising throughput.

• InstanceCount: Adding instances distributes the data for parallel processing. The most direct horizontal scaling.
• Data splitting (SplitType): Split a large file into records with Line (per record line), RecordIO, TFRecord, and so on, and distribute them across instances/batches.
• Mini-batch (BatchStrategy, MaxPayloadInMB, MaxConcurrentTransforms): Adjust the size of the bundle of records sent to the model at once and the number of concurrent requests.

Parallel distribution structure of batch transform
 
 S3 input (several files or a splittable format)
   part-000.csv  part-001.csv  part-002.csv  part-003.csv
        │             │             │             │
        ▼             ▼             ▼             ▼
   ┌─────────┐  ┌─────────┐  ┌─────────┐  ┌─────────┐
   │Instance 1│  │Instance 2│  │Instance 3│  │Instance 4│   ← instance_count=4
   └─────────┘  └─────────┘  └─────────┘  └─────────┘
        │             │             │             │
        └─────────────┴──────┬──────┴─────────────┘
                             ▼
                       S3 output (.out)
                             │
                             ▼
                    Job ends → the whole cluster is deleted (cost 0)
 
 ⚠ If the input is a single giant file that can't be split, only Instance 1 works and the rest sit idle.

from sagemaker.transformer import Transformer
 
transformer = Transformer(
    model_name="churn-model",
    instance_count=8,                 # Process in parallel on 8 instances
    instance_type="ml.m5.xlarge",
    strategy="MultiRecord",           # Put several records into one mini-batch
    max_payload=6,                    # Mini-batch max of 6MB
    max_concurrent_transforms=4,      # 4 concurrent requests per instance
    output_path="s3://bkt/out/",
)
transformer.transform(
    data="s3://bkt/in/",
    content_type="text/csv",
    split_type="Line",                # Split per line
)

What each parameter controls is summarized below.

• strategy="MultiRecord" bundles several records into one request to raise throughput, while SingleRecord sends only one record at a time (when the model accepts only single-record input).
• Choose split_type from None (whole file) · Line (per line) · RecordIO · TFRecord to match the input format.

Batch transform throughput is ultimately "parallelism × efficiency per instance." Adding instances speeds things up nearly linearly, but cost grows proportionally too, and distribution takes effect only if the data is divided into several files or can be split with SplitType. If the input is one giant file that cannot be split, a single instance takes it all and scaling doesn't work. So the start of tuning is to shard the batch input into several files or keep it in a format that allows per-line splitting.

⚠️ Pitfall: In batch transform, re-joining the input records with the output predictions can be tricky. To output the input ID together with the prediction, you must merge part of the input into the output with the input_filter/output_filter/join_source parameters (JSONPath). "How do I match the prediction result to the original record" is a practical and easily confused pitfall.

Combining Input IDs with Prediction Results

You must not feed the ID column to the model, but the output is useless without the ID. This contradiction is solved with the filter parameters.

transformer.transform(
    data="s3://bkt/in/",
    content_type="text/csv",
    split_type="Line",
    input_filter="$[1:]",        # Drop column 0 (customer ID) before passing to the model
    join_source="Input",         # Attach the input record back onto the output
    output_filter="$[0,-1]",     # Keep only [customer ID, prediction] in the final output
)

Input  : [C1001, 34, Seoul, 12.5]
          └ ID    └───── features ─────┘
 
input_filter="$[1:]"   ─▶ Model input: [34, Seoul, 12.5]
Model output            ─▶ Prediction: 0.87
join_source="Input"    ─▶ Joined: [C1001, 34, Seoul, 12.5, 0.87]
output_filter="$[0,-1]"─▶ Final: [C1001, 0.87]

Batch vs Real-time — When Batch Wins

Suppose you process 1 million records with the same model. Calling a real-time endpoint 1 million times incurs always-on instance cost plus call overhead, and you also have to manage traffic spikes. Batch transform briefly brings up a cluster, distributes the 1 million records across it, and shuts it down when done, so cost is incurred only during processing time.

A real-time endpoint is optimized for "an immediate response per request" and so has low latency for individual requests, but for bulk processing it must keep instances on and receive sequential/concurrent calls, which is inefficient. Batch maximizes bulk throughput through "data locality + bulk distribution." In summary, bulk processing with no need for immediacy = batch is almost always cheaper and faster. But since you cannot receive results in real time, batch is out if even a little immediacy is needed.

Serverless Inference — Concurrency and Cold Starts

Serverless inference has exactly two things to specify.

• MemorySizeInMB (1024~6144): Increasing memory also increases the compute allocated.
• MaxConcurrency: The maximum number of requests processed at the same time. Requests beyond this are throttled.
• You don't choose an instance type → which also means you cannot use a GPU.
• It automatically brings up compute when traffic arrives and scales to 0 when there is none.

from sagemaker.serverless import ServerlessInferenceConfig
 
serverless_config = ServerlessInferenceConfig(
    memory_size_in_mb=2048,   # 1024 ~ 6144
    max_concurrency=20,       # Max requests processed concurrently
)
predictor = model.deploy(serverless_inference_config=serverless_config)

The problem is the cold start. If there have been no calls for a while and the first request then arrives, a delay occurs while the container is brought up and the model is loaded.

Cold start vs warmed state
 
[Cold]  Request ─▶ │Container start│Model download·load│Inference│ ─▶ Response
                   └──────────── several seconds ─────────────┘ ↑ only this part is real inference
 
[Warm]  Request ─▶ │Inference│ ─▶ Response            (milliseconds to a few hundred ms)
 
Provisioned Concurrency = holding the specified number in a "warm" state
                          → removes cold starts, but incurs that much standing cost

If you need consistently low latency, you can set Provisioned Concurrency to pre-warm a set number of instances. However, in that case that much standing cost is incurred, so part of the "completely zero cost" advantage is lost.

The serverless cost model is "compute time of the requests processed + (optionally) the standing portion of Provisioned Concurrency." The rarer the traffic, the larger the savings versus real-time thanks to scaling to 0, and the steadier and heavier the traffic, the more the savings advantage disappears or reverses. Provisioned Concurrency removes cold starts but brings standing cost, so it makes sense only in the compromise situation of "traffic is intermittent but the first-response delay is intolerable."

Serverless vs Real-time — The Break-Even

As a rule of thumb, if the endpoint is idle most of the time serverless is cheaper, and if it is busy almost all the time real-time is cheaper. The break-even lies somewhere between the two.

⚠️ Pitfall: A simple mapping of "cost reduction = serverless" leads to wrong answers. Serverless is favorable when ① traffic is intermittent and ② you can tolerate cold starts. If you use serverless for steadily high traffic, compute-time billing accumulates and it can become more expensive than real-time, and cold starts make you miss the SLA. On the exam, first check the traffic pattern (intermittent vs steady) and the latency SLA (whether cold starts are tolerated).

Revisiting the Boundary with Asynchronous Inference

When wrapping up batch and serverless, it is easy to confuse them with asynchronous. Separating the three looks like this.

Individual requests whose payload/processing time exceed the serverless limits go to asynchronous, and bulk work with no immediacy goes to batch.

Symptom → Cause → Action

The next article reviews all of Week 6 (inference options, real-time endpoints, advanced inference, batch and serverless) comprehensively.

📖 Glossary

• Batch Transform: A job that runs inference on bulk data in S3 all at once without an endpoint and disappears when finished.
• Sharding: Dividing large data into several pieces. It must be divided for several instances to share the processing.
• SplitType: An option that specifies in what units to split the input. Per line (Line) is the most common.
• Mini-batch (BatchStrategy): The bundle of records sent to the model at once. Bundling several (MultiRecord) reduces overhead.
• MaxConcurrentTransforms: The number of requests one instance processes at the same time. It governs instance utilization.
• join_source: An option that attaches the original input record back onto the batch output. Used for ID matching.
• JSONPath filter: A notation like $[1:] for picking only certain columns from a record. Used for input and output filters.
• MaxConcurrency: The maximum number of requests a serverless endpoint processes at the same time. Beyond it, requests are throttled.
• Provisioned Concurrency: A serverless setting that pre-warms a set number. It removes cold starts but incurs standing cost.
• Throttling: The service rejecting or deferring requests that exceed the allowed limit.