Day 4 - Operations & Cost: Cost Optimization, Logging/Audit, Disaster Recovery

📌 Key Takeaways

• Operations rests on three pillars: cost, audit, and resilience. An ML workflow keeps spending money on data pipelines, training GPUs, and always-on endpoints.
• The economics of training and inference are opposites. You cut training by tolerating interruption (Spot), and you cut inference by eliminating idle cost.
• Managed Spot Training saves up to 90%, but saving checkpoints is practically mandatory.
• "Why did it fail / how slow is it" is CloudWatch; "who did what, and when" is CloudTrail.
• DR plans for three assets separately: data → model → endpoint. An endpoint must be re-creatable from code (IaC) plus the Model Registry.

Cost Optimization: Training and Inference Have Different Economics

• For training, the problem is the total volume; for inference, the problem is idle time. This difference separates the answer options.
• If usage is stable over the long term, a 1-year or 3-year commitment discount such as SageMaker Savings Plans is also a candidate.

Reducing Training Cost

• Managed Spot Training: Uses spare capacity in exchange for possible interruption. Saves up to 90%.
• Right-sizing: Don't use a GPU if a CPU is enough, and turn on distributed training only when needed.
• AMT early stopping: Drops hopeless trials early in hyperparameter tuning.
• Maximum runtime limit: Cuts off an infinite loop or a diverged training run that would spin all night, using max_run.
• Warm-start tuning: Inherits the results of a previous tuning job to reduce the number of search trials.

Managed Spot Training in Detail

estimator = Estimator(
    ...,
    use_spot_instances=True,
    max_run=3600,        # maximum time allowed for the actual training
    max_wait=7200,       # total allowed time, waiting + training (must be >= max_run)
    checkpoint_s3_uri="s3://ml-train-bucket/checkpoints/job-01/",
)

[Spot training timeline]
  Wait for capacity ──▶ Training starts ──▶ Checkpoint saved ──▶ ✗ Interruption
                                     │
                                     └─▶ Capacity recovers ──▶ Resume from checkpoint ──▶ Done
  ◀────────────────── max_wait (wait + all of training) ──────────────────▶
                      ◀──────── max_run (training time) ────────▶

⚠️ Pitfall: The answer to "How do you cut cost with Spot without losing progress?" is always checkpoints. Network isolation, KMS, and sampling options have nothing to do with recovering from an interruption.

Reducing Inference Cost: Four Deployment Options

• Serverless: No billing while idle, but if there are no invocations for a long time a cold start delay occurs.
• Async: Puts requests in a queue and writes results to S3. When the queue is empty, instances can be scaled down to 0.
• Batch Transform: No endpoint is created at all, so the always-on cost is 0. If you don't need real-time responses, it is the cheapest.
• MME: The model is loaded into memory on the first call, so only that request is slow. It is unsuitable if you are extremely latency-sensitive.

Cost Selection Decision Tree

Need to reduce inference cost
├─ Is a real-time response needed?
│    ├─ No ↓
│    │    ├─ Periodically score the entire dataset → Batch Transform (always-on cost 0)
│    │    └─ Per-request but no immediate answer needed, large payload → Asynchronous Inference
│    └─ Yes ↓
├─ Is the traffic intermittent and hard to predict?
│    ├─ Yes → Serverless Inference (idle cost 0, tolerate cold starts)
│    └─ No ↓
├─ Are there very many models, each invoked rarely?
│    ├─ Yes → Multi-Model Endpoint (share one endpoint)
│    └─ No ↓
└─ Steady but with large time-of-day swings?
     ├─ Yes → Real-time + Auto Scaling
     └─ No → Real-time (right-sizing + consider a commitment discount)

Logging and Audit: CloudWatch vs CloudTrail

Training loss curve, container stdout   → CloudWatch Logs
Endpoint latency, error rate            → CloudWatch Metrics
"User alice called DeleteEndpoint at 3:14 UTC" → CloudTrail

Where Logs and Metrics Go

• CloudTrail lets you view recent events directly in the console, but long-term retention and auditing require a Trail that exports to S3.
• CloudTrail logs accumulated in S3 are protected from tampering by bucket policies and Object Lock, and are queried with Athena and similar tools to serve as audit evidence.
• Old audit logs are moved to low-cost storage classes with S3 lifecycle rules to reduce retention cost.

"Debugging performance" = CloudWatch. "Proving compliance" = CloudTrail. CloudTrail records the who, what, when, and where of every API, and is the foundation for regulatory response.

⚠️ Pitfall: CloudTrail records management (control plane) APIs. Don't try to count high-frequency data plane calls like InvokeEndpoint, which arrive thousands of times per second, as audit logs. Invocation volume and latency are viewed with CloudWatch metrics.

Disaster Recovery: Data → Model → Endpoint

Three assets need a recovery plan.

Data

• Replicate training data with S3 cross-Region replication (CRR). For replication, versioning must be enabled on both the source and destination buckets.
• Versioning and Object Lock prevent accidental deletion and ransomware-style overwrites.

Model

• Make artifacts redundant with S3 versioning plus cross-Region copy.
• Use Model Registry to track which version must be restored (including approval status).

Endpoint

• Re-create it as is in another Region or account with IaC (CloudFormation, SageMaker Pipelines).
• Survive a single-AZ failure with multi-AZ (instance_count ≥ 2).

• RTO is "the time it takes to recover," and RPO is "the time range of data you can afford to lose." You must decide both first to determine the strategy.
• Key point: An endpoint must be re-creatable from code plus the Model Registry. A hand-built endpoint is not a recovery target but a cause of the incident.

Availability and Automation

Question Stem Clue → Answer Mapping

In the next note we wrap up Week 11 by tying monitoring, operations, and governance together.

📖 Glossary

• Managed Spot Training: A training mode that uses spare capacity cheaply. It can be interrupted, so checkpoints are practically mandatory.
• Checkpoint: A snapshot of the state saved to S3 during training. Even after an interruption, training continues from here.
• max_run / max_wait: Respectively "the maximum time allowed for training" and "the total allowed time including waiting for capacity."
• Cold start: The phenomenon where, after a long period without calls, preparation time is added when it is called again and the first response is slow.
• Multi-Model Endpoint: A mode where one endpoint loads multiple models from S3 when needed. The more models, the greater the advantage.
• Batch Transform: A batch inference job that scores an entire dataset at once without an endpoint.
• CloudTrail Trail: A setting that persistently stores API call records in S3. It becomes the basis for long-term auditing.
• RTO / RPO: The time allowed until recovery / the time range of data you can afford to lose. The two criteria for choosing a DR strategy.
• S3 cross-Region replication (CRR): A feature that automatically replicates objects to a bucket in another Region. Both buckets require versioning.
• Model Registry: A catalog that manages model versions and approval status. During a failure, it tells you "what to roll back."