Day 5 - Week 9 Wrap-Up: Security, Governance, and Cost Review

📌 Key Takeaways

• Week 9 in one sentence: don't trust defaults, narrow them explicitly: permissions to the minimum, the network to isolation, data to encryption, and cost to removing idle resources.
• Permissions split into two paths. A failure when creating a job is the user policy (sagemaker:*, iam:PassRole), and a failure while the job is running is the execution role (S3, ECR, KMS).
• The network has 3 stages: VPC mode (traffic into my VPC) → VPC endpoints (reach AWS services without the internet) → EnableNetworkIsolation (block all outbound).
• Encryption has three branches: at-rest (KMS/CMK), in-transit (TLS, between nodes), and Secrets, and KMS requires passing through both doors, the IAM policy and the key policy.
• For cost: training = Spot + checkpoints, instances = fit to the workload, inference = zero idle, operations = auto shutdown, tags, and Budgets.

The Four Axes at a Glance

The four branches of operating "safely, controllably, and cheaply" look separate, but in practice they are bound into a single operating policy.

These four axes apply the security and cost optimization pillars of the AWS Well-Architected Framework to ML. The common philosophy is "don't trust defaults, narrow them explicitly." The exam asks you to pick the "most secure" or "most cost-effective" option, and almost always the narrowest, most explicit option is the answer. The four axes are not independent; they interlock. The execution role meets encryption at the KMS key policy and the STS endpoint, and the network meets cost in interface endpoint charges.

Permissions Review: User Policy vs. Execution Role

The key is separating the two paths. The user policy decides API call permission (sagemaker:CreateTrainingJob) and iam:PassRole, and the execution role decides the S3, ECR, CloudWatch, and KMS that SageMaker touches during a job.

[Path along which permissions flow]
  User / CI
      │ ① sagemaker:CreateTrainingJob  (user policy)
      │ ② iam:PassRole → SageMakerExecutionRole
      ▼
  SageMaker service
      │ ③ sts:AssumeRole  (the execution role's trust policy must allow it)
      ▼
  Execution role  ── ④ Access actual resources with the permissions policy
      ├──▶ S3 (Get/PutObject) · ECR (image pull) · CloudWatch Logs
      └──▶ KMS (Decrypt / GenerateDataKey) ← the key policy must also allow this role

If ① and ② are blocked, the job isn't even created (is not authorized to perform: iam:PassRole), and if ③ and ④ are blocked, the job starts and then fails during execution (S3/KMS Access Denied). The point at which the error appears is the first fork in diagnosis.

Least privilege is narrowed along three axes: actions, resources, and conditions. The managed AmazonSageMakerFullAccess is convenient for training but excessive for production, and guardrails are enforced with an explicit Deny (an explicit deny takes precedence over any allow). If you open PassRole's resource to * it becomes a privilege escalation path, so you specify the role ARN to be passed and pin down even the service it is passed to with iam:PassedToService.

Network Review: Three Stages of Isolation

The three stages are not substitutes but layers used together. The further down you go, the stronger and the more constraining it is.

[Endpoints needed for VPC-mode training]
com.amazonaws.<region>.s3                  (gateway) → training data / artifacts
com.amazonaws.<region>.sagemaker.api       (interface) → job control
com.amazonaws.<region>.sagemaker.runtime   (interface) → inference calls
com.amazonaws.<region>.ecr.dkr / ecr.api   (interface) → container image pull
com.amazonaws.<region>.logs                (interface) → CloudWatch Logs
com.amazonaws.<region>.sts                 (interface) → role assume

A security group is an ENI-level firewall. The container SG has a self-referencing inbound rule (node-to-node communication) and 443 outbound (to the endpoint SG), and the endpoint SG has 443 inbound. Without the self-referencing rule, multi-node training won't start.

⚠️ Trap: "I switched to VPC mode and the job starts but doesn't progress and times out" is almost always not a hyperparameter problem but a missing S3/ECR endpoint. Conversely, if the job isn't created at all, look at permissions (PassRole), not the network.

Encryption Review: At-Rest, In-Transit, and Secrets

Let's firm up what to apply where, by what is being protected.

Permissions and encryption meet at KMS. To read encrypted data you need both kms:Decrypt in the IAM policy AND the role being allowed in the key policy, and for writing kms:GenerateDataKey is additionally required. That is why the answer to "I granted S3 but it still doesn't work" is KMS.

Cost Review: Training, Inference, and Operations

[Inference cost decision tree]
Are there idle periods (zero traffic)?
 ├─ Yes ─▶ Is it OK to defer to batch processing?
 │         ├─ Yes ─▶ Batch transform (instances come down when the job ends → cheapest)
 │         └─ No ─▶ Is the payload large?
 │                       ├─ Yes ─▶ Asynchronous inference
 │                       └─ No ─▶ Serverless inference (accept the cold start)
 └─ No (constant) ─▶ Real-time + autoscaling + MME (many models) + Inferentia (deep learning)

⚠️ Trap: Don't pick Spot just because you see "reduce cost." Spot is only for resumable training, and the candidate answers for reducing real-time endpoint cost are switching to serverless, autoscaling, MME, Inferentia, and Savings Plans.

Requirement Keyword → Feature Mapping

The keywords in exam questions almost directly point to the feature. Memorizing them whole is the fastest way.

Symptom → Cause → Action

Week 9's debugging boils down to four branches: "user or execution role," "missing endpoint," "kms:Decrypt," and "resource left on."

Integrated Scenario: Fraud Detection on Sensitive Financial Data

Decomposing the requirement "train and serve a deep learning fraud-detection model on sensitive financial data while controlling cost" along the 4 axes gives this.

[Requirement] Sensitive data · no internet path · customer key control · cost savings
 
Permissions  A least-privilege execution role allowing only reading the data bucket / writing the artifact bucket
    │    + PassRole limited to that role + a Deny guardrail for unencrypted uploads
Network  VPC mode (private subnets) + 6 endpoints, no IGW
    │    + SG self-reference if distributed / network isolation for extreme sensitivity
Encryption    One CMK for volume and artifact at-rest + node-to-node in-transit
    │    + allow the execution role in the key policy / DB credentials in Secrets Manager
Cost     Spot + checkpoints / inference per traffic pattern / tags and idle auto shutdown

When you translate this into a single training job request, all four axes show up as parameters.

import boto3
sm = boto3.client("sagemaker")
 
sm.create_training_job(
    TrainingJobName="fraud-dl-2026-01",
    RoleArn=EXEC_ROLE_ARN,                       # Permissions: least-privilege execution role
    OutputDataConfig={"S3OutputPath": "s3://ml-artifacts/fraud/",
                      "KmsKeyId": CMK_ARN},      # Encryption: artifacts at-rest
    ResourceConfig={"InstanceType": "ml.g5.xlarge", "InstanceCount": 2,
                    "VolumeKmsKeyId": CMK_ARN},  # Encryption: volume at-rest
    VpcConfig={"Subnets": ["subnet-0a1b2c3d", "subnet-4e5f6a7b"],
               "SecurityGroupIds": ["sg-0123456789abcdef0"]},   # Network
    EnableNetworkIsolation=True,                 # Network: block all outbound
    EnableInterContainerTrafficEncryption=True,  # Encryption: node-to-node in-transit
    EnableManagedSpotTraining=True,              # Cost: Spot
    CheckpointConfig={"S3Uri": "s3://ml-checkpoints/fraud/"},
    StoppingCondition={"MaxRuntimeInSeconds": 86400,
                       "MaxWaitTimeInSeconds": 90000},          # Cost: block runaway jobs
    Tags=[{"Key": "Team", "Value": "risk"},      # Governance: cost attribution
          {"Key": "Project", "Value": "fraud-ml"}],
)

You can confirm the savings from the difference between TrainingTimeInSeconds (actual training time) and BillableTimeInSeconds (billed time) in the describe_training_job response.

This snippet is the compressed version of Week 9: RoleArn (permissions) → VpcConfig and EnableNetworkIsolation (network) → KmsKeyId and VolumeKmsKeyId (encryption) → EnableManagedSpotTraining and Tags (cost and governance). When reading a question, if you ask yourself "which parameter does this requirement land on?", telling the options apart is over instantly.

With this we've gone one full lap through the operations area of ML solutions. Next is the stage of consolidating the knowledge built so far with a full-length mock exam.

📖 Glossary

• Execution role: The IAM role SageMaker assumes on your behalf. All access to S3, ECR, and KMS during a job is done with this role's permissions.
• iam:PassRole: The permission for the act of a user "handing over" a role to a service. The key point is that it is not the role's own permission but the permission to hand it over.
• Trust policy: The policy that defines "who can assume this role." For a SageMaker execution role, sagemaker.amazonaws.com must be in it.
• VPC mode: A setting that attaches the container to customer VPC subnets as an ENI so that familiar VPC controls such as security groups and routing apply.
• EnableNetworkIsolation: The strongest isolation, which cuts all of the container's outbound calls. In exchange, all dependencies must be put in the image beforehand.
• VPC endpoint: A pathway that reaches AWS services without going through the internet. S3 and DynamoDB use the free gateway type, and everything else uses the charged interface (PrivateLink) type.
• Customer managed key (CMK): A KMS key you create yourself, control access to with the key policy, and audit with CloudTrail. The de facto standard in regulated environments.
• crypto-shredding: A way of making data effectively unreadable by cutting off key access rather than deleting the data. One reason to use a CMK.
• Managed Spot Training: Training by renting spare capacity cheaply (up to 90% savings). Since it can be reclaimed, checkpoints are its partner.
• Cost allocation tags: Tags like Team and Project that you attach to resources. Only with these can you split costs by team and project in Cost Explorer.