Day 3 - ML Security: IAM Execution Roles, VPC Isolation, KMS Encryption

📌 Key Takeaways

• ML security rests on three pillars: IAM execution roles (who does what), VPC isolation (where traffic goes), and KMS encryption (who can unlock it).
• SageMaker training jobs and endpoints run with an Execution Role, not the user's credentials. When you see AccessDenied, suspect the execution role first.
• A requirement that "it must never reach the internet" is met by the combination of private subnets + VPC endpoints + enable_network_isolation.
• With encryption, the key question is not turning it on but who can use the key. Decryption works only when the KMS key policy and the IAM policy both allow it.
• Separating CMKs per team or project enforces data isolation: even with IAM permissions, the key policy blocks access.

The Three Security Pillars at a Glance

An ML system gathers sensitive training data, expensive model artifacts, and broad permissions in one place. If it leaks, data theft, model theft, and permission abuse all happen at once.

                    ┌──────────────────────────────────┐
                    │  SageMaker training / inference   │
                    │            workloads              │
                    └────────────────┬─────────────────┘
        ┌────────────────────────────┼────────────────────────────┐
        │                            │                            │
  ① IAM execution role         ② VPC isolation              ③ KMS encryption
  "who does what"              "where it communicates"      "who can unlock it"
        │                            │                            │
  - least-privilege policy     - private subnets            - S3 SSE-KMS
  - trust policy               - VPC endpoints              - volume_kms_key
  - scoped to S3/KMS paths     - security groups            - output_kms_key
                               - network isolation          - inter-node traffic encryption

IAM Execution Roles: The Principle of Least Privilege

SageMaker training jobs and endpoints run with the permissions of an Execution Role. If you grant this role too much, the damage from a breach grows accordingly. Give least privilege only for the S3 paths and KMS keys that are needed.

If a user can read the bucket fine in the console but only the training job gets AccessDenied, it is almost always an execution role problem. For a user to pass a role to a service, iam:PassRole is also required separately.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:PutObject"],
      "Resource": [
        "arn:aws:s3:::ml-train-bucket/data/*",
        "arn:aws:s3:::ml-train-bucket/output/*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": ["kms:Decrypt", "kms:GenerateDataKey"],
      "Resource": "arn:aws:kms:us-east-1:111122223333:key/abcd-efgh"
    }
  ]
}

• The key point is that it is narrowed by prefix, not to the whole bucket.
• For an SSE-KMS bucket, S3 permissions alone are not enough. You need kms:Decrypt for reads and kms:GenerateDataKey for writes as well.

The Trust Policy separately restricts "who can assume this role."

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": { "Service": "sagemaker.amazonaws.com" },
    "Action": "sts:AssumeRole"
  }]
}

AccessDenied Diagnosis Flow

Training job gets AccessDenied on S3
        │
① Are you looking at the "execution role" permissions, not the user's?
     └ No → check the execution role ARN first
        │
② Does the execution role policy include that bucket/prefix?
     └ No → fix the s3:GetObject resource scope
        │
③ Is there an explicit Deny in the bucket policy?
     └ Yes → Deny always wins. Fix the bucket policy
        │
④ Is the object SSE-KMS?
     └ Yes → check kms:Decrypt / GenerateDataKey + the key policy
        │
⑤ Is the job running inside a VPC?
     ├ Yes → check the VPC endpoint policy, routing, and security groups
     └ No  → check organization SCPs and the Permissions Boundary

On the exam, "the training job cannot access S3 (AccessDenied)" is usually insufficient execution role permissions, not a user permission problem. Conversely, "the role can access all buckets" violates least privilege. Remember that the principal of the permissions is the execution role.

⚠️ Pitfall: In IAM, an explicit Deny takes precedence over any Allow. If even one of the role policy, bucket policy, or SCP denies, opening up everything else will not let the request through.

VPC Isolation: Training Without the Internet

By default, SageMaker jobs can reach the internet from the AWS-managed network. For sensitive data you must block this. Use VpcConfig to place the job in a private subnet of your own VPC, and use VPC endpoints to reach S3 and other services without the internet.

estimator = Estimator(
    image_uri=image, role=role,
    instance_count=1, instance_type="ml.m5.xlarge",
    subnets=["subnet-priv-a", "subnet-priv-b"],
    security_group_ids=["sg-0123456789"],
    encrypt_inter_container_traffic=True,   # encrypt traffic between distributed training nodes
    enable_network_isolation=True,          # block the container's external network access
)

What Each Isolation Option Blocks

When you run untrusted third-party code or models, turning on network isolation is the default.

Two Kinds of VPC Endpoints

A gateway endpoint is "a routing change that diverts the path to S3 inside AWS," while an interface endpoint is "creating a private-IP entry point for the service inside my subnet."

        [VPC]
 ┌────────────────────────────────────────────────────────────┐
 │  [Private Subnet A]                    [Private Subnet B]  │
 │   training container (ENI) ◀─ distributed training (SG self-reference) ─▶ training container │
 │        │     │                                             │
 │        │     └─(Interface Endpoint)─▶ SageMaker API        │
 │        │                              CloudWatch Logs      │
 │        └─(Gateway Endpoint · routing)─▶ S3 / DynamoDB      │
 │                                                            │
 │   ✗ No internet gateway           ✗ No NAT gateway         │
 └────────────────────────────────────────────────────────────┘

• With distributed training, the security group needs a rule that references itself, because the nodes must communicate with each other.
• Place at least two subnets in different AZs and keep spare IP addresses. Without spare IPs, instance placement fails.

⚠️ Pitfall: With enable_network_isolation=True, the code inside the container cannot call S3 directly. Training still runs because the platform, outside the container, handles the input channel download and artifact upload on its behalf. If the training script does a pip install at runtime or fetches more data from an external URL or S3, it fails at that point.

If you only place the job in a private subnet and do not create an S3 gateway endpoint, training cannot receive its data and stalls. "The job will not start after I isolated it" is usually a missing endpoint.

KMS Encryption: Protection at Rest and in Transit

Encrypt data and models both at rest and in transit. A KMS customer managed key (CMK) controls who can decrypt.

estimator = Estimator(
    ...,
    output_kms_key="arn:aws:kms:us-east-1:111122223333:key/abcd-efgh",   # encrypt model artifacts
    volume_kms_key="arn:aws:kms:us-east-1:111122223333:key/wxyz-1234",   # encrypt training EBS
)

The Double Gate: Key Policy + IAM Policy

• Most resources are accessed with an IAM policy alone, but for KMS the key policy is the primary gate. If the key policy does not delegate to IAM, decryption does not work even with administrator permissions.

Per-Team CMK Isolation Pattern

If you share an AWS managed key, there is effectively no isolation within the account. Remember: isolation requirement = CMK + separate key policies.

"Encryption is on, but who can use that key?" is the real control point. The KMS key policy and IAM work together, and if the key policy denies, decryption fails even with IAM permissions. The pattern of splitting CMKs per team or project to isolate data is within exam scope.

Supporting Services for Data and Model Protection

SageMaker Studio and Notebook Security Options

Security Review Checklist

• Is the execution role restricted by prefix rather than to the whole bucket?
• For an SSE-KMS bucket, are kms:Decrypt and kms:GenerateDataKey both present?
• Is there an S3 gateway endpoint in the private subnet?
• For distributed training, does the security group have a self-referencing rule?
• Are notebooks and Studio not directly exposed to the internet?

Stem Clue → Answer Mapping

In the next post, we cover the other half of operations: cost optimization, CloudWatch/CloudTrail logging and auditing, and disaster recovery.

📖 Glossary

• Execution Role: The IAM role that a training job or endpoint uses "on its own behalf." It is separate from a person's credentials.
• Trust Policy: The role-side policy that defines "who can assume this role." It pairs with the permission policy.
• iam:PassRole: The permission that lets a user hand a role to a service. Without it, job creation itself is blocked.
• Network Isolation: A setting that prevents the container from making any outbound network calls. The platform moves inputs and outputs on its behalf.
• VPC endpoint: A private channel that reaches AWS services without going through the internet. There is a gateway type for S3 and DynamoDB and an ENI-based interface type.
• Customer managed key (CMK): A KMS key that you create yourself and whose key policy you also control. It is the core tool for per-team isolation.
• Key Policy: The resource policy attached to a KMS key. Separate from IAM permissions; if it blocks here, decryption does not happen.
• SSE-KMS: A method where S3 performs server-side encryption of objects with a KMS key. Reading and writing also require KMS permissions.
• Encryption in transit: Encryption while data flows over the network. Endpoints use TLS, and traffic between distributed training nodes is enabled with a separate option.