Day 3 - CI/CD for ML: SageMaker Projects, CodePipeline/CodeBuild Integration, Automated Deployment

📌 Key Takeaways

• ML CI/CD automates two different triggers separately: code change → automatic retraining and registration (CI) and model approval → automatic deployment (CD).
• SageMaker Projects provisions the repository, CodePipeline, CodeBuild, Model Registry, and EventBridge all at once through a Service Catalog template.
• The standard structure is two pipelines: a Build pipeline (triggered by a code push) and a Deploy pipeline (triggered by model approval).
• The standard CD chain: Registry approval → EventBridge → CodePipeline → CloudFormation deployment.
• Deploying to staging and then placing a manual approval gate before going to production is the safe pattern that reduces the blast radius.

Applying CI/CD to ML

If every code change means a person reruns the pipeline in a notebook, clicks the approval button, and updates the endpoint by hand, it is still manual operation. Traditional software CI/CD is code push → build → test → deploy, but in ML there is an added axis of data and models.

• CI (continuous integration): When the preprocessing or training script changes, automatically run the pipeline to train and evaluate, then register in the Registry.
• CD (continuous deployment): When a model is approved, automatically deploy it to the staging and production endpoints.
• You could wire this together by hand, but AWS lays down the whole skeleton at once with a template called SageMaker Projects.

💡 Related theory: The core idea of ML CI/CD is "make model deployment version-managed, automated, and rollback-capable like software deployment." Code, data, and models are all tracked, changes lead to automatic triggers, and when a problem occurs you must be able to go back to a previous version. In the exam, a flow of "Git commit → automatic retraining and deployment" points to the combination of SageMaker Projects + Code* services.

SageMaker Projects — The MLOps Template

SageMaker Projects provisions the CI/CD infrastructure that MLOps needs as a preconfigured template (Service Catalog based). When you create a project, the following are created all at once.

• What you built by hand yesterday and the day before comes out bundled whole as a standard template.
• The representative template is "model building, training, and deployment," which creates two repositories, one for build and one for deploy.

The Whole CI/CD Chain at a Glance

 ┌──────────── CI (code axis) ────────────┐
 Developer git push
      │
      ▼
 CodeCommit / Git repository
      │  (push event)
      ▼
 CodePipeline(Build) ──▶ CodeBuild (buildspec.yml)
                              │
                              ▼
                    SageMaker Pipeline run
          preprocessing → training → evaluation → ConditionStep
                              │ passed
                              ▼
                    RegisterModel → Model Registry
                    (PendingManualApproval)
 ─────────────────────────────────────────────────
 ┌──────────── CD (model axis) ────────────┐
      Reviewer changes it to Approved
                              │
                              ▼
              EventBridge "Model Package State Change"
                              │
                              ▼
                    CodePipeline(Deploy)
                              │
                ┌─────────────┴─────────────┐
                ▼                           │
      CloudFormation → staging endpoint     │
                ▼                           │
           automated tests pass             │
                ▼                           │
      ManualApproval action (human check) ──┘
                ▼
      CloudFormation → production endpoint

The key point is that the triggers above and below the dividing line are different. The one above is a code push, and the one below is model approval.

Two Pipelines: Build and Deploy

1) Build (model build) pipeline — code-centric

• A developer pushes the training code to the repository
• CodePipeline detects the change and runs CodeBuild
• CodeBuild runs the SageMaker Pipeline (preprocessing → training → evaluation → registration)
• The resulting model is registered in the Model Registry as PendingManualApproval

2) Deploy (model deploy) pipeline — model-centric

• The model in the Model Registry is changed to Approved
• EventBridge detects the approval event and triggers the Deploy pipeline
• The staging endpoint is deployed with CloudFormation → automated tests
• When the manual approval gate is passed, the production endpoint is deployed

Roles of CodeCommit / CodeBuild / CodePipeline

Clearly separating what each service is responsible for is an exam point.

CodeBuild's work is defined in buildspec.yml. For example, the build phase that runs a SageMaker Pipeline looks like this.

# buildspec.yml — defines the commands CodeBuild will run
version: 0.2
phases:
  install:
    runtime-versions:
      python: 3.11
    commands:
      - pip install -r requirements.txt
  build:
    commands:
      # On a code change, create/update the SageMaker Pipeline and run it
      - python pipelines/run_pipeline.py --upsert
      - python pipelines/run_pipeline.py --start

Automatic Deployment Trigger: Approval → EventBridge → Deployment

The heart of CD is the model approval event you learned yesterday. When the approval status changes in the Model Registry, EventBridge catches it and starts the Deploy pipeline.

import boto3
events = boto3.client("events")
 
# A rule that triggers the Deploy pipeline when the model approval status changes to Approved
events.put_rule(
    Name="trigger-deploy-on-approval",
    EventPattern="""{
      "source": ["aws.sagemaker"],
      "detail-type": ["SageMaker Model Package State Change"],
      "detail": {
        "ModelApprovalStatus": ["Approved"]
      }
    }""",
)

If you set this rule's target to the Deploy CodePipeline, deployment rolls automatically the moment a person approves the model. After staging validation and before production deployment, adding one more manual approval action is the safe pattern.

# CodePipeline stage example (conceptual): staging → manual approval → production
# 1. DeployStaging   (create the staging endpoint with CloudFormation)
# 2. ApprovalGate    (a person checks the staging results and approves)
# 3. DeployProd      (deploy the production endpoint)

The staged deployment of staging → manual approval → production in CI/CD follows the principle of "minimizing the blast radius." If you validate a new model in staging rather than putting it straight into production, and then pass it through a gate, defects are filtered out before they reach users. In the exam, a sentence like "to deploy automatically but safely while keeping control just before production" points to CodePipeline's manual approval action.

⚠️ Trap: "Automatic deployment on model approval" and "human confirmation just before production" are not contradictory. Approval happens twice — the first is the Registry's Approved (the decision to put the model into the deployment flow), and the second is CodePipeline's ManualApproval action (the decision to send it to production after seeing the staging validation results). Answer choices that lump these two into one are often wrong.

Direct Setup vs. SageMaker Projects

You can produce the same result by wiring CodePipeline, CodeBuild, and EventBridge together yourself without SageMaker Projects.

Either way, what triggers retraining has to be designed separately. A code push alone cannot catch situations where the data changes.

When CI/CD Goes Wrong: Symptom → Cause → Action

CI/CD failures are mostly one of three: trigger, permission, or definition. If nothing happens, it is the trigger; if it starts but is denied midway, it is permissions; if it runs to the end but the result is wrong, it is a definition (buildspec, template) problem. Narrowing down in this order means you do not have to dig through all the logs.

Tomorrow we compare IaC, which manages the very infrastructure that supports this deployment as code, with orchestrators that reach beyond SageMaker.

📖 Glossary

• CI (continuous integration): The stage that automatically runs through training and evaluation when code changes and registers the resulting model.
• CD (continuous deployment): The stage that pushes an approved model all the way to the endpoint without human hands.
• SageMaker Projects: A feature that lays down the MLOps repository, pipelines, Registry, and triggers all at once with a single template.
• Service Catalog: An AWS service that collects organization-approved infrastructure templates and makes people create things only the standard way. It is the foundation of Projects.
• Build pipeline: The pipeline that starts with a code push and goes as far as training and registering the model.
• Deploy pipeline: The pipeline that starts with model approval and deploys the endpoint.
• buildspec.yml: A file that states which commands CodeBuild runs and in what order.
• ManualApproval action: A stopping point in the middle of CodePipeline that moves on only when a person presses approve.
• EventBridge rule: A routing rule saying "when this event arrives, send it there." The trigger of deployment automation.
• Blast radius: The extent of damage when a problem erupts. The reason for going through staging is to reduce this.