Day 2 - MLOps: SageMaker Pipelines, Model Registry, CI/CD

📌 Key Takeaways

• Even if you detect drift, operations fall apart when retraining is manual. MLOps means defining processing → training → evaluation → registration → deployment as code so that it is repeatable and traceable.
• SageMaker Pipelines links Steps into a DAG. Steps come in kinds such as Processing, Training, Tuning, Condition, RegisterModel, and Transform.
• ConditionStep is the quality gate. "If the evaluation AUC is below the threshold, do not register" prevents a bad model from being promoted automatically.
• Model Registry manages versions, approval status, and lineage. A model is registered as PendingManualApproval, and only once approved does it become Approved, which lets deployment automation continue.
• SageMaker Projects automatically provisions two repositories (ModelBuild and ModelDeploy) plus a CodePipeline CI/CD flow, and deploys safely with Blue/Green, Canary, Linear, and Shadow strategies.

The 3 Problems MLOps Solves

Running a model once in a notebook and deploying it by hand from the console gives you no reproducibility, no traceability, and no collaboration.

💡 Concept: The three problems are not independent. Without reproducibility, governance records are meaningless, and without governance gates, automation just deploys bad models faster. Automation is safe only when the gates are already in place.

SageMaker Pipelines: ML Workflows as a DAG

Pipelines connects Steps into a directed acyclic graph (DAG). A dependency between Steps is created automatically when one Step's output (properties) is referenced as another Step's input.

Main Step Types

Pipeline DAG

[ParameterString input_data]
        │
        ▼
 ┌───────────────┐    ┌──────────────┐    ┌────────────────┐
 │ ProcessingStep│──▶ │ TrainingStep │──▶ │ ProcessingStep │
 │  "Preprocess" │    │   "Train"    │    │   "Evaluate"   │
 └───────────────┘    └──────────────┘    └────────┬───────┘
   train/val/test        model.tar.gz        evaluation.json
                                                   │
                                                   ▼
                                        ┌──────────────────────┐
                                        │  ConditionStep       │
                                        │  AUC >= 0.80 ?       │
                                        └───┬──────────────┬───┘
                                    if_steps│              │else_steps
                                            ▼              ▼
                                   ┌─────────────┐   ┌───────────┐
                                   │RegisterModel│   │ FailStep  │
                                   └──────┬──────┘   └───────────┘
                                          ▼
                        [Model Registry] PendingManualApproval
                                          │ approve (Approved)
                                          ▼
                      [EventBridge] ─▶ [ModelDeploy CodePipeline] ─▶ endpoint

Definition Code

from sagemaker.workflow.pipeline import Pipeline
from sagemaker.workflow.steps import ProcessingStep, TrainingStep
from sagemaker.workflow.condition_step import ConditionStep
from sagemaker.workflow.conditions import ConditionGreaterThanOrEqualTo
from sagemaker.workflow.functions import JsonGet
from sagemaker.workflow.parameters import ParameterString, ParameterFloat
 
input_data   = ParameterString(name="InputData", default_value="s3://bucket/raw/")
auc_threshold = ParameterFloat(name="AucThreshold", default_value=0.80)
 
step_process = ProcessingStep(name="Preprocess", processor=sklearn_proc, ...)
step_train   = TrainingStep(name="Train", estimator=xgb, ...)
step_eval    = ProcessingStep(name="Evaluate", processor=eval_proc,
                              property_files=[eval_report], ...)
 
# Register only when the evaluation metric is at or above the threshold
cond = ConditionGreaterThanOrEqualTo(
    left=JsonGet(step_name=step_eval.name, property_file=eval_report,
                 json_path="metrics.auc.value"),
    right=auc_threshold,
)
step_cond = ConditionStep(name="AUCThreshold", conditions=[cond],
                          if_steps=[step_register], else_steps=[step_fail])
 
pipeline = Pipeline(
    name="churn-pipeline",
    parameters=[input_data, auc_threshold],
    steps=[step_process, step_train, step_eval, step_cond],
)
pipeline.upsert(role_arn=role)
pipeline.start()

• Parameterization: ParameterString, ParameterInteger, and ParameterFloat inject input paths, thresholds, and instance types at execution time. You can re-run without changing code.
• Caching: Attaching CacheConfig(enable_caching=True, expire_after="30d") to a Step skips re-running a Step with identical input, saving time and cost.
• Passing evaluation metrics: Declare the JSON written by the evaluation Step as a PropertyFile, read a specific path with JsonGet, and feed it into the condition.

ConditionStep is the heart of the MLOps quality gate. As in "if the evaluation AUC is below 0.80, do not register," it keeps a bad model from flowing into production automatically. On the exam, a stem such as "automatically block deployment of a model that does not meet the criteria" maps to the combination of ConditionStep + the Model Registry approval gate.

Model Registry: Versions and Approval Governance

A trained model is registered in a Model Package Group as a version (Model Package).

from sagemaker.workflow.step_collections import RegisterModel
 
step_register = RegisterModel(
    name="RegisterModel",
    estimator=xgb,
    model_data=step_train.properties.ModelArtifacts.S3ModelArtifacts,
    content_types=["text/csv"], response_types=["text/csv"],
    inference_instances=["ml.m5.large"],
    transform_instances=["ml.m5.large"],
    model_package_group_name="churn-models",
    approval_status="PendingManualApproval",   # approval gate
    model_metrics=model_metrics,               # attach evaluation metrics to the version
)

churn-models (Model Package Group)
├── v1  Approved                (in production)
├── v2  Rejected                (did not meet evaluation criteria)
└── v3  PendingManualApproval   (awaiting review)

Approval State Transitions

• The status is changed with the UpdateModelPackage API (or the Studio UI). Because the status change itself becomes an event, approval is the deployment trigger.
• A pipeline can register directly with approval_status="Approved", but that removes human review, so it is not recommended in regulated environments.

Lineage and Rollback

[Dataset S3] ─▶ [Processing Job] ─▶ [Training Job] ─▶ [Model Package v3]
   Artifact          Action             Action            Artifact
      └──────── Tracked end to end through Associations (links) ────────┘
 
Rollback: v3 → Rejected , v1 → Approved  ─▶ ModelDeploy redeploys v1

• Lineage Tracking automatically links data, processing, training, and models using Artifacts/Actions/Contexts and Associations. This lets you answer "which data was the model that produced this prediction trained on, and who approved its deployment?"

Model Registry is the center of lineage, tying together "which model artifact came from which training job, metrics, and data." The approval status (Approved/Rejected/Pending) is the switch with which a person or automation enforces governance, and a rollback is done not as a new deployment but as a redeployment of the previous Approved version.

SageMaker Projects and CI/CD

SageMaker Projects is an MLOps template that automatically provisions two repositories and the pipelines.

┌──── ModelBuild repo (pipelines/ , tests/) ────┐
│ git push ─▶ CodePipeline ─▶ CodeBuild         │
│                └─▶ SageMaker Pipeline run      │
│                       └─▶ Model Registry entry │
└───────────────────────────────────────────────┘
                 │ approval (Approved) event
                 ▼
┌──── ModelDeploy repo (CloudFormation templates) ──┐
│ EventBridge ─▶ CodePipeline                       │
│     ├─▶ Deploy Staging endpoint                   │
│     ├─▶ Manual approval stage                     │
│     └─▶ Deploy Production endpoint                │
└───────────────────────────────────────────────────┘

# buildspec.yml excerpt (CodeBuild)
phases:
  build:
    commands:
      - python pipelines/run_pipeline.py --module-name pipelines.churn.pipeline
      - python -m pytest tests/    # code and data validation tests

⚠️ Pitfall: Attaching CI/CD does not create a gate. You need three layers of gates in place for automation to be safe: ConditionStep (model quality), Model Registry approval (a person or policy), and Staging validation.

Comparing Deployment Strategies

You do not send a new model to all traffic at once. SageMaker endpoints support gradual deployment.

• Blue/Green, Canary, Linear, and Rolling are endpoint update methods, while A/B and Shadow are simultaneous operation and comparison methods. Their purposes differ.
• In the deployment configuration, attach CloudWatch alarms for automatic rollback. If the error rate or latency exceeds the threshold, the endpoint reverts to the previous configuration.

Stem Clue → Answer Mapping

Pipeline Review Checklist

• Is the evaluation Step's metric connected to the condition through PropertyFile and JsonGet?
• Is the ConditionStep's else_steps non-empty and does it explicitly signal failure?
• Does the approval status at registration match the organization's policy (manual/automatic)?
• Is a CloudWatch alarm for automatic rollback attached to the deployment strategy?

In the next post, we cover ML security, which underpins all of these pipelines: IAM execution roles, VPC isolation, and KMS encryption.

📖 Glossary

• MLOps: An operating approach that ties data processing, training, evaluation, registration, and deployment together with code and automation so that they are repeatable and traceable.
• DAG (directed acyclic graph): A structure in which Steps are connected by arrows and there are no cycles. It defines what runs before what.
• Step: The smallest execution unit of a pipeline. Its types are fixed, such as processing, training, tuning, condition, and registration.
• ConditionStep: A Step that compares a value such as an evaluation metric against a condition and splits the execution path. It is the core of the quality gate.
• PropertyFile / JsonGet: The mechanism that pulls a specific metric value out of the JSON result left by an evaluation Step and feeds it into a condition.
• Model Package Group: A box that collects model versions for the same purpose. Versions v1, v2, … accumulate inside it.
• PendingManualApproval: The default approval status of a new model version. Until it passes review and becomes Approved, it is not a deployment target.
• Lineage: The chain of linked records running from data → processing → training → model. It is the basis for auditing, reproduction, and rollback.
• SageMaker Projects: An MLOps template that creates the ModelBuild/ModelDeploy repositories and the CI/CD pipelines in one go.
• Production Variant / Shadow deployment: A Production Variant is the unit that splits weights within a single endpoint to serve multiple models at once, used for A/B comparison. Shadow deployment is a validation method that copies real traffic to the new model but does not return its responses to users.