Day 2 - Hybrid CI/CD: Bridging On-Premises and AWS into One Deployment Model

Cloud migration rarely happens "all at once, all together." Decades-old COBOL batches, legacy with licenses tied to hardware, workloads that regulation forbids to leave the datacenter — they stay on-prem. New services go to AWS. Immediately: operational pain. "Two environments = two deployment pipelines, two credential systems, two patch tools, two monitoring stacks. If the team runs both, they double their work and double their mistakes. How do we unify operations?" Today: a realistic org with 5,000 on-prem VMs and 200 EC2s, strict security with inbound internet completely blocked. SSM Hybrid Activation·CodeDeploy On-Prem·IAM Roles Anywhere·PrivateLink solve this. Underneath: network, identity, crypto theory.

In DOP exams, hybrid appears as: "how does a DC server call AWS APIs when internet is blocked?", "how do you give DC servers temp credentials without static keys?", "how do you deploy to EC2 and DC servers as one?", "does Direct Connect encrypt the line?" Each touches SSM·CodeDeploy·Roles Anywhere·PrivateLink·DX.

Pull vs Push — Firewall Determines Deployment Architecture

All hybrid design starts from one constraint: On-prem firewalls block inbound. To have AWS "push" commands into DC servers requires opening inbound ports — security teams usually refuse. Result: hybrid operations almost always converge on Pull model — DC server agents reach out to AWS (outbound connection) to "pull" commands.

This is how SSM Agent and CodeDeploy Agent work. Both open outbound HTTPS (443) from DC to AWS, receiving commands and artifacts over that connection. No inbound ports needed — this same philosophy as bastion-less ops (SSM Session Manager kills bastion + port 22).

💡 Related theory: Pull model's advantage flows from distributed systems' connection directionality principle. Firewalls are asymmetric by nature — outbound allowed, inbound blocked is the default policy. To manage many nodes behind NAT, the pattern is: nodes connect to center (pull) not center to nodes (push). Message queue consumers pull from broker; IoT devices maintain outbound MQTT; GitOps agents poll Git. Core trade-off: Pull is firewall-friendly, scales well (center doesn't need every node address), but command propagation has polling latency. Push is immediate but needs inbound paths and node inventory. Hybrid almost always chooses Pull.

Network Topology — Two Decisions to Reach AWS Without Internet

Reaching AWS from internet-blocked DC requires two separate calls. (1) The line carrying traffic (connectivity) and (2) How to reach AWS APIs (endpoints).

Direct Connect (DX) is canonical for the line — telecom line bonds DC and AWS privately. No internet means superior latency, bandwidth, consistency. High availability usually means DX pairs (10Gbps × 2), with IPSec VPN as backup if DX fails. For multiple VPCs and DC paths, Transit Gateway (TGW) acts as hub, concentrating routing.

Reaching AWS APIs is subtler. Internet's blocked, so s3.amazonaws.com public endpoints are unreachable. VPC Endpoint (PrivateLink) is the answer — S3·ECR·SSM·Secrets Manager·CodeBuild APIs exposed inside VPC on private IPs, DC reaches those private IPs via DX/TGW. Traffic never leaves the internet.

⚠️ Gotcha: Direct Connect does not encrypt by itself. Classic trap. DX is a "private dedicated line" so it feels safe, but data on the line is cleartext. If regulation mandates line encryption, use MACsec (MAC Security, IEEE 802.1AE) on the DX port (layer 2, supported on 10Gbps+ dedicated connections) or IPSec over DX (layer 3, VPN over DX). "DX = auto-encrypted" is wrong. VPN (IPSec) encrypts by default but hits internet, so latency/bandwidth vs DX.

🔍 Deeper: DNS is a hidden gotcha in hybrid. DC servers need private IPs for VPC Endpoint names. secretsmanager.ap-northeast-2.amazonaws.com should resolve to private IP, but DC's on-prem DNS doesn't know that. Route 53 Resolver Inbound/Outbound Endpoints bridge both directions — Inbound lets DC query AWS private zones, Outbound lets AWS query DC internal domains. When making VPC Endpoint, "Private DNS enabled" is key — this makes standard AWS endpoint names resolve to private IPs. Exam gotcha: "PrivateLink Endpoint made but DC still resolves to public IP" — usually Resolver Endpoint unconfigured or Private DNS disabled.

SSM Hybrid Activation — Treat DC Servers Like EC2

5,000 DC servers need to patch like EC2s, exist in Systems Manager as managed inventory. SSM Hybrid Activation enables this.

Flow: (1) create-activation issues activation code (ActivationCode + ActivationId). (2) SSM Agent on DC server, register with that code (-register). (3) Server gets mi-xxxx managed instance ID, becomes EC2-like target for Run Command·Patch Manager·State Manager.

aws ssm create-activation \
  --description "OnPrem-DC-App" \
  --default-instance-name dc-app \
  --iam-role SSMServiceRole \
  --registration-limit 100
 
# Output code to DC server
sudo amazon-ssm-agent -register \
  -code "ACTIVATION_CODE" -id "ACTIVATION_ID" -region ap-northeast-2

⚠️ Gotcha: Activation token vs managed instance ID — easy to confuse. ActivationCode/ActivationId are one-time registration tokens with expiry and count limit. Used only when registering a server first; after registration they're meaningless. mi-xxxx is permanently attached after registration. Exam: "activation code expired, what happens to already-registered server?" Answer: "mi-xxxx servers unaffected — continue managed. Activation token only needed for new registration." Exceed registration-limit and can't register more servers; large deployments scope the limit generous or issue multiple tokens.

CodeDeploy On-Prem — Single AppSpec for Both Sides

Deployment unification's core: "EC2 and DC servers use identical AppSpec." CodeDeploy supports On-Premises Instances.

(1) Register DC server with CodeDeploy (aws deploy register), credentials via IAM User or Roles Anywhere. (2) Tag it; bundle into Deployment Group. (3) AppSpec identical to EC2. (4) Single deployment hits EC2 ASG and DC servers simultaneously.

aws deploy register \
  --instance-name dc-server-01 \
  --iam-user-arn arn:aws:iam::ACCT:user/CodeDeployUser \
  --tags Key=Env,Value=Prod
 
aws deploy add-tags-to-on-premises-instances \
  --instance-names dc-server-01 --tags Key=App,Value=Billing

🔍 Deeper: CodeDeploy On-Prem's biggest architectural limit is no Auto Scaling integration. EC2 deployment, CodeDeploy auto-applies latest to new instances spinning up. On-Prem is static inventory — servers don't auto-grow/shrink, so registration and tagging are manual or separate automation. Blue/Green's "spin new instance set, shift traffic" is also constrained on-prem (can't clone physical servers on-the-fly). On-Prem uses primarily In-Place deployment (overwrite code in-place). Exam: "On-Prem + ASG auto-scale deployment" is a trap answer.

IAM Roles Anywhere — Replace Static Keys with X.509 Certificates

DC servers need AWS API credentials. Classically: long-lived IAM User access keys on the DC server — security anti-pattern. Key leaked, valid until revoked; rotation is manual; tracking 5,000 spread keys is hard. IAM Roles Anywhere fundamentally changes this.

Core idea: DC's on-prem PKI (Public Key Infrastructure) X.509 certificates prove identity to AWS; in return, AWS issues temporary credentials (STS tokens). No static keys exist on the server. Server authenticates with cert, gets short-lived token.

# Register on-prem CA as trust anchor
aws rolesanywhere create-trust-anchor \
  --name CorpCA --source sourceType=CERTIFICATE_BUNDLE,...
# Downstream: credential helper gets temp creds

💡 Related theory: Roles Anywhere combines identity federation + PKI. PKI's essence: trust chain — root CA signs intermediate, intermediate signs server cert, verifier trusts root alone, verifies whole chain (RFC 5280, X.509 standard). Roles Anywhere's "Trust Anchor" registers that root/intermediate as AWS's trust anchor. Deeper: ephemeral credentials — "secrets live short, die safer." Static keys accumulate leak risk over time; short-lived tokens leak and expire. Instance Profile (EC2), IRSA/Pod Identity (EKS), OIDC federation (GitHub Actions) all follow this — "replace static secrets with short-lived tokens." Codecov (2021) showed CI environment compromise's scope when static keys there; industry shifted to OIDC, Roles Anywhere, IRSA.

📚 Case study: Codecov (2021) supply chain: Bash Uploader script was compromised, exfiltrated static cloud access keys from CI environment variables to attacker server. Lesson: "Static long credentials in CI = full infrastructure key bleed on CI compromise." Industry moved to OIDC, Roles Anywhere, IRSA — ephemeral, cert-based credentials.

Secret, Config, Monitoring Unification — One Tool Both Sides

Integrated credentials and deployment; integrate the rest.

• Secrets/Config: Secrets Manager·Parameter Store via PrivateLink DC access. Rotated RDS creds cascade to DC servers via SSM Document. Blocked-internet build? CodeArtifact as on-prem npm/Maven mirror cuts external registry dependency.
• Monitoring: CloudWatch Agent on EC2 and DC identical config (Hybrid Activation Role) collects metrics/logs centrally. X-Ray daemon/ADOT on DC traces end-to-end requests spanning DC→AWS.

🎯 Scenario: "5,000 DC servers inbound-internet-blocked: (1) patch like EC2, (2) query Secrets Manager via private route, (3) call AWS API stateless keys, (4) single release with EC2." → (1) SSM Hybrid Activation + Patch Manager. (2) Interface VPC Endpoint (PrivateLink) + Route 53 Resolver private DNS. (3) Roles Anywhere + on-prem CA. (4) CodeDeploy On-Prem + EC2 ASG same AppSpec (In-Place). Encryption: if required, MACsec or IPSec over DX.

Summary

Today covered five. First, firewall inbound block forces Pull model — SSM/CodeDeploy agents outbound-pull commands. Second, internet-blocked AWS reach needs line (DX+TGW) and endpoints (PrivateLink), Route 53 Resolver bridges DNS. Third, SSM Hybrid Activation makes DC servers EC2-like, distinguish activation token (one-time, expires) from mi-xxxx (permanent). Fourth, CodeDeploy On-Prem unifies with EC2, but ASG integration off-limits, static inventory. Fifth, Roles Anywhere replaces static keys with X.509 cert-based ephemeral creds (Codecov lesson), and DX needs MACsec/IPSec if line encryption required.

Next: containers scale to 100+ microservices, large-scale ECS/EKS ops at depth.

📌 Today's Summary

There are five key takeaways today. First, the firewall's inbound blocking forces the Pull model (the SSM/CodeDeploy agents that pull via outbound connections), which is the distributed-systems principle of connection directionality. Second, reaching AWS without the internet requires two decisions, the link (DX+TGW; DX is unencrypted → MACsec/IPSec) and the endpoint (PrivateLink), and Route 53 Resolver connects DNS in both directions. Third, SSM Hybrid Activation manages DC servers like EC2, distinguishing the activation token (one-time, expiring) from mi-xxxx (permanent). Fourth, CodeDeploy On-Prem deploys to both sides with a single AppSpec but is limited by no ASG integration, a static inventory, and mainly In-Place deployment. Fifth, IAM Roles Anywhere replaces static keys with short-lived credentials based on X.509 certificates (the lesson of the Codecov incident, the PKI chain of trust per RFC 5280).