Day 2 - SNS: Topics, Fanout, and How 1:N Notifications Get Made

The SQS we saw on Day 1 was a queue model — "decouple two components in time, but each message is consumed once and disappears." Yet in real systems it's far more common for a single event — "an order came in" — to need to be known independently by six different places: payment, inventory, shipping, email, search index, and data warehouse. In a queue, once one consumer takes a message, no other consumer can see it, so to solve this scenario with queues you'd have to create six queues and have the producer send six times. Every time a new consumer is added, the producer's code has to change — and this is the very definition of the "tightly coupled module" Edsger Dijkstra pointed out in 1968.

The Publish/Subscribe (Pub/Sub) pattern has been the answer to this problem since the ISIS system days of 1987. The producer publishes to a "topic" just once and doesn't know who subscribes to it. Subscribers are added and removed freely, and the topic is the indirection layer between the two. AWS's SNS (Simple Notification Service), launched in April 2010, is the managed implementation of this model, and paired with SQS (launched in 2006) it serves as the backbone of nearly every event-driven architecture on AWS. This article looks at why SNS was designed the way it was, which trade-offs it shares with SQS, and what the pitfalls are on the exam and in operations.

Topics and Subscriptions: The Freedom That Indirection Creates

SNS's model is simple. A Topic is the indirection point, a Subscription is an endpoint listening to that topic, and Publish is the act of throwing a message at the topic. Publish once, and SNS fans it out across all subscriptions of that topic and pushes it to each one. Push is the key here — this is the exact opposite of SQS's pull (where the consumer calls ReceiveMessage), and it's the branch point that separates the two systems' design choices.

The subscribable protocols have kept growing over time. At launch in 2010 it was just HTTP/HTTPS, Email, SQS, and SMS, but now it includes Lambda (2015), Mobile Push (APNS/FCM/Baidu, 2010-2013), Kinesis Data Firehose (2021), and EventBridge Pipes (2022). This is SNS's real value — publish once, and it goes to email, to mobile push, into a queue, and gets loaded into a data warehouse. The producer needs to know nothing about this diversity.

Standard SNS is at-least-once + best-effort ordering, similar to SQS Standard, and FIFO was launched belatedly in 2020 to pair with SQS FIFO. The fact that SNS FIFO can only take SQS FIFO as a subscription is an exam trap — email and HTTP are channels where ordering/duplication guarantees are inherently impossible (email can be reordered by SMTP relays, HTTP retries on 4xx/5xx), so FIFO semantics break down.

💡 Related theory: The difference between Pub/Sub and Message Queue was laid out clearly in Gregor Hohpe and Bobby Woolf's Enterprise Integration Patterns (2003). A Point-to-Point Channel (queue) is one message = one consumer; a Publish-Subscribe Channel (topic) is one message = N consumers. The SNS+SQS fanout pattern is the exact managed implementation of "Publish-Subscribe Channel + Message Endpoint" from that book. This pattern exists in the same form in Kafka's consumer groups, Google Pub/Sub's subscriptions, and Azure Service Bus's topic+subscription.

🔍 Going deeper: How does SNS implement fanout internally? It's not officially disclosed, but from its behavior it's a subscription registry + per-protocol delivery worker structure. Publish once → SNS looks up the subscription list → per-protocol workers (SQS sender, Lambda invoker, HTTP poster, etc.) push in parallel. Each worker has its own retry policy, so a slow subscription doesn't affect others. An HTTP/HTTPS subscription gives up immediately on 4xx, and on 5xx retries with exponential backoff up to 100,015 times (over about 23 days) — this is the RFC 7807 recommended behavior and other clouds are similar.

The Fanout Pattern: Why SNS + SQS Became the De Facto Standard for AWS Decoupling

In theory, SNS can push directly to Lambda or HTTP. Yet most real production systems are designed with one more stage added: SNS → multiple SQS → each consumer. Why?

The answer is buffer. Because SNS is a push model, if a subscriber can't keep up, you have to rely on SNS's retry, and if too many messages pile up that a Lambda subscription can't process, it gets throttled and the messages back up in SNS's retry queue. SQS, on the other hand, has the consumer pull at its own pace, so the backlog safely accumulates in the queue, and even if the consumer dies and comes back, it processes the messages in the queue as they were. In other words, SQS acts as a shock absorber between SNS's push and the consumer's processing speed.

[ Order event fanout ]

Order Service
   │ Publish (orderId, customerId, amount, items)
   ▼
SNS Topic: order-events
   │ fanout (parallel)
   ├─→ SQS-Payment       → Lambda payment-handler
   ├─→ SQS-Inventory     → ECS inventory-worker
   ├─→ SQS-Shipping      → Lambda shipping-handler
   ├─→ SQS-Notification  → Lambda email-sender
   ├─→ SQS-Analytics     → Firehose → S3 → Athena
   └─→ SQS-Search-Index  → Lambda → OpenSearch

Each SQS has its own DLQ, its own visibility timeout, its own consumer count
Even if one consumer goes down, the others are unaffected

The second benefit of this pattern is DLQ separation. SNS can have a DLQ too (launched 2019), but it only catches the case where "SNS failed to push to SQS." What actually matters in production is the case where "the message was received but processing failed," and that has to be caught by each SQS's DLQ. If you rely on the SNS DLQ, you can't tell which consumer failed, making post-mortem analysis hard.

The third benefit is zero cost to add a new subscriber. If a new system needs to listen to order events, you just create a new SQS queue and add a subscription to SNS — done. No code change to the existing producer or the other consumers. This is the most frequently seen evolution pattern in microservice architectures.

⚠️ Pitfall: It's easy to think "isn't SNS → Lambda direct subscription simpler?" but it's weak against traffic spikes due to Lambda concurrency limits, cold starts, and the absence of a DLQ (with direct subscription). The SNS → SQS → Lambda (Event Source Mapping) pattern uses SQS as a natural buffer that protects Lambda. In practice, aside from simple ad-hoc notifications (e.g., CloudWatch Alarm → SNS → Email), it's almost always the standard to insert SQS as an intermediate stage.

Message Filtering: How to Mix Several Event Types in a Single Topic

By the book, you'd separate topics per event type. Order events, payment events, refund events, shipping-status-change events each in their own topic. But at a real company, once you exceed 30 topics, management explodes. IAM policies, alarms, and the subscription graph all grow, and it takes time to figure out "which topic does this event go to?"

Subscription Filter Policy (launched 2018) solves this. Publish several kinds of events to a single topic, but each subscription declares "the conditions for the messages I want to receive" as a JSON policy. SNS filters at publish time and pushes only to matching subscriptions. Subscribers receive only the events they care about, reducing processing cost, and the topic count shrinks, simplifying management.

// Message attributes the publisher sends
"MessageAttributes": {
  "event_type": {"DataType": "String", "StringValue": "order_placed"},
  "customer_tier": {"DataType": "String", "StringValue": "premium"},
  "amount": {"DataType": "Number", "StringValue": "150000"}
}
 
// Subscription A: receives only orders from premium customers
{
  "event_type": ["order_placed"],
  "customer_tier": ["premium"]
}
 
// Subscription B: receives only orders of 100,000 KRW or more (numeric comparison)
{
  "amount": [{"numeric": [">=", 100000]}]
}
 
// Subscription C: all events except refunds
{
  "event_type": [{"anything-but": "refund"}]
}

Payload-Based Filtering, launched in 2023, goes one step further and filters on fields in the body (JSON body) rather than message attributes. Previously the publisher had to duplicate body data into message attributes, but now you can leave the body as is and filter on it. The only condition is that the body must be JSON.

🔍 Going deeper: SNS filter policy matching happens on the SNS side, so "a message that doesn't match the filter" costs the subscriber 0. Neither the cost of a message going to SQS nor the cost of a Lambda invocation is incurred. If you instead filter on the subscriber side (an if-else inside Lambda), every message invokes Lambda first, incurring all the cold-start and request cost. On high-traffic topics, server-side filtering often makes a cost difference of more than 10x.

📚 Case study: In 2022, the Netflix Tech Blog published the story of redesigning its internal event bus around SNS filtering. Previously each microservice operated its own Kafka topic, reaching thousands of topics, but after moving to the "one SNS topic per domain + branching per subscriber via filter policies" pattern, the topic count dropped to 1/10 and the time to add a new consumer shrank from days to minutes. That said, due to throughput limits Netflix didn't fully replace Kafka, moving only the "low throughput + diverse subscribers" workloads to SNS.

SNS FIFO and the Essence of Ordering Guarantees

On Day 1 we said SQS FIFO's 300 msg/s limit is due to "single-partition serialization per MessageGroupId," and SNS FIFO is exactly the same mechanism. The topic itself is a FIFO topic, and when you specify a MessageGroupId at publish time, the publish order within that group is propagated as-is to all SQS FIFO subscriptions. Turn on ContentBasedDeduplication and the SHA-256 hash of the body becomes the automatic dedup ID, so duplicate publishes within the 5-minute window are ignored.

SNS FIFO's real value is that "multiple subscribers receive the same messages in the same order." If, without fanout, each consumer sent separately to its own SQS FIFO, each consumer could see a subtly different order — but going through SNS FIFO, all subscribers see the same publish order. In distributed systems, "global ordering as seen by all subscribers" is a famously expensive guarantee, and even Kafka only guarantees it within a partition.

The constraints are considerable, though.

• Subscribable: SQS FIFO + Lambda (added 2023). Email/HTTP/Mobile Push not possible.
• Throughput: 300 TPS per topic (3,000 with batching). High Throughput FIFO is still SQS-only.
• Price: roughly 10x the unit price of Standard SNS.

💡 Related theory: In distributed systems, the famous 1996 theorem by Tushar Chandra and Sam Toueg holds that "total order broadcast" (all nodes see messages in the same order) is equivalent to consensus. It can be implemented with consensus algorithms like Paxos/Raft, but at the cost of latency and throughput. The SNS FIFO + SQS FIFO combination being tied to 300 TPS is because there's no way to bypass this fundamental cost. If you need exactly-once + total order, you should first ask "do I really need it, and is there no way to partition by MessageGroupId?"

Comparison with Other Pub/Sub Systems

The biggest branch point in this table is whether there's retention. SNS has no retention, so "if there's no subscriber at publish time or the subscriber is down, that message is lost" (unless there's an SNS DLQ). Kafka, Google Pub/Sub, and Service Bus have retention, so consumers can catch up on lag days later. If you need this "replayable log" on AWS, you have to go to Kinesis (Day 4) or MSK.

EventBridge (Day 3) looks similar to SNS but is a different tool. EventBridge specializes in gathering SaaS/AWS-service/custom events onto one bus and routing them with rules, while SNS specializes in simple pub/sub. SNS has overwhelmingly higher throughput, and EventBridge has overwhelmingly richer integration.

📚 Case study: During the massive AWS S3 us-east-1 outage on February 28, 2017, many companies had their "S3 event notification → SNS → downstream" chains all severed, so no alarms came through. The post-mortem produced the recommendation "don't depend all your notifications on a single-region SNS; separate the availability of the notification itself with multi-region SNS publish + Route 53 failover." Afterward, monitoring companies like PagerDuty and Datadog built their own multi-region notification fabrics.

DLQ, Encryption, and Operational Details

SNS Subscription DLQ was launched in 2019. If a subscription repeatedly fails to push (e.g., an HTTP endpoint permanently returning 5xx), it sends that message to the designated SQS DLQ. The thing to watch is that an SNS DLQ is per-subscription, not per-topic. So for the same topic, subscription A can have a DLQ while subscription B doesn't, and many companies put an SCP in place to enforce a DLQ on every subscription for operational standardization.

Encryption is SSE-SNS (AWS-managed key, free by default) and SSE-KMS (customer-managed KMS, key usage permissions controlled by IAM). With SSE-KMS, both publish and subscribe need KMS Decrypt permission, and a common incident is a push failure caused by a Lambda subscription lacking KMS permission and failing to decrypt the message. You have to add the subscriber principal not only to the topic policy but also to the KMS key policy.

Cross-account subscriptions require the topic policy (resource-based policy) to explicitly allow the other account's ARN. In large organizations, the hub-and-spoke pattern — placing topics in a central event account and having workload accounts subscribe cross-account — is standard.

Using a VPC Endpoint (Interface, based on AWS PrivateLink) makes publish traffic flow over the AWS backbone without traversing the internet. It's nearly essential in domains with compliance requirements like finance and healthcare, and maps directly to the NIST SP 800-53 SC-7 (boundary protection) control.

Message Archive and Replay

SNS FIFO, via the Message Archive and Replay feature launched in 2023, retains messages published to a topic for up to 365 days and lets a new subscriber replay past messages. Standard topics are not yet supported. Before this feature, you had to receive via SNS → SQS to leverage SQS's 14-day retention, or archive via SNS → Firehose → S3.

Replay is useful in the scenario "when a new system is added, it must process past events too," but this kind of scenario is essentially better suited to event sourcing (EventStoreDB, Kafka). SNS FIFO Archive is best viewed as a short-term (days to a few months) partial replay tool, and if you need true event sourcing you should go to Kinesis or MSK.

# Standard Topic
aws sns create-topic --name order-events
aws sns subscribe --topic-arn arn:...:order-events \
  --protocol sqs --notification-endpoint arn:...:order-payment-queue
 
# Filter Policy (server-side filtering)
aws sns set-subscription-attributes --subscription-arn arn:... \
  --attribute-name FilterPolicy \
  --attribute-value '{"event_type":["order_placed"],"amount":[{"numeric":[">=",10000]}]}'
 
# FIFO Topic
aws sns create-topic --name order-events.fifo \
  --attributes FifoTopic=true,ContentBasedDeduplication=true
 
# Subscription DLQ (isolate failed messages)
aws sns set-subscription-attributes --subscription-arn arn:... \
  --attribute-name RedrivePolicy \
  --attribute-value '{"deadLetterTargetArn":"arn:aws:sqs:ap-northeast-2:123:order-events-dlq"}'
 
# Publish with attributes (used as filtering keys)
aws sns publish --topic-arn arn:...:order-events \
  --message '{"orderId":"o-1001","amount":15000}' \
  --message-attributes '{"event_type":{"DataType":"String","StringValue":"order_placed"},"amount":{"DataType":"Number","StringValue":"15000"}}'
 
# Allow cross-account publish (topic policy)
aws sns set-topic-attributes --topic-arn arn:... \
  --attribute-name Policy \
  --attribute-value file://cross-account-policy.json

Wrapping Up

SNS exists for the "1:N event broadcast" that SQS's "1:1 work distribution" model can't solve. And the SNS + SQS fanout pattern combines the strengths of both services (SNS's diverse push protocols + SQS's safe buffer) to become the most frequently seen decoupling architecture in AWS. Message filtering prevents topic explosion, FIFO is an option for domains that need ordering, and subscription DLQ and cross-account subscriptions are essential operational features in large organizations.

In the next article we'll look at a problem that's similar to SNS but on a different dimension — EventBridge, which gathers AWS-service events, SaaS events, and custom events in one place and routes them by rule. If SNS is "notify N people of the event I publish," EventBridge is a one-level-higher abstraction: "receive events that occur all over the place and send them somewhere according to conditions," and how the two divide labor is the crux of both the exam and practice.