Day 3 - ADOT: The Deep Story of OpenTelemetry Ending the Tracing Tool Wars

In the mid-2010s, distributed tracing was a good idea but an operations nightmare. Zipkin, Jaeger, X-Ray, Datadog, New Relic, Dynatrace — each had its own SDK, its own data format, its own propagation headers. If a company started with X-Ray and wanted to switch to Datadog, every service's instrumentation code had to be rewritten from scratch. Tracing libraries were buried deep in business code, so the cost of changing tools exceeded the value of using tools. This vendor lock-in was the biggest obstacle to distributed tracing adoption. OpenTelemetry solved this through standardization, and ADOT is AWS's official distribution of that standard.

Today we don't see ADOT simply as "AWS's OTel package," but understand what political and technical alliances birthed standardization, why the Collector's receiver-processor-exporter pipeline became the universal architecture for trace data processing, and how ADOT is fundamentally different from X-Ray SDK. In the DOP exam, ADOT increasingly appears in scenarios like "standardize multi-cloud/multi-backend observability," "collect EKS metrics in Prometheus-compatible format," "auto-instrument without code changes."

OpenTelemetry's Birth — A Merger of Two Competing Standards

OpenTelemetry's history is remarkably a merger of two competing projects. Around 2016, OpenTracing (CNCF, focused on API standardization) and OpenCensus (led by Google, including library + agent) both existed to standardize trace instrumentation. They split the market by solving the same problem differently. In 2019, CNCF led their merger into OpenTelemetry — combining OpenTracing's API abstraction and OpenCensus's implementation and auto-instrumentation. This created today's OTel: "API standardization (OpenTracing heritage), auto-instrumentation and Collector provided as implementation (OpenCensus heritage)."

OTel's core promise is decoupling instrumentation from backend. Code instruments only with OTel API, and "send this data to X-Ray or Datadog" is decided later via configuration (exporter). Switch tools and business code stays the same. This is the key idea that unlocked vendor lock-in.

💡 Related theory: OTel's "separation of API and implementation" is the infrastructure version of the classic software principle Dependency Inversion Principle (DIP). Applications depend not on concrete vendor SDKs (low-level modules) but on OTel API (abstraction), and which backend to send to is injected as runtime configuration. The same pattern appears everywhere — SLF4J standardizes logging API and Logback/Log4j plug in as implementations, JDBC standardizes DB access and drivers plug in. OTel applies this adapter/facade pattern to the telemetry layer, realizing "instrument once, choose backend freely." This is how standards win — when enough vendors agree to abandon their SDKs and just provide exporters for the common standard, that standard becomes the de facto industry standard.

Three Signals Unified — logs, metrics, traces

OTel's other ambition is unifying observability's three pillars — logs, metrics, traces — into one standard and one pipeline. Traditionally these were separate tools and SDKs (Fluentd for logs, Prometheus for metrics, Jaeger for traces). OTel binds them with common data model and common transmission protocol (OTLP, OpenTelemetry Protocol).

The core value is correlation. When three signals share the same trace ID and resource attributes, "this error log → this trace → this metric spike at that moment" connects automatically. If signals scatter across isolated tools, humans must manually match them.

# ADOT Collector — receiving three signals, sending to multiple backends
receivers:
  otlp:
    protocols: { grpc: {}, http: {} }
  prometheus:
    config:
      scrape_configs:
        - job_name: app
          static_configs: [{ targets: ['localhost:8080'] }]
processors:
  batch: {}
  resource:
    attributes:
      - { key: service.environment, value: prod, action: insert }
exporters:
  awsxray: { region: ap-northeast-2 }
  awsemf:  { namespace: MyApp/OTel, region: ap-northeast-2 }
  prometheusremotewrite:
    endpoint: https://aps-workspaces.../api/v1/remote_write
    auth: { authenticator: sigv4auth }
  otlp/jaeger:
    endpoint: jaeger:4317
    tls: { insecure: true }
service:
  pipelines:
    traces:  { receivers: [otlp], processors: [batch, resource], exporters: [awsxray, otlp/jaeger] }
    metrics: { receivers: [otlp, prometheus], processors: [batch], exporters: [awsemf, prometheusremotewrite] }

Collector Pipeline — receiver, processor, exporter

ADOT's heart is the Collector. Its structure is a simple yet powerful three-stage pipeline.

• Receivers: Ingestion point for data. Accepts diverse formats — OTLP (gRPC/HTTP), Prometheus scrape, StatsD, Zipkin, Jaeger, etc.
• Processors: Transform received data. batch (batch for transmission efficiency), resource (add common attributes), filter (drop unnecessary data), tail_sampling (Day 2's result-based sampling), attributes (PII masking), etc.
• Exporters: Send transformed data to backend. X-Ray, EMF/CloudWatch, AMP, Jaeger, OpenSearch, OTLP, etc.

These three are assembled in service.pipelines by signal (traces/metrics/logs). The same data can fan-out to multiple exporters simultaneously — the example above sends traces to both X-Ray and Jaeger (essential when running both tools during migration).

🔍 Going deeper: The Collector's receiver-processor-exporter pipeline is not a new invention but the telemetry version of ETL (Extract-Transform-Load) or pipes-and-filters architecture. Unix pipelines (cat | grep | sort), Logstash's input-filter-output, Fluentd's source-filter-match, Kafka Connect's source-transform-sink all use the same structure. The key advantage is composability — each stage is independent, so changing a receiver or inserting a processor doesn't affect the rest. You can add an attributes processor to the processor chain to mask PII, or filter to drop health-check traces, or tail_sampling to keep only error traces, all independent of the backend. Think of Collector as "Logstash for telemetry" and its role becomes clear.

⚠️ Pitfall: Don't conflate the two Collector deployment patterns: agent mode (sidecar/DaemonSet beside applications) and gateway mode (centralized Collector cluster). Agent mode runs beside each node/Pod, collecting and transforming locally so network latency is low, but consumes resources per instance. Gateway mode aggregates all telemetry to a central Collector for batch processing (tail-based sampling needs all spans of one trace in one place, practically requiring gateway), but becomes a single point of failure. The real-world recommendation is 2-tier: agent (collection + 1st pass) + gateway (tail sampling + routing). Placing tail-based sampling in agent mode is a common mistake — a trace's spans scatter across multiple agents, making result-based decisions impossible.

ADOT Deployment Forms — Lambda to EKS

ADOT comes in different forms per runtime.

• Lambda Layer: Simplest. Attach Layer and set one environment variable — code change-free auto-instrumentation.
• ECS Sidecar / EKS DaemonSet: Collector as a cohabiting container.
• EC2 systemd service: Traditional servers.
• EKS Add-on: Operator + CRD for declarative management.

# SAM — Lambda ADOT Layer
Globals:
  Function:
    Tracing: Active
    Layers:
      - !Sub 'arn:aws:lambda:${AWS::Region}:901920570463:layer:aws-otel-python-amd64-ver-1-25-0:1'
    Environment:
      Variables:
        AWS_LAMBDA_EXEC_WRAPPER: /opt/otel-instrument

The magical one-liner is AWS_LAMBDA_EXEC_WRAPPER: /opt/otel-instrument. Lambda runs this wrapper script before starting the runtime; the script injects OTel auto-instrumentation into the runtime. Handler code unchanged, OTel instrumentation applied.

💡 Related theory: AWS_LAMBDA_EXEC_WRAPPER uses Lambda runtime's wrapper script mechanism, intercepting the actual runtime bootstrap to inject code upfront — an interception pattern. The same idea appears everywhere — JVM's -javaagent (intercept class loading), LD_PRELOAD (intercept shared libraries), servlet filter chains (intercept requests). The common principle is "inject cross-cutting concerns (here, instrumentation) at boundaries without modifying the core" — infrastructure implementation of AOP (Aspect-Oriented Programming). This separation means instrumentation stays physically separate from business logic, so toggling instrumentation becomes a deployment configuration problem.

ADOT vs X-Ray SDK — What's Fundamentally Different

The fundamental difference is coupling. X-Ray SDK locks code to X-Ray, ADOT locks code to a standard, leaving backends free. There are trade-offs — ADOT requires managing more components (Collector operations, propagator config), and some of X-Ray's deep features (Insights, etc.) might integrate more smoothly with X-Ray SDK. "AWS only, keep simple: X-Ray SDK; multi-backend, multi-cloud, vendor-neutral: ADOT" is the decision rule.

📚 Case study: A company traced with Jaeger on on-premises Kubernetes, migrating to AWS EKS but wanted "keep Jaeger dashboards temporarily while gradually switching to X-Ray." With X-Ray SDK, every service would need rewriting. Instead, they deployed ADOT, adding both awsxray and otlp/jaeger exporters to the trace pipeline, sending identical traces to both backends simultaneously. During migration, they validated X-Ray dashboards side-by-side with Jaeger, then removed the Jaeger exporter for a zero-downtime cutover. Lesson: exporter fan-out dramatically lowers observability tool migration risk — parallel validation before flip-switch reduces danger.

EKS Observability Standard Stack — ADOT + AMP + AMG

ADOT shines on Kubernetes. The standard stack for collecting Prometheus-compatible metrics on EKS is ADOT + AMP + AMG.

• ADOT Collector (DaemonSet): Prometheus-scrape Pod /metrics endpoints, receive traces via OTLP.
• AMP (Amazon Managed Prometheus): Collector sends metrics via remote_write to managed Prometheus (Day 4).
• AMG (Amazon Managed Grafana): Query AMP via PromQL for dashboards (Day 4).

# Install ADOT Operator as EKS Add-on
aws eks create-addon --cluster-name prod --addon-name adot

EKS Add-on installs ADOT Operator and OpenTelemetryCollector CRD, managing Collector Kubernetes-natively (declarative YAML). ADOT handles trace ID compatibility — X-Ray uses 128-bit+timestamp, OTel uses random 128-bit; ADOT converts between formats.

🎯 Scenario: "Organization goes multi-cloud (AWS + on-premises), standardizing observability. Must swap tracing tools later freely, collect EKS Prometheus metrics and traces together, minimize application code changes. Design?" — Answer: OpenTelemetry/ADOT standardization. ① Instrument all services with OTel API (or auto-instrumentation) for vendor neutrality — later swap backends via exporter config only. ② Deploy ADOT Collector on EKS as DaemonSet/Add-on with 2-tier (agent collecting + gateway tail-sampling). ③ Exporters simultaneously point to X-Ray (traces), AMP (metrics), optionally on-premises Jaeger, multi-backend. ④ Minimize code changes via auto-instrumentation (Lambda's AWS_LAMBDA_EXEC_WRAPPER, Java's -javaagent). Key: "instrument standard once, choose backend freely via config."

Wrapping Up

Today we covered five things. First, OpenTelemetry was born from OpenTracing and OpenCensus merger (2019), ending the tracing tool wars, with core principle decoupling instrumentation from backend (Dependency Inversion). Second, OTel integrates logs·metrics·traces signals into one OTLP, automating correlation. Third, Collector is receiver-processor-exporter pipeline (telemetry's ETL/pipes-filters), fan-outing same data to multiple backends, transforming in one place (filter·mask·tail sample). Fourth, ADOT deploys as Lambda Layer, EKS Add-on, auto-instrumenting code-change-free via interception (wrapper/javaagent). Fifth, ADOT vs X-Ray SDK difference is coupling — bound to standard for backend freedom vs bound to X-Ray for simplicity; EKS standard stack is ADOT + AMP + AMG.

The next article examines these telemetry's destination backends — OpenSearch (logs/search), AMP (Prometheus metrics), AMG (Grafana visualization). We'll explore why inverted indices and time-series DBs are different engines, what workload fits what.

📌 Today's Summary

There are five key takeaways today. First, OpenTelemetry ended the tracing-tool wars through the 2019 merger of OpenTracing (API) and OpenCensus (implementation), and its core value is eliminating vendor lock-in by separating instrumentation from the backend (dependency inversion). Second, OTel unifies the three signals of logs, metrics, and traces into a single OTLP and automates correlation through shared trace IDs and resources. Third, the Collector is a receiver-processor-exporter pipeline (the telemetry version of ETL/pipe-and-filter) that fans out the same data to multiple backends and performs PII masking, filtering, and tail sampling in one place; tail sampling requires gateway mode. Fourth, ADOT is deployed through Lambda Layers, EKS Add-ons, and similar means, and auto-instruments without code changes through interception such as AWS_LAMBDA_EXEC_WRAPPER (the infrastructure implementation of AOP). Fifth, the difference between ADOT and the X-Ray SDK is coupling (standard vs X-Ray lock-in), and the standard EKS observability stack is ADOT + AMP + AMG.