Observability Architecture Diagram: Logs, Metrics and Traces with OpenTelemetry

Most architecture diagrams stop at the services. The telemetry path gets drawn later, during an incident, by whoever is trying to work out why half the traces are missing or why the log bill doubled. By then, the answer depends on details nobody wrote down: which component samples, which one adds the pod name, and what happens when a backend is slow.

An observability architecture diagram shows how logs, metrics and traces leave each service, where they are enriched, sampled and filtered, and where they end up. This post builds one around OpenTelemetry on Kubernetes, explains the agent and gateway patterns behind it, and lists what to label so the diagram still helps during the next outage.

Three signals, three questions

The three are only useful together if they share context. A trace ID in the log line and the same service.name and pod name on every signal let you jump from an alert to the slow span to the log line that explains it. The architecture has to keep that context intact.

The building blocks

OTLP is the OpenTelemetry protocol for all three signals. The specification sets the default ports: 4317 for OTLP over gRPC and 4318 for OTLP over HTTP.

SDKs and auto-instrumentation run inside each service and export OTLP. Point them at a local Collector rather than at a vendor, so routing changes do not need a redeploy.

The Collector is a single binary with pipelines per signal: receivers take data in, processors change it, exporters send it on. The deployment docs describe two main ways to run it:

• Agent. A Collector runs next to the application, as a sidecar or as a DaemonSet with one pod per node. On Kubernetes, the agent is also where the Kubernetes components do their work: the Kubernetes Attributes Processor adds pod, namespace and deployment names, the Filelog Receiver tails container stdout and stderr, and the Kubeletstats Receiver pulls node and pod metrics.
• Gateway. One or more Collectors run as a standalone service, typically one endpoint per cluster, data center or region. The gateway is the place for decisions you want to make once: sampling, filtering sensitive attributes, holding backend credentials and sending to more than one backend.

Head or tail sampling

Traces are the expensive signal, so most teams sample them. Head sampling decides when the trace starts, for example keeping 10 percent. It is cheap, but it cannot keep a trace because it later failed. Tail sampling decides after seeing all or most spans of a trace, so it can keep every error and every slow request.

Tail sampling has a structural cost: all spans of a trace must reach the same Collector. Behind a round-robin load balancer they will not. The gateway pattern solves this with two tiers: the first runs the load-balancing exporter with the routing key set to the trace ID, and the second runs the tail sampling processor. That is two boxes on the diagram, and it is worth showing both.

The example: a shop on Kubernetes

The shop runs three Deployments in a shop namespace: web, checkout-api and payments-worker, each with the OpenTelemetry SDK. The observability namespace holds the pipeline: an otel-agent DaemonSet, an otel-lb Deployment that routes spans by trace ID, and an otel-sampler Deployment that applies the tail sampling policy. Metrics, traces and logs each go to their own store, and alert rules run on metrics.

How to read it:

• Left, the services. Each Deployment exports traces, metrics and logs over OTLP/gRPC on port 4317 to the agent on its own node. The dashed edge is container stdout, which the agent reads from the node rather than receiving over the network.
• Middle, the agent. The agent adds Kubernetes metadata to everything, then sends metrics and logs straight to their stores and traces to the gateway.
• Middle, the gateway. otel-lb routes each span by trace ID, so all spans of a trace reach the same otel-sampler pod, which keeps errors and slow traces and samples the rest.
• Right, storage and alerts. Each signal has its own backend. Alert rules read metrics and page whoever is on call.
• The Markdown block states the sampling policy and the one rule every pipeline follows: the memory limiter first.

What to label

• Protocol and port on every hop. OTLP/gRPC 4317 or OTLP/HTTP 4318, and the export format to each backend.
• Which signals travel on each edge. One pipeline rarely carries all three the same way.
• Where sampling happens, and the policy. Write the numbers; "sampled" alone invites the wrong assumptions.
• Where data is enriched or redacted. Kubernetes metadata in the agent, removal of tokens and personal data in the gateway.
• Retention per backend. It explains the cost and what an investigation can still find.
• Failure behavior. Queues, retries and what is dropped when a backend is down.

Common mistakes

• SDKs exporting straight to a vendor. Every routing or credential change becomes a redeploy, and there is no place to filter data before it leaves.
• Tail sampling behind a plain load balancer. Spans of one trace land on different Collectors, and the sampling decisions are made on fragments.
• No memory limiter. The configuration docs include a memory_limiter processor; put it first in each pipeline so a spike makes the Collector refuse data instead of running out of memory.
• High-cardinality metric labels. A user ID or request ID as a metric label multiplies time series. Put it on spans and logs instead.
• An unmonitored pipeline. The Collector exposes its own metrics, by default on port 8888, as described in its internal telemetry docs. Alert on failed exports, or you find out about lost data during the incident that needed it.

For the cluster itself, the Kubernetes architecture diagram covers control plane, nodes and workloads, and the microservices architecture diagram covers the services this pipeline observes.

Build it in Archboard

1. Sign in with Google or GitHub and create a diagram.
2. From the Kubernetes pack, add two Namespace frames, shop and observability. Put a Deployment for each service in shop.
3. In observability, add a DaemonSet for the agent and two Deployment nodes for the load-balancing and sampling tiers.
4. From the DevOps pack, add Metrics, Traces and Logs nodes for the backends and an Alert node for alerting.
5. Connect them and label each edge with the signals and the protocol, such as "OTLP/gRPC 4317". Make the container log edge dashed.
6. Add a Core Markdown block with the sampling policy.

The example uses 13 diagram nodes, within the Free plan. If your services already run on Kubernetes, the Kubernetes YAML to diagram tool turns the manifests into a first draft of the left half.

FAQ

Do I need a gateway Collector, or are agents enough? Agents alone work for small setups. Add a gateway when you need tail sampling, one place for credentials and filtering, or several backends; the agent then forwards to it.

Should logs go through the Collector too? They can. The Filelog Receiver in the agent reads container logs, adds the same Kubernetes metadata as traces and metrics, and lets you filter before storage. Many teams move logs to the Collector last, after traces and metrics.

Is head sampling ever enough? Yes, when traffic is low or every trace looks alike. If you need every error and slow request, use tail sampling, and draw the two gateway tiers it requires.

Where do dashboards and alerts belong on the diagram? Next to the store they query. Alert rules on metrics are part of the architecture: they decide who is woken up.

How do I keep this diagram current? Change it in the same pull request as the Collector configuration, and link it from the runbook. See architecture diagrams that don't rot.

Start free — one seat, no credit card.