eBPF Observability: Useful Coverage and Important Blind Spots

Begin with the missing signal

eBPF lets Linux-based tooling attach programs to supported execution points and collect information about system behavior. Observability tools use it for capabilities such as network visibility, profiling and selected application-level measurements.

The verifier and execution restrictions are important safeguards. They do not establish that an agent has zero risk, zero overhead or visibility into every operation. Kernel support, privileges, attachment points and the tool's implementation all matter.

Flows and traces answer different questions

Cilium Hubble provides network and service visibility built on Cilium. Flow information can help explain connectivity, policy verdicts and traffic relationships. It should not be described as a universal replacement for a trace containing every application span. Hubble documentation.

A flow can show that a connection was dropped. An application span can explain that an order was waiting for inventory approval. Those observations complement one another.

For encrypted traffic, useful visibility depends on where and how the tool observes the application or network. Do not assume raw packet inspection reveals encrypted HTTP bodies.

Check the supported runtime

Grafana Beyla documents language and protocol coverage, kernel requirements and capabilities. It positions automatic instrumentation as useful baseline coverage while retaining SDK instrumentation for deeper application detail. Its compatibility matrix includes important differences between runtimes and execution models. Beyla compatibility and requirements.

Context propagation also has limits. For example, Beyla documents constraints when TCP propagation encounters proxies that terminate and recreate connections, and differences for multiplexed protocols. Instrumentation configuration.

Treat those as part of the evaluation, not installation footnotes.

A coverage experiment

Take an illustrative request that enters a Node.js API, queries a database and calls a second service through a proxy. Give the request a known identifier and record its application-level operations in a controlled environment.

Check which observations the candidate agent produces:

Mark absent signals as absent. A visually connected service graph is not proof of complete causal tracing.

Measure collection costs

Record CPU, memory, network export volume and request latency with and without the agent under representative load. Include label cardinality, sampling and destination failure. There is no universal overhead percentage that applies to all tools and workloads.

Review what data leaves the host. Paths, headers and query content can contain sensitive information. Configure collection, redaction, access and retention deliberately.

Deploy as infrastructure

An observability agent may need elevated capabilities and access to host resources. Follow the exact tool's documented requirements rather than granting unrestricted privileges by default.

Changing a Kubernetes networking implementation is a separate infrastructure migration, not a generic one-line observability setup. Evaluate Cilium in the context of the existing cluster networking and supported upgrade path.

Use automatic signals to establish broad coverage, then add targeted application instrumentation where domain context and reliable trace relationships are needed.

Sardar Mukhshaf

Full-Stack & DevOps Engineer

I build scalable web applications, secure backend systems, and production-ready cloud infrastructure. My engineering focus spans interface design, API architecture, deployment pipelines, and observability. Follow for more deep dives into system architecture, performance engineering, and emerging technologies.

ShareTwitterLinkedInCopy Link