Skip to content

Commit cb38b30

Browse files
wankai123wankai123
authored
Analyze OTLP spans opt-in by service.layer, and file the OTLP service catalog by layer (#14170)
TAL is opt-in by layer: - The shipped tal/otlp-general.yaml (was otlp-default.yaml) declares its rules for OTLP_GENERAL (renamed from OTLP_SERVICE). Only spans whose resource carries service.layer=OTLP_GENERAL are analyzed; the others are stored and searchable, and the OAP logs one INFO line per service whose spans carry no layer. - New OTLP_MESH layer, with no shipped rules, for mesh sidecars. - No layer: auto. A rule declares a registered layer, or the OAP does not start. The OTLP service catalog keeps the layer: - OTLPServiceTraffic stores service.layer in otlp_service_layer, one row per name and layer, UNDEFINED when absent or unregistered. IOTLPTraceQueryDAO#getServiceNames is replaced by getServices(layer), filtered in BanyanDB, Elasticsearch and JDBC. - The stored span carries the same layer, and a TraceQL q on resource.service.layer alone is answered from the catalog. Network aliases come from strict span pairing: - A caller half, keyed by its span id, meets the callee half of the SERVER span whose parent is that span, on the trace's owner node. prepareNetworkAlias, buildNetworkAlias and resolveNetworkAlias replace the alias statement, resolveAlias and KnownServices. - The alias is keyed by the caller and the address it dialed, frontend->cart:7070 (callerScopedAddress). The topology query folds an address node by the caller-scoped row, then by the plain one. - pairingSamplePercent (default 25) picks the traces paired. An address listed in gateways.yml is never aliased and is drawn as a VIRTUAL_GATEWAY node (isGateway). - A SERVER span that passed its own context upstream, as nginx's OpenTelemetry modules do, may send a caller half, so a proxy that reports one span per request is learned on both hops. Network address alias writes (agent path too): - NetworkAddressAlias#combine skips the write of an unchanged mapping written less than five minutes ago, and last_update_time_bucket is the minute OAP received the setup: a stable alias costs about twelve writes an hour instead of about 140. Docs: tal.md, trace-analyzer.md, otlp-trace.md, traceql-service.md, configuration-vocabulary.md, and the new trace-analyzer-practices.md.
1 parent 4c831c7 commit cb38b30

84 files changed

Lines changed: 3558 additions & 1875 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎docs/en/api/traceql-service.md‎

Lines changed: 10 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -868,8 +868,16 @@ values from the span-name catalog of the service given in `q`, `remote.service`
868868
other `resource.` and `span.` keys from the tag autocomplete index of that scope, both scopes for an unscoped `.key`;
869869
`limit` caps every value list. The three catalogs list every name the
870870
storage retains and ignore `start` and `end`: a catalog row is written once, when its name is first seen, so a time
871-
window would hide every name older than it; the autocomplete index and the sample below honor the window. A `q`
872-
narrower than a service name, for
871+
window would hide every name older than it; the autocomplete index and the sample below honor the window. The service
872+
catalog keeps each name with the `service.layer` its resource carried when that is a registered layer, `UNDEFINED`
873+
otherwise, so a `q` whose only condition is the layer, `{ resource.service.layer = "OTLP_GENERAL" }`, is answered from
874+
it in full with the services stored with that layer, which leaves out the services of another layer, such as service
875+
mesh sidecars under `OTLP_MESH`, and those that set none; `UNDEFINED` lists the services that set no registered
876+
layer. The stored span carries the same value as its `service.layer` resource attribute, `UNDEFINED` in place of
877+
an unregistered or absent one, so a search on the layer finds the traces of the services the catalog lists under it.
878+
The unscoped `{ .service.layer = "OTLP_GENERAL" }` also matches a span attribute of that name, so it is
879+
answered from the sample below. Without a layer condition every service is listed. A
880+
`q` narrower than a service name, for
873881
instance `{ resource.service.name = "frontend" && kind = client }`, switches both endpoints to a sample: the newest 50
874882
matching traces are read and the names or values are taken from the spans in them that match the filter, which is
875883
how Grafana's dependent dropdowns narrow down. Every attribute is indexed, so every key such a sample offers is

‎docs/en/changes/changes.md‎

Lines changed: 5 additions & 2 deletions
Large diffs are not rendered by default.

‎docs/en/concepts-and-designs/tal.md‎

Lines changed: 109 additions & 53 deletions
Large diffs are not rendered by default.

‎docs/en/setup/backend/configuration-vocabulary.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -178,10 +178,10 @@ It divided into several modules, each of which has its own settings. The followi
178178
| - | - | lalFiles | The LAL configuration file names (without file extension) to be activated. Read [LAL](../../concepts-and-designs/lal.md) for more details. | SW_LOG_LAL_FILES | default |
179179
| - | - | malFiles | The MAL configuration file names (without file extension) to be activated. Read [LAL](../../concepts-and-designs/lal.md) for more details. | SW_LOG_MAL_FILES | "" |
180180
| trace-analyzer | default | Trace Analyzer. | SW_TRACE_ANALYZER | default | |
181-
| - | - | talFiles | The TAL configuration file names (without file extension) to be activated, comma separated. Empty disables the span analysis while keeping the module loaded. Read [TAL](../../concepts-and-designs/tal.md) for more details. | SW_TRACE_TAL_FILES | otlp-default |
181+
| - | - | talFiles | The TAL configuration file names (without file extension) to be activated, comma separated. Empty disables the span analysis while keeping the module loaded. Read [TAL](../../concepts-and-designs/tal.md) for more details. | SW_TRACE_TAL_FILES | otlp-general |
182182
| - | - | dedupWindow | Seconds within which a span seen again on this node, by trace id and span id, is analyzed only once, so an exporter that retries a batch does not count a request twice. `0` keeps no set and treats every arrival as new. | SW_TRACE_TAL_DEDUP_WINDOW | 0 |
183-
| - | - | dedupCapacity | The most span ids the dedup set holds on this node while `dedupWindow` is above `0`; ignored at `0`. Size it as spans per second times `dedupWindow`, with headroom. About 150 bytes per entry. | SW_TRACE_TAL_DEDUP_CAPACITY | 1000000 |
184-
| - | - | pairingSamplePercent | In a cluster, the percentage of traces whose client and server spans are routed to the trace's owner node to learn a network address alias from the pair. A single OAP node pairs every call and ignores this. | SW_TRACE_TAL_PAIRING_SAMPLE_PERCENT | 25 |
183+
| - | - | dedupCapacity | The most span ids the dedup set holds on this node while `dedupWindow` is above `0`; ignored at `0`. Only a span some rule runs on takes an entry; size it as such spans per second times `dedupWindow`, with headroom. About 150 bytes per entry. | SW_TRACE_TAL_DEDUP_CAPACITY | 1000000 |
184+
| - | - | pairingSamplePercent | The percentage of traces whose client and server spans are paired to learn a network address alias, on the trace's owner node in a cluster and on the node itself when it is alone; a new caller + peer pair is learned after about four calls at 25. 0 pairs none. | SW_TRACE_TAL_PAIRING_SAMPLE_PERCENT | 25 |
185185
| ai-agent-conversation | default | Conversations of long-lived AI agents landed by the AI Sessionizer as Session Data and Session Flow files over OTLP logs under the `AI_AGENT` layer. | SW_AI_AGENT_CONVERSATION | default | - |
186186
| - | - | readWindow | How many Session Data files, or Session Flow rounds, one storage query fetches; a conversation is read batch by batch up to its head. | SW_AI_AGENT_CONVERSATION_READ_WINDOW | 16 |
187187
| - | - | conversationListMaxLimit | The most rounds one list query reads before folding to one row per conversation. | SW_AI_AGENT_CONVERSATION_LIST_MAX_LIMIT | 10000 |

‎docs/en/setup/backend/otlp-trace.md‎

Lines changed: 15 additions & 43 deletions
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@ The handler stores them in one of two ways, selected by `receiver-otel.default.o
55

66
| `otlpTraceStorage` | What is stored | Query API |
77
|--------------------|------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------|
8-
| `otlp` (default) | The span exactly as it arrived, with its resource and instrumentation scope, in the `otlp_span` record. | The TraceQL `/otlp` datasource, served to Grafana Tempo as native OTLP. |
8+
| `otlp` (default) | The span as it arrived, with its resource and instrumentation scope, in the `otlp_span` record; only the resource's `service.layer` is set to the registered layer, `UNDEFINED` otherwise. | The TraceQL `/otlp` datasource, served to Grafana Tempo as native OTLP. |
99
| `zipkin` | Spans converted to Zipkin v2 and handed to the Zipkin receiver, the only mode of earlier releases. Follows the [Zipkin Exporter](https://opentelemetry.io/docs/specs/otel/trace/sdk_exporters/zipkin/#summary) conversion rules. | [Zipkin query API](./zipkin-trace.md#zipkin-query), Lens UI, and the TraceQL `/zipkin` datasource. |
1010

1111
Span listeners such as the GenAI analyzer and AI evaluation run in both modes. In `otlp` mode they read the
@@ -113,56 +113,26 @@ The grouped name is the service everywhere OAP names one: the [trace analysis](t
113113
instances, endpoints and topology; the stored span, in both storage modes, so a TraceQL search names the service as
114114
the layer does, `{resource.service.name="opentelemetry-demo::cart"}`; and the OTLP logs, which are filed under it.
115115
The resource attributes are stored as sent, `service.name` and the grouping attribute among them, so
116-
`{resource.service.namespace="opentelemetry-demo"}` still finds the group's traces. A query names each span's
116+
`{resource.service.namespace="opentelemetry-demo"}` still finds the group's traces; only `service.layer` is stored as
117+
the registered layer it names, `UNDEFINED` otherwise, as the [service catalog](../../api/traceql-service.md) files it. A query names each span's
117118
service by the grouped name: a search lists it as the span's `service.name`, and a trace fetched by id carries it as
118119
its resource's `service.name`, the other resource attributes as sent. In the `zipkin` storage mode the stored service
119120
is lower case, since Zipkin lowercases every service name, so a group value with capitals, `service.namespace=Shop`,
120121
gives `Shop::cart` in the analysis and the logs but `shop::cart` in the stored span and the query; Kubernetes
121122
namespace names are always lower case.
122123

123-
A caller that dials the host `cart` still reaches the grouped `cart`: the trace analyzer resolves a host the way
124-
Kubernetes DNS resolves a Service name, against the services the OAP node has seen spans from, in this order:
125-
126-
1. A host qualified with a group names that group's service: `cart.outlet`, `cart.outlet.svc` and
127-
`cart.outlet.svc.cluster.local` are `outlet::cart`. A qualified host is never resolved in the caller's group by its
128-
first label, since it says which group it means; only a service of the caller's group named exactly the whole host,
129-
as in step 2, matches it there.
130-
2. A host names the service of exactly that name in the caller's own group: `cart` dialed by `shop::frontend` is
131-
`shop::cart`, and dialed by `outlet::frontend` is `outlet::cart`, even when both are deployed. A `service.name`
132-
that holds a dot is matched whole, `checkout.api` is `shop::checkout.api`, never by its first label.
133-
3. A host, or its first DNS label, that is an ungrouped service's whole name is that service.
134-
4. The network address alias row of the address, written by the callee that was dialed at it.
135-
5. The service an earlier pair of a call's CLIENT and SERVER spans named for the address.
136-
6. The one grouped service, in any group, whose short name is the whole host. This is ranked last because an OAP node
137-
that has seen only `outlet::cart` would otherwise name it for a `shop` caller; a qualified host such as
138-
`cart.outlet` is never guessed by its first label.
139-
140-
The first two steps match Kubernetes only when the group is the namespace, so list `k8s.namespace.name` first, or
141-
alone, on Kubernetes; with `service.namespace` first, a qualified host such as `cart.otel-demo` names nothing, while a
142-
bare host still resolves in the caller's group as long as caller and callee carry the same value. Both steps also need
143-
the Kubernetes Service named as the SDK's `service.name`. Calls these steps cannot name land on the address row, which
144-
is keyed by the address alone, as on the agent path, and the last callee to register it wins. So calls to a short name
145-
two groups share, `shop::cart` and `outlet::cart`, may land on either group's service when they come from an ungrouped
146-
caller, such as a browser app without the grouping attribute, or when the Kubernetes Service name differs from
147-
`service.name`. An OAP node also records, when it starts, the OTLP services storage already holds, so a restart does
148-
not forget a callee that emits only now and then, such as a feature flag service; a stored service that sends no span
149-
to the node is forgotten again a day after it was read. A client's peer is expected to name one target. When services
150-
of two groups are reached at the same address, such as a bare `cart:7070` dialed in both namespaces, and the node
151-
cannot name the callee in the caller's own group, the call is named by the address row when there is one, which may
152-
name either group's service, as on the agent path, and otherwise by the pairing or the step 6 guess. A node that never
153-
receives the callee's spans, for example one the collectors are not connected to over their long-lived gRPC
154-
connections, stays in that state once the day the startup read covers has passed. To pin which TAL rules analyze such
124+
A caller that dials the host `cart` still reaches the grouped `cart`: the trace analyzer learns a network address alias
125+
from the two spans of a call, keyed by the caller and the address it dialed, so `shop::frontend` dialing `cart:7070`
126+
reaches `shop::cart` and `outlet::frontend` dialing the same `cart:7070` reaches `outlet::cart`, see [Address-named
127+
peers](../../concepts-and-designs/tal.md#address-named-peers). No host is matched against service names or
128+
Kubernetes DNS, so the group needs no particular relation to the namespace for this, and a callee is named only once
129+
one of its caller's calls has been paired; the calls before then are drawn to the address-named node, which the
130+
topology query folds onto the service once the row lands. To pin which TAL rules analyze the
155131
spans, set their resource's `service.layer` to a registered layer name and declare TAL rules for that layer. Only
156-
those rules run; the shipped `layer: auto` rules do not. An unknown layer name is logged and its spans are not
132+
those rules run. An unknown layer name is logged and its spans are not
157133
analyzed. See [Layer](../../concepts-and-designs/tal.md#layer). The group has no cluster part either: the namespace `shop` of two Kubernetes
158134
clusters groups both clusters' `cart` as one `shop::cart`.
159135

160-
The guard that keeps a proxy's address its own works on short names too: an address whose first DNS label, or whose
161-
whole host, is the short name of any group's service is never registered for a callee of another short name, so a
162-
proxy that passes the `Host` header through keeps its node whichever group it is in. The callee's own short name is
163-
exempt for the first label only: `edge.proxy:8080` stays `infra::edge.proxy`'s even against a callee named `edge`. See
164-
[Address-named peers](../../concepts-and-designs/tal.md#address-named-peers).
165-
166136
Pick an attribute that holds one low-cardinality value per service, such as `service.namespace`,
167137
`deployment.environment.name` or `k8s.namespace.name`; a value that changes per pod or per request splits a service
168138
into many, and a service whose spans carry the attribute only some of the time is split into its grouped and its
@@ -199,5 +169,7 @@ The status module exposes the same queries with the execution trace for debuggin
199169

200170
Every accepted span also goes through the [Trace Analysis Language](../../concepts-and-designs/tal.md) rules of the
201171
`trace-analyzer` module, in both storage modes and before the head sampling and the rate limit above. The shipped
202-
rule file registers the services under the `OTLP_SERVICE` layer with their metrics, topology and slow statements;
203-
see [OpenTelemetry Trace Analysis](trace-analyzer.md) for the settings and for what it produces.
172+
rule file analyzes the spans whose resource carries `service.layer=OTLP_GENERAL`, and registers their services under
173+
that layer with their metrics, topology and slow statements; every other span is only stored. See [OpenTelemetry
174+
Trace Analysis](trace-analyzer.md#enabling-the-analysis-for-an-application) for how an application enables it, the
175+
settings, and what it produces.

0 commit comments

Comments
 (0)