Apache SkyWalking – Metrics

Blog: Monitoring WeChat and Alipay Mini Programs with SkyWalking

Thu, 30 Apr 2026 00:00:00 +0000

Mini programs are a major part of the mobile experience in China, but the open-source observability ecosystem has long focused on web browsers and native apps. SkyWalking already covers browser (client-js), iOS, and the server side; mini programs and Android were the remaining gaps. With SkyAPM/mini-program-monitor joining the SkyWalking ecosystem, the mini-program half of that gap is closed — one SDK supports both WeChat and Alipay, and the matching OAP-side component IDs, MAL rules, and UI templates are merged on main and will ship with 10.5.0.

This post is for teams that already run a SkyWalking backend and want to bring their mini programs into the same observability stack. The interesting parts aren’t that the project exists — they are how the data flows from a mini program to a SkyWalking dashboard, how the two platforms coexist, and what design trade-offs you should know about before rolling this out.

Data path

The SDK uses two protocols:

OTLP HTTP (error logs, performance metrics, request metrics) → OAP /v1/logs, /v1/metrics
SkyWalking native (distributed tracing segments, optional) → OAP /v3/segments

Why not a single protocol? OTLP already covers logs and metrics, so there’s no point reinventing native endpoints for those. But for tracing, OAP’s native SegmentObject maps more cleanly onto SkyWalking’s trace model, and sw8 header propagation to the backend works without any conversion. So traces go native, everything else goes OTLP, and neither side has to translate.

OTLP defaults to protobuf; JSON is available for debugging. The SDK has zero runtime dependencies.

Two platforms, two independent Layers and dashboards

Many teams maintain a WeChat mini program and an Alipay mini program against a shared backend. Rather than collapsing them into a single tagged service, the design promotes each platform to its own Layer — WECHAT_MINI_PROGRAM and ALIPAY_MINI_PROGRAM — with its own dashboard set. The SDK tags every signal with a resource attribute miniprogram.platform = wechat | alipay and assigns each platform its own component ID (WeChat = 10002, Alipay = 10003).

On the OAP side, the MAL rule’s filter routes data into the right Layer at ingest:

metricPrefix: meter_wechat_mp
filter: "{ tags -> tags.miniprogram_platform == 'wechat' }"

The Alipay rule mirrors this with 'alipay'. The two rules are mutually exclusive — no double counting — and produce distinct metric prefixes (meter_wechat_mp_* vs meter_alipay_mp_*) that feed each Layer’s dashboards. Even when both platforms use the same service.name (e.g. mini-program-demo), the UI exposes two completely separate entry points.

Asymmetric metric semantics

This is the design choice I want to highlight. WeChat’s base library exposes PerformanceObserver, which gives you renderer-authoritative timings: app launch, first render, route navigation, script execution, sub-package load — all real measurements. Alipay’s base library doesn’t offer an equivalent, so the SDK falls back to lifecycle hooks: the App.onLaunch → App.onShow delta is used as an approximation of launch time, and renderer-level timings simply aren’t available.

So the two MAL rule sets are deliberately not the same:

WeChat: app_launch_duration, first_render_duration, route_duration, script_duration, package_load_duration, request_duration_percentile, request_cpm
Alipay: app_launch_duration, first_render_duration, request_duration_percentile, request_cpm

The Alipay app_launch_duration is a lifecycle approximation and is not directly comparable to WeChat’s renderer timing — the dashboard tooltip says so explicitly. Putting the two numbers side by side is comparing two different measurement definitions.

What the SDK does

Four signals:

Errors — JS exceptions, unhandled promise rejections, and pageNotFound go out as OTLP logs, following the OTel exception.* semantic conventions (exception.type, exception.stacktrace). Anything downstream that speaks OTLP — SkyWalking, OTel Collector, Grafana — recognizes them.
Performance — the metrics listed above. OTLP gauge.
Requests — wx.request / my.request / downloadFile / uploadFile are reported as OTLP delta histograms, one batch per flushInterval (default 5s). The le bucket labels are already in milliseconds, and the MAL rule explicitly declares MILLISECONDS to disable the default SECONDS→MS rescale. Failed requests (4xx / 5xx / timeout) additionally emit an error log so you can pivot from a dashboard to a concrete failure.
Tracing (opt-in) — when enabled, outbound requests get sw8 header injection, and the resulting segments stitch together with backend traces into one end-to-end view. Trace data goes out as SkyWalking SegmentObject, not OTLP traces.

Two reliability and cardinality details worth calling out:

Persisting events on app hide. Mini programs get killed by the framework after some time in background, and weak networks make in-flight events easy to lose. The SDK writes unsent events to wx.setStorage / my.setStorage on onAppHide and restores them on the next launch.

Avoiding cardinality explosions. Set serviceInstance to the app version (e.g. 1.4.2), not a device ID — at a million DAU the device-ID dimension blows up the OAP instance index. For request paths, the SDK exposes urlGroupRules regex patterns to fold parameterized URLs like /api/user/12345 into /api/user/{id} so the endpoint dimension doesn’t blow up either.

What OAP needs

If you’re on main or a release ≥ 10.5.0, the following are already shipped:

config/component-libraries.yml registers WeChat-MiniProgram: 10002 and AliPay-MiniProgram: 10003
config/otel-rules/miniprogram/ holds four MAL rules — service-scoped and instance-scoped for each platform
config/ui-initialized-templates/wechat_mini_program/ and alipay_mini_program/ carry root / service / instance / endpoint dashboards
config/ui-initialized-templates/menu.yaml registers both layers under the Mobile menu group

The only thing left is enabling the OTel receiver and giving the SDK an OTLP HTTP port it can reach. SkyWalking OAP binds its OTLP HTTP handler onto the receiver-sharing-server port, and that port defaults to 0 — meaning it’s folded into the core REST port (12800). If you want the SDK to use the standard OTLP HTTP port 4318, set the sharing port to 4318:

docker run -d --name sw-oap \
  -p 11800:11800 -p 12800:12800 -p 4318:4318 \
  -e SW_STORAGE=banyandb \
  -e SW_STORAGE_BANYANDB_TARGETS=banyandb:17912 \
  -e SW_OTEL_RECEIVER=default \
  -e SW_RECEIVER_SHARING_REST_PORT=4318 \
  apache/skywalking-oap-server:latest

All receivers (OTLP, native segment, browser perf, log report) move to 4318 together, while GraphQL stays on 12800 for the UI.

Minimal SDK config:

import MiniProgramMonitor from 'mini-program-monitor';

MiniProgramMonitor.init({
  service: 'mini-program-demo',
  serviceInstance: '1.4.2',          // Recommended: app version
  collector: 'http://your-oap:4318',
  enable: {
    error: true,
    perf: true,
    request: true,
    tracing: false,                  // Off by default; enable as needed
  },
});

WeChat and Alipay use the same config — the SDK detects the platform at runtime and tags the data accordingly.

Compatibility

WeChat base library ≥ 2.11
Alipay base library ≥ 2.0
Apache SkyWalking OAP main or ≥ 10.5.0, with the OTLP HTTP receiver enabled
Any other OTLP-compatible backend (OpenTelemetry Collector, Grafana, etc.) also works, but you won’t get the SkyWalking-specific cross-platform dashboards

What’s next

To get involved, head over to SkyAPM/mini-program-monitor and open an issue or PR. The repo also ships a make preview target that boots OAP, the UI, and both platform simulators locally — handy if you want to play with it end-to-end.

Android end-user experience monitoring is still a gap in the SkyWalking ecosystem; contributors interested in closing that one are very welcome.

Blog: Query SkyWalking and Zipkin Traces with TraceQL and Visualize in Grafana

Wed, 08 Apr 2026 00:00:00 +0000

Query SkyWalking and Zipkin Traces with TraceQL and Visualize in Grafana

Apache SkyWalking introduced TraceQL support in version 10.4.0, implementing Grafana Tempo’s HTTP query APIs so that Grafana can query and visualize traces stored in SkyWalking without any additional plugins. This means you can now use the familiar Grafana Tempo data source to search, filter, and drill into both SkyWalking native traces and Zipkin-compatible traces — all served by your existing SkyWalking OAP server.

Architecture Overview

┌────────────────────┐         Tempo HTTP API           ┌─────────────────────────────┐
│                    │  ──── /skywalking/api/search ──► │  SkyWalking Native Backend  │
│      Grafana       │                                  │  (Query Traces V2 API)      │
│  (Tempo Data Src)  │                                  ├─────────────────────────────┤
│                    │  ──── /zipkin/api/search ──────► │  Zipkin-Compatible Backend  │
└────────────────────┘                                  └──────────┬──────────────────┘
                                                                   │
                                                        ┌──────────▼──────────────────┐
                                                        │    SkyWalking OAP Server    │
                                                        │  ┌───────────────────────┐  │
                                                        │  │   TraceQL Service     │  │
                                                        │  │  (port 3200)          │  │
                                                        │  └───────────────────────┘  │
                                                        │  ┌───────────────────────┐  │
                                                        │  │  Storage (BanyanDB /  │  │
                                                        │  │  Elasticsearch / …)   │  │
                                                        │  └───────────────────────┘  │
                                                        └─────────────────────────────┘

The TraceQL Service sits inside the OAP server and exposes the Tempo-compatible HTTP API on port 3200 (default). It converts traces from their native format into Tempo’s format, where the trace detail part (Trace message) reuses OTLP Trace definitions.

Limitations and Supported TraceQL Features

TraceQL is a rich query language, but SkyWalking currently implements a practical subset. The following features are supported:

Feature	Examples
Spanset filter	`{resource.service.name="frontend"}`
Resource attributes	`resource.service.name`
Span attributes	`span.http.method`, `span.http.status_code`
Intrinsic fields	`duration`, `name`, `status`
Comparison operators	`=`, `>`, `>=`, `<`, `<=`
Compound conditions	`{resource.service.name="frontend" && duration>100ms}`
Duration units	`us`/`µs`, `ms`, `s`, `m`, `h`

The following features are not yet supported:

Spanset logical operations ({...} AND {...}, {...} OR {...})
Pipeline operations (| operator)
Aggregate functions (count(), avg(), max(), min(), sum())
Regular expression matching (=~, !~)
event and link scopes
kind intrinsic field
Streaming mode (must be disabled in the Grafana Tempo data source settings)

Important: SkyWalking native trace support in TraceQL is based on the Query Traces V2 API. Currently, only BanyanDB storage implements this API. Other storage backends (e.g., Elasticsearch, MySQL, PostgreSQL) do not support SkyWalking native trace queries via TraceQL. Zipkin-compatible traces are not subject to this restriction.

Trace Format Conversion

Since the trace detail part of Tempo’s format reuses OTLP Trace definitions, the conversion descriptions below refer to OTLP field names (e.g., span kind, status code).

SkyWalking Native Trace

Trace ID Encoding

SkyWalking native trace IDs are arbitrary strings (e.g., 2a2e04e8d1114b14925c04a6321ca26c.38.17739924187687539), while Grafana Tempo requires pure hex-encoded trace IDs. The TraceQL Service encodes each UTF-8 byte of the original trace ID as two lowercase hex characters:

Original:  2a2e04e8d1114b14925c04a6321ca26c.38.17739924187687539
Encoded:   32613265303465386431313134623134393235633034613633323163613236632e33382e3137373339393234313837363837353339

This encoded hex trace ID is what appears in all API responses and in Grafana. When you click a trace ID in Grafana, the TraceQL Service automatically decodes it back to the original SkyWalking trace ID for the internal query.

Span Kind Mapping

SkyWalking Span Type	OTLP Span Kind
`Entry`	`SPAN_KIND_SERVER`
`Exit`	`SPAN_KIND_CLIENT`
`Local`	`SPAN_KIND_INTERNAL`

Status Mapping

SkyWalking `isError`	OTLP Status Code
`true`	`STATUS_CODE_ERROR`
`false`	`STATUS_CODE_OK`

SpanAttachedEvents

SkyWalking SpanAttachedEvents are converted to OTLP span events, with tags mapped as string attributes and summary mapped as numeric attributes (serialized as strings).

Zipkin Trace

Span Kind Mapping

Zipkin Span Kind	OTLP Span Kind
`CLIENT`	`SPAN_KIND_CLIENT`
`SERVER`	`SPAN_KIND_SERVER`
`PRODUCER`	`SPAN_KIND_PRODUCER`
`CONSUMER`	`SPAN_KIND_CONSUMER`

Status Mapping

If the otel.status_code tag is present, it is used directly.
Otherwise, if the error tag equals true, the status is STATUS_CODE_ERROR.
If neither tag is present, the status defaults to STATUS_CODE_UNSET.

Endpoint and Annotation Mapping

Zipkin endpoint fields are mapped to OTLP attributes (e.g., localEndpoint.ipv4 → net.host.ip), and Zipkin annotations are converted to OTLP span events.

For the full conversion details, see the TraceQL Service documentation.

How to Enable TraceQL

Step 1: Enable the TraceQL Module

By default, the TraceQL module is disabled (selector: ${SW_TRACEQL:-}). To enable it, set the selector to default:

# In application.yml
traceQL:
  selector: ${SW_TRACEQL:default}
  default:
    enableDatasourceSkywalking: ${SW_TRACEQL_ENABLE_DATASOURCE_SKYWALKING:true}
    enableDatasourceZipkin: ${SW_TRACEQL_ENABLE_DATASOURCE_ZIPKIN:true}

Or via environment variables:

export SW_TRACEQL=default
export SW_TRACEQL_ENABLE_DATASOURCE_SKYWALKING=true
export SW_TRACEQL_ENABLE_DATASOURCE_ZIPKIN=true

Step 2: Enable the Zipkin Receiver (for Zipkin traces only)

If you want to query Zipkin traces, you also need to enable the Zipkin receiver so that SkyWalking can ingest Zipkin trace data:

# In application.yml
receiver-zipkin:
  selector: ${SW_RECEIVER_ZIPKIN:default}
  default:
    searchableTracesTags: ${SW_ZIPKIN_SEARCHABLE_TAG_KEYS:http.method}
    sampleRate: ${SW_ZIPKIN_SAMPLE_RATE:10000}
    restHost: ${SW_RECEIVER_ZIPKIN_REST_HOST:0.0.0.0}
    restPort: ${SW_RECEIVER_ZIPKIN_REST_PORT:9411}

Or via environment variable:

export SW_RECEIVER_ZIPKIN=default

Full Configuration Reference

For the complete list of all configuration options and their default values, see the Configuration section of the TraceQL Service documentation.

Configuring Grafana Tempo Data Source

Prerequisite: Grafana 12 or later is required.

Each trace backend (SkyWalking native / Zipkin) needs its own Tempo data source in Grafana, because each is served under a different context path.

Context Paths

The two backends are served under separate context paths on the same port:

Backend	Default Context Path	Env Variable	Full Default URL
SkyWalking native	`/skywalking`	`SW_TRACEQL_REST_CONTEXT_PATH_SKYWALKING`	`http://<oap-host>:3200/skywalking`
Zipkin	`/zipkin`	`SW_TRACEQL_REST_CONTEXT_PATH_ZIPKIN`	`http://<oap-host>:3200/zipkin`

Setting Up the SkyWalking Data Source

In Grafana, go to Configuration → Data Sources → Add data source.
Choose Tempo.
Set the URL to http://<oap-host>:3200/skywalking.
Disable the Streaming option (SkyWalking does not support streaming mode).

Save and test the data source.

Setting Up the Zipkin Data Source

Same as above, but set the URL to http://<oap-host>:3200/zipkin.

Configuring Trace List Result Tags

When you search for traces in Grafana, the trace list panel shows a summary of each trace. The tracesListResultTags configuration controls which span tags are included in the search result and displayed as columns in the trace list.

Env Variable	Default Value	Purpose
`SW_TRACEQL_ZIPKIN_TRACES_LIST_RESULT_TAGS`	`http.method,error`	Tags shown for Zipkin traces
`SW_TRACEQL_SKYWALKING_TRACES_LIST_RESULT_TAGS`	`http.method,http.status_code,rpc.status_code,db.type,db.instance,mq.queue,mq.topic,mq.broker`	Tags shown for SkyWalking traces

Note that service.name and span.kind are always included regardless of this setting.

These tags appear as attribute columns in the Grafana Tempo trace search results, making it easier to identify and group traces at a glance:

SkyWalking native trace list:

Zipkin trace list:

You can customize these tags based on your application’s instrumentation. For example, if your services heavily use messaging, you might add mq.destination or messaging.system to the list.

Building a Trace Dashboard in Grafana

SkyWalking Native Trace Dashboard

Step 1: Explore and Save

Go to the Explore page in Grafana.
Select the Tempo data source you configured for SkyWalking (e.g., SkyWalkingTraceQL).
Run a test query, then click Add to dashboard and save it as SkyWalking Trace.

Step 2: Configure Variables

Add dashboard variables so users can filter traces dynamically (e.g., by service name):

Step 3: Add a Trace Panel

Choose a Table chart (or edit the panel you saved).
Set Query type to Search.
Set the Service Name query condition to the variable $Service.
Add other query conditions as needed (e.g., duration, span name, tags).
Test and save.

Step 4: View Trace Details

Click any trace ID in the trace panel to jump to the Explore page showing the full trace waterfall view with all spans, tags, and events:

Zipkin Trace Dashboard

The setup for Zipkin traces is identical to SkyWalking native traces — just use the Zipkin Tempo data source you configured (e.g., ZipkinTraceQL).

Zipkin trace detail view:

Summary

With TraceQL support in SkyWalking 10.4.0, you can now leverage Grafana’s powerful Tempo data source to query and visualize both SkyWalking native traces and Zipkin-compatible traces. The key points to remember:

Enable the TraceQL module by setting SW_TRACEQL=default and enabling the desired backends.
Configure separate Tempo data sources in Grafana for each backend (/skywalking and /zipkin).
Disable the Streaming option in the Grafana Tempo data source settings.
Customize result tags via SW_TRACEQL_SKYWALKING_TRACES_LIST_RESULT_TAGS and SW_TRACEQL_ZIPKIN_TRACES_LIST_RESULT_TAGS to control what’s shown in search results.
SkyWalking native trace queries require BanyanDB storage (Zipkin traces work with all storage backends).

For the complete API reference and conversion details, see the TraceQL Service documentation. For Grafana integration details, see Use Grafana As The UI.

Zh: 使用 TraceQL 查询 SkyWalking 和 Zipkin 链路追踪数据并在 Grafana 中可视化

Wed, 08 Apr 2026 00:00:00 +0000

使用 TraceQL 查询 SkyWalking 和 Zipkin 链路追踪数据并在 Grafana 中可视化

Apache SkyWalking 在 10.4.0 版本中引入了 TraceQL 支持，实现了 Grafana Tempo 的 HTTP 查询 API，使 Grafana 无需任何额外插件即可查询和可视化 SkyWalking 中存储的链路追踪数据。这意味着你现在可以使用熟悉的 Grafana Tempo 数据源来搜索、过滤和深入分析 SkyWalking 原生链路追踪和 Zipkin 兼容链路追踪 —— 所有数据都由现有的 SkyWalking OAP 服务器提供。

架构概览

┌────────────────────┐         Tempo HTTP API           ┌─────────────────────────────┐
│                    │  ──── /skywalking/api/search ──► │  SkyWalking Native Backend  │
│      Grafana       │                                  │  (Query Traces V2 API)      │
│  (Tempo Data Src)  │                                  ├─────────────────────────────┤
│                    │  ──── /zipkin/api/search ──────► │  Zipkin-Compatible Backend  │
└────────────────────┘                                  └──────────┬──────────────────┘
                                                                   │
                                                        ┌──────────▼──────────────────┐
                                                        │    SkyWalking OAP Server    │
                                                        │  ┌───────────────────────┐  │
                                                        │  │   TraceQL Service     │  │
                                                        │  │  (port 3200)          │  │
                                                        │  └───────────────────────┘  │
                                                        │  ┌───────────────────────┐  │
                                                        │  │  Storage (BanyanDB /  │  │
                                                        │  │  Elasticsearch / …)   │  │
                                                        │  └───────────────────────┘  │
                                                        └─────────────────────────────┘

TraceQL Service 位于 OAP 服务器内部，在端口 3200（默认）上暴露 Tempo 兼容的 HTTP API。它将链路追踪数据从原生格式转换为 Tempo 的格式，其中链路追踪详情部分（Trace 消息）复用了 OTLP Trace 定义。

支持的 TraceQL 特性与限制

TraceQL 是一种功能丰富的查询语言，但 SkyWalking 目前实现了一个实用的子集。以下特性已支持：

Feature	Examples
Spanset filter	`{resource.service.name="frontend"}`
Resource attributes	`resource.service.name`
Span attributes	`span.http.method`, `span.http.status_code`
Intrinsic fields	`duration`, `name`, `status`
Comparison operators	`=`, `>`, `>=`, `<`, `<=`
Compound conditions	`{resource.service.name="frontend" && duration>100ms}`
Duration units	`us`/`µs`, `ms`, `s`, `m`, `h`

以下特性暂不支持：

Spanset logical operations ({...} AND {...}, {...} OR {...})
Pipeline operations (| operator)
Aggregate functions (count(), avg(), max(), min(), sum())
Regular expression matching (=~, !~)
event and link scopes
kind intrinsic field
Streaming mode (must be disabled in the Grafana Tempo data source settings)

重要提示：TraceQL 中的 SkyWalking 原生链路追踪支持基于 Query Traces V2 API。目前只有 BanyanDB 存储实现了该 API。其他存储后端（如 Elasticsearch、MySQL、PostgreSQL）不支持通过 TraceQL 查询 SkyWalking 原生链路追踪数据。 Zipkin 兼容链路追踪不受此限制。

Trace 格式转换

由于 Tempo 格式的链路追踪详情部分复用了 OTLP Trace 定义，以下转换描述使用 OTLP 字段名称（如 span kind、status code）。

SkyWalking 原生链路追踪

Trace ID 编码

SkyWalking 原生 trace ID 是任意字符串（例如 2a2e04e8d1114b14925c04a6321ca26c.38.17739924187687539），而 Grafana Tempo 要求纯十六进制编码的 trace ID。TraceQL Service 将原始 trace ID 的每个 UTF-8 字节编码为两个小写十六进制字符：

原始值:  2a2e04e8d1114b14925c04a6321ca26c.38.17739924187687539
编码后:  32613265303465386431313134623134393235633034613633323163613236632e33382e3137373339393234313837363837353339

编码后的十六进制 trace ID 会出现在所有 API 响应和 Grafana 中。当你在 Grafana 中点击 trace ID 时，TraceQL Service 会自动将其解码回原始的 SkyWalking trace ID 进行内部查询。

Span Kind 映射

SkyWalking Span Type	OTLP Span Kind
`Entry`	`SPAN_KIND_SERVER`
`Exit`	`SPAN_KIND_CLIENT`
`Local`	`SPAN_KIND_INTERNAL`

状态映射

SkyWalking `isError`	OTLP Status Code
`true`	`STATUS_CODE_ERROR`
`false`	`STATUS_CODE_OK`

SpanAttachedEvents

SkyWalking SpanAttachedEvents 被转换为 OTLP span events，其中 tags 映射为字符串属性，summary 映射为数值属性（序列化为字符串）。

Zipkin 链路追踪

Span Kind 映射

Zipkin Span Kind	OTLP Span Kind
`CLIENT`	`SPAN_KIND_CLIENT`
`SERVER`	`SPAN_KIND_SERVER`
`PRODUCER`	`SPAN_KIND_PRODUCER`
`CONSUMER`	`SPAN_KIND_CONSUMER`

状态映射

如果存在 otel.status_code 标签，则直接使用。
否则，如果 error 标签等于 true，则状态为 STATUS_CODE_ERROR。
如果以上标签都不存在，则状态默认为 STATUS_CODE_UNSET。

Endpoint 与 Annotation 映射

Zipkin endpoint 字段被映射为 OTLP 属性（例如 localEndpoint.ipv4 → net.host.ip）， Zipkin annotations 被转换为 OTLP span events。

完整的转换详情请参阅 TraceQL Service 文档。

如何启用 TraceQL

步骤 1：启用 TraceQL 模块

默认情况下，TraceQL 模块是禁用的（selector: ${SW_TRACEQL:-}）。要启用它，将 selector 设置为 default：

# 在 application.yml 中
traceQL:
  selector: ${SW_TRACEQL:default}
  default:
    enableDatasourceSkywalking: ${SW_TRACEQL_ENABLE_DATASOURCE_SKYWALKING:true}
    enableDatasourceZipkin: ${SW_TRACEQL_ENABLE_DATASOURCE_ZIPKIN:true}

或通过环境变量设置：

export SW_TRACEQL=default
export SW_TRACEQL_ENABLE_DATASOURCE_SKYWALKING=true
export SW_TRACEQL_ENABLE_DATASOURCE_ZIPKIN=true

步骤 2：启用 Zipkin 接收器（仅用于 Zipkin 链路追踪）

如果你需要查询 Zipkin 链路追踪数据，还需要启用 Zipkin 接收器，以便 SkyWalking 能够接收 Zipkin 链路追踪数据：

# 在 application.yml 中
receiver-zipkin:
  selector: ${SW_RECEIVER_ZIPKIN:default}
  default:
    searchableTracesTags: ${SW_ZIPKIN_SEARCHABLE_TAG_KEYS:http.method}
    sampleRate: ${SW_ZIPKIN_SAMPLE_RATE:10000}
    restHost: ${SW_RECEIVER_ZIPKIN_REST_HOST:0.0.0.0}
    restPort: ${SW_RECEIVER_ZIPKIN_REST_PORT:9411}

或通过环境变量设置：

export SW_RECEIVER_ZIPKIN=default

完整配置参考

所有配置选项及其默认值的完整列表，请参阅 TraceQL Service 文档的配置章节。

配置 Grafana Tempo 数据源

前提条件：需要 Grafana 12 或更高版本。

每个链路追踪后端（SkyWalking 原生 / Zipkin）需要在 Grafana 中配置各自独立的 Tempo 数据源，因为它们分别在不同的上下文路径下提供服务。

上下文路径

两个后端在同一端口上使用不同的上下文路径提供服务：

Backend	Default Context Path	Env Variable	Full Default URL
SkyWalking native	`/skywalking`	`SW_TRACEQL_REST_CONTEXT_PATH_SKYWALKING`	`http://<oap-host>:3200/skywalking`
Zipkin	`/zipkin`	`SW_TRACEQL_REST_CONTEXT_PATH_ZIPKIN`	`http://<oap-host>:3200/zipkin`

配置 SkyWalking 数据源

在 Grafana 中，前往 Configuration → Data Sources → Add data source。
选择 Tempo。
将 URL 设置为 http://<oap-host>:3200/skywalking。
禁用 Streaming 选项（SkyWalking 不支持流式模式）。

保存并测试数据源。

配置 Zipkin 数据源

与上述步骤相同，但将 URL 设置为 http://<oap-host>:3200/zipkin。

配置链路追踪列表结果标签

在 Grafana 中搜索链路追踪时，链路追踪列表面板会显示每条追踪的摘要信息。 tracesListResultTags 配置控制哪些 span 标签会包含在搜索结果中并作为列显示在追踪列表中。

Env Variable	Default Value	Purpose
`SW_TRACEQL_ZIPKIN_TRACES_LIST_RESULT_TAGS`	`http.method,error`	Tags shown for Zipkin traces
`SW_TRACEQL_SKYWALKING_TRACES_LIST_RESULT_TAGS`	`http.method,http.status_code,rpc.status_code,db.type,db.instance,mq.queue,mq.topic,mq.broker`	Tags shown for SkyWalking traces

注意，无论此设置如何，service.name 和 span.kind 始终包含在结果中。

这些标签在 Grafana Tempo 链路追踪搜索结果中显示为属性列，方便快速识别和分组追踪数据：

SkyWalking 原生追踪列表：

Zipkin 追踪列表：

你可以根据应用程序的埋点情况自定义这些标签。例如，如果你的服务大量使用消息队列，可以在列表中添加 mq.destination 或 messaging.system。

在 Grafana 中构建链路追踪仪表板

SkyWalking 原生追踪仪表板

步骤 1：探索并保存

前往 Grafana 的 Explore 页面。
选择你为 SkyWalking 配置的 Tempo 数据源（例如 SkyWalkingTraceQL）。
运行一个测试查询，然后点击 Add to dashboard 并保存为 SkyWalking Trace。

步骤 2：配置变量

添加仪表板变量，以便用户可以动态过滤追踪数据（例如按服务名称过滤）：

步骤 3：添加追踪面板

选择 Table 图表（或编辑你保存的面板）。
将 Query type 设置为 Search。
将 Service Name 查询条件设置为变量 $Service。
根据需要添加其他查询条件（如 duration、span name、tags）。
测试并保存。

步骤 4：查看追踪详情

点击追踪面板中的任意 trace ID，即可跳转到 Explore 页面，查看完整的追踪瀑布图，包含所有 span、标签和事件：

Zipkin 追踪仪表板

Zipkin 追踪的设置与 SkyWalking 原生追踪完全相同 —— 只需使用你配置的 Zipkin Tempo 数据源（例如 ZipkinTraceQL）。

Zipkin 追踪详情视图：

总结

通过 SkyWalking 10.4.0 中的 TraceQL 支持，你现在可以利用 Grafana 强大的 Tempo 数据源来查询和可视化 SkyWalking 原生链路追踪和 Zipkin 兼容链路追踪数据。需要记住的要点：

启用 TraceQL 模块：设置 SW_TRACEQL=default 并启用所需的后端。
在 Grafana 中配置独立的 Tempo 数据源：为每个后端分别配置（/skywalking 和 /zipkin）。
禁用 Streaming 选项：在 Grafana Tempo 数据源设置中关闭流式模式。
自定义结果标签：通过 SW_TRACEQL_SKYWALKING_TRACES_LIST_RESULT_TAGS 和 SW_TRACEQL_ZIPKIN_TRACES_LIST_RESULT_TAGS 控制搜索结果中显示的内容。
SkyWalking 原生追踪查询需要 BanyanDB 存储（Zipkin 追踪支持所有存储后端）。

完整的 API 参考和转换详情，请参阅 TraceQL Service 文档。 Grafana 集成详情，请参阅使用 Grafana 作为 UI。

Blog: Monitoring LLM Applications with SkyWalking 10.4: Insights into Performance and Cost

Sun, 05 Apr 2026 00:00:00 +0000

With the deep penetration of Generative AI (GenAI) into enterprise workflows, developers face a challenging paradox: while powerful LLM capabilities are easily integrated via Spring AI or OpenAI SDKs, the actual performance and reliability of these calls remain largely invisible.

1. The “Black Box” of Cost and Performance: Is the Expensive Model Worth It?

Facing high LLM bills, organizations often only see a total sum paid to a provider, but cannot calculate the “ROI” within the application.

Blind Upgrades: You might switch to a premium flagship model for a better experience. But in your specific business scenario, does paying several times more per token actually yield lower latency or a faster TTFT (Time to First Token)?
Lack of Real-World Benchmarks: Official benchmarks mean little without your real-world business requests. You need to know which model achieves the perfect balance between “Token/Cost Consumption” and “Response Speed” under your actual prompt lengths and concurrency levels.

2. The Vanishing “Golden Timeout”

Many teams set timeouts for LLM calls arbitrarily (e.g., 30s or 60s).

Too Short: During peak periods or long-text generation, requests are frequently interrupted, causing business failure rates to soar.
Too Long: If a provider hangs, requests pile up in memory, blocking execution threads and potentially leading to the collapse of the entire Java application or microservice cluster. Only by mastering the P99/P95 Latency can you set rational timeout policies based on data rather than intuition.

3. The Overlooked Experience Killer: TTFT

In GenAI scenarios, a user’s perception of speed depends less on the total duration of the conversation and more on “when the first word appears.” * A streaming response with a 10s total duration but a 500ms TTFT feels instantaneous.

A non-streaming response with a 5s total duration but a 4s TTFT feels “frozen.” If your observability system only tracks total latency, you miss the core UX metric that explains why users complain about “AI slowness.”

SkyWalking 10.4: A “Digital Dashboard”
From the Application Perspective The Virtual GenAI capability introduced in Apache SkyWalking 10.4 fills this “observability vacuum.” It avoids reliance on external gateways by using application-side probes (like the Java Agent) to collect the most authentic data from the client’s perspective.

Precise Latency Distribution: Multi-dimensional metrics (P50, P90, P99) help visualize LLM fluctuations to inform dynamic timeout strategies.
Core UX Metric — TTFT Monitoring: Native support for first-token latency in streaming calls.
Multi-dimensional Model Profiling: Aligns token usage, estimated cost, and performance across Providers and Models, helping you choose the most cost-effective solution for your specific needs.

Virtual GenAI Observability

Virtual GenAI represents Generative AI service nodes detected by probe plugins. All performance metrics are based on the GenAI Client Perspective.

For instance, the Spring AI plugin in the Java Agent detects the response latency of a Chat Completion request. SkyWalking then visualizes these in the dashboard:

Traffic & Success Rate (CPM & SLA)
Latency & TTFT
Token Usage (Input/Output)
Estimated Cost

Screenshots:

How It Works

When the SkyWalking Java Agent or OTLP probes intercept calls to mainstream AI frameworks (e.g., Spring AI, OpenAI SDK), they report Trace data to the SkyWalking OAP. The OAP aggregates and computes this data to generate performance metrics for both Providers and Models, which are then rendered in the built-in Virtual-GenAI dashboards.

Installation & Configuration

Requirements

SkyWalking Java Agent: >= 9.7
SkyWalking OAP: >= 10.4

Semantic Conventions & Compatibility

SkyWalking Virtual GenAI follows OpenTelemetry GenAI Semantic Conventions. OAP identifies GenAI-related Spans based on:

SkyWalking Java Agent

Spans must be of type Exit, have the SpanLayer attribute set to GENAI, and contain the gen_ai.response.model tag.

OTLP / Zipkin Probes

Spans must contain the gen_ai.response.model tag.

For details, refer to the E2E configurations:

GenAI Estimated Cost Configuration

Overview

SkyWalking provides a built-in GenAI Billing Configuration File.

This file defines how SkyWalking maps model names from Trace data to their corresponding providers and estimates the token cost for each LLM call. The estimated cost is displayed in the SkyWalking UI alongside trace and metric data, helping users intuitively understand the financial impact of their GenAI usage.

Important: The pricing in this file is intended for cost estimation only and must not be treated as actual billing or invoice amounts. Users are advised to regularly verify the latest rates on the providers’ official pricing pages.

Configuration Structure

Top-level Fields

Field	Type	Description
`last-updated`	`date`	The last update date of the pricing data. All prices are based on public billing standards announced by providers prior to this date.
`providers`	`list`	List of GenAI provider definitions. Each entry contains matching rules and specific model pricing information.

Provider Definition

Each entry under providers defines a GenAI provider:

providers:
- provider: <provider-name>
  prefix-match:
    - <prefix-1>
    - <prefix-2>
  models:
    - name: <model-name>
      aliases: [<alias-1>, <alias-2>]
      input-estimated-cost-per-m: <cost>
      output-estimated-cost-per-m: <cost>

Field	Type	Required	Description
`provider`	`string`	Yes	The provider identifier (e.g., `openai`, `anthropic`, `gemini`). It is displayed as the Virtual GenAI service name in SkyWalking.
`prefix-match`	`list[string]`	Yes	A list of prefixes used to match model names to this provider. If a model name in the Trace data starts with any of these prefixes, it will be mapped to this provider.
`models`	`list[model]`	No	A list of model definitions containing pricing information. If omitted, the system can still identify the provider but will not perform cost estimation.

Model Definition

Each entry under models defines the pricing for a specific model:

Field	Type	Required	Description
`name`	`string`	Yes	The standard model name used for matching.
`aliases`	`list[string]`	No	Alternative names that should resolve to the same billing entry. This is useful when providers use different naming conventions (see the “Model Aliases” section).
`input-estimated-cost-per-m`	`float`	No	Estimated cost per 1,000,000 (one million) input (Prompt) tokens. The default unit is USD.
`output-estimated-cost-per-m`	`float`	No	Estimated cost per 1,000,000 (one million) output (Completion) tokens. The default unit is USD.

Model Matching Mechanism

Provider-Level Prefix Matching

When SkyWalking receives a Trace containing a GenAI call, it determines the Provider based on the following priority order:

gen_ai.provider.name tag: This tag is retrieved first. It follows the latest OpenTelemetry GenAI semantic conventions.
gen_ai.system tag: If the above tag is missing, the system falls back to this legacy tag. Note: This tag is only parsed when processing OTLP or Zipkin format data, primarily for compatibility with older versions of libraries like the Python auto-instrumentation.
Prefix Matching: If neither of the above tags exists, SkyWalking reads the prefix-match rules defined in gen-ai-config.yml and attempts to identify the provider by matching the Model Name.

- provider: openai
  prefix-match:
    - gpt

Any model name starting with gpt (such as gpt-4o, gpt-4.1-mini, or gpt-5-nano) will be mapped to the openai provider. A single provider can have multiple prefixes:

- provider: tencent
  prefix-match:
    - hunyuan
    - Tencent

Model-level Longest-Prefix Matching

Once the provider is determined, SkyWalking uses a Trie-based longest-prefix matching algorithm to find the best billing entry. This is crucial because model names returned in provider API responses often include version numbers or timestamps, differing from the base model name in the config. Example OpenAI config:

models:
- name: gpt-4o
  input-estimated-cost-per-m: 2.5
  output-estimated-cost-per-m: 10.0
- name: gpt-4o-mini
  input-estimated-cost-per-m: 0.15
  output-estimated-cost-per-m: 0.6

Matching behavior:

Model Name in Trace	Matched Configuration Entry	Reason
`gpt-4o`	`gpt-4o`	Exact match
`gpt-4o-2024-08-06`	`gpt-4o`	Longest prefix is `gpt-4o`
`gpt-4o-mini`	`gpt-4o-mini`	Exact match (Longer prefix `gpt-4o-mini` takes priority over `gpt-4o`)
`gpt-4o-mini-2024-07-18`	`gpt-4o-mini`	Longest prefix is `gpt-4o-mini`

This mechanism ensures versioned API model names map to the correct pricing tier without requiring exact full names in the configuration file.

Model Aliases

Some providers use different naming conventions across API responses and documentation. For example, Anthropic’s model might appear as claude-4-sonnet or claude-sonnet-4. The aliases field supports both formats under a single billing entry:

- name: claude-4-sonnet
  aliases: [claude-sonnet-4]
  input-estimated-cost-per-m: 3.0
  output-estimated-cost-per-m: 15.0

Under this configuration, claude-4-sonnet and claude-sonnet-4 (as well as any versioned variants, such as claude-sonnet-4-20250514) will resolve to the same billing entry.
Note: Aliases also participate in longest prefix matching. Therefore, claude-sonnet-4-20250514 will match the alias claude-sonnet-4, which in turn resolves to the pricing information for claude-4-sonnet.

Custom Configuration

Adding a New Provider

To add a provider that is not included in the default configuration:

providers:
# ... Existing providers ...

- provider: ollama
  prefix-match:
    - mymodel
  models:
    - name: mymodel-large
      input-estimated-cost-per-m: 1.0
      output-estimated-cost-per-m: 5.0
    - name: mymodel-small
      input-estimated-cost-per-m: 0.1
      output-estimated-cost-per-m: 0.5

For OTLP/Zipkin data, a dedicated estimated tag has been added. You can now view the cost of each GenAI call directly on the UI.

Main Metrics

1.Provider Level

Metric ID	Description	Meaning
`gen_ai_provider_cpm`	Calls Per Minute	Requests per minute (Throughput)
`gen_ai_provider_sla`	Success Rate	Request success rate
`gen_ai_provider_resp_time`	Avg Response Time	Average response time
`gen_ai_provider_latency_percentile`	Latency Percentiles	Response time percentiles (P50, P75, P90, P95, P99)
`gen_ai_provider_input_tokens_sum/avg`	Input Token Usage	Total and average input token usage
`gen_ai_provider_output_tokens_sum/avg`	Output Token Usage	Total and average output token usage
`gen_ai_provider_total_estimated_cost/avg`	Estimated Cost	Total estimated cost and average cost per call

2. Model Level

Metric ID	Description	Meaning
`gen_ai_model_call_cpm`	Calls Per Minute	Requests per minute for this specific model
`gen_ai_model_sla`	Success Rate	Model-specific request success rate
`gen_ai_model_latency_avg/percentile`	Latency	Average and percentiles of model response duration
`gen_ai_model_ttft_avg/percentile`	TTFT	Time to First Token (Streaming only)
`gen_ai_model_input_tokens_sum/avg`	Input Token Usage	Detailed input token consumption for the model
`gen_ai_model_output_tokens_sum/avg`	Output Token Usage	Detailed output token consumption for the model
`gen_ai_model_total_estimated_cost/avg`	Estimated Cost	Estimated total cost and average cost for the model

Recommended Usage Scenarios

Performance Evaluation: Use Latency and Time to First Token (TTFT) metrics to analyze model inference efficiency and the end-user interaction experience.
Token Monitoring: Real-time monitoring of Input and Output token consumption to analyze resource utilization across different business scenarios.
Cost Alerting: Set alert thresholds based on Estimated Cost or token consumption to promptly detect abnormal calls and prevent budget overruns.

Blog: Monitoring Flink with SkyWalking

Fri, 25 Apr 2025 00:00:00 +0000

Background

Apache Flink is a framework and distributed processing engine for stateful computations over unbounded and bounded data streams. Flink has been designed to run in all common cluster environments, perform computations at in-memory speed and at any scale.

Apache SkyWalking is an application performance monitor tool for distributed systems, especially designed for microservices, cloud native and container-based (Kubernetes) architectures.

OpenTelemetry is a collection of APIs, SDKs, and tools. Use it to instrument, generate, collect, and export telemetry data (metrics, logs, and traces) to help you analyze your software’s performance and behavior.

Since SkyWalking 10.3, a new out-of-the-box feature has been introduced that enables Flink monitoring data to be visualized on the SkyWalking UI via the OpenTelemetry Collector, which gathers metrics from Flink endpoints.

Development

Preparation

Process

Set up SkyWalking oap and UI.
Set up the Flink cluster By configuring jobmanager and taskmanager to expose prometheus http endpoints.
Set up OpenTelemetry-collector.
Run your job.

Data flow

Configuration

docker-compose

version: "3"

services:
  oap:
    extends:
      file: ../../script/docker-compose/base-compose.yml
      service: oap
    ports:
      - "12800:12800"
    networks:
      - e2e

  banyandb:
    extends:
      file: ../../script/docker-compose/base-compose.yml
      service: banyandb
    ports:
      - 17912

  jobmanager:
    image: flink:2.0-preview1
    environment:
      - |
        FLINK_PROPERTIES=
        jobmanager.rpc.address: jobmanager
        metrics.reporter.prom.factory.class: org.apache.flink.metrics.prometheus.PrometheusReporterFactory
        metrics.reporter.prom.port: 9260
    ports:
      - "8081:8081"
      - "9260:9260"
    command: jobmanager
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8081"]
      interval: 30s
      timeout: 10s
      retries: 3
    networks:
      - e2e

  taskmanager:
    image: flink:2.0-preview1
    environment:
      - |
        FLINK_PROPERTIES=
        jobmanager.rpc.address: jobmanager
        metrics.reporter.prom.factory.class: org.apache.flink.metrics.prometheus.PrometheusReporterFactory
        metrics.reporter.prom.port: 9261
    depends_on:
      jobmanager:
        condition: service_healthy
    ports:
      - "9261:9261"
    command: taskmanager
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9261/metrics"]
      interval: 30s
      timeout: 10s
      retries: 3
    networks:
      - e2e

  executeJob:
    image: flink:2.0-preview1
    depends_on:
      taskmanager:
        condition: service_healthy
    command: >
      bash -c "
      ./bin/flink run -m jobmanager:8081 examples/streaming/WindowJoin.jar"
    networks:
      - e2e

  otel-collector:
    image: otel/opentelemetry-collector:${OTEL_COLLECTOR_VERSION}
    networks:
      - e2e
    command: [ "--config=/etc/otel-collector-config.yaml" ]
    volumes:
      - ./otel-collector-config.yaml:/etc/otel-collector-config.yaml
    expose:
      - 55678
    depends_on:
      oap:
        condition: service_healthy

networks:
  e2e:

If you plan to expose metrics data using the pushGateway pattern, please refer to the documentation.

OpenTelemetry-collector

receivers:
  prometheus:
    config:
      scrape_configs:
        - job_name: "flink-jobManager-monitoring"
          scrape_interval: 30s
          static_configs:
            - targets: ['jobmanager:9260']
              labels:
                cluster: flink-cluster
          relabel_configs:
            - source_labels: [ __address__ ]
              target_label: jobManager_node
              replacement: $$1
          metric_relabel_configs:
            - source_labels: [ job_name ]
              action: replace
              target_label: flink_job_name
              replacement: $$1
            - source_labels: [ ]
              target_label: job_name
              replacement: flink-jobManager-monitoring

        - job_name: "flink-taskManager-monitoring"
          scrape_interval: 30s
          static_configs:
            - targets: [ "taskmanager:9261" ]
              labels:
                cluster: flink-cluster
          relabel_configs:
            - source_labels: [ __address__ ]
              regex: (.+)
              target_label: taskManager_node
              replacement: $$1
          metric_relabel_configs:
            - source_labels: [ job_name ]
              action: replace
              target_label: flink_job_name
              replacement: $$1
            - source_labels: [ ]
              target_label: job_name
              replacement: flink-taskManager-monitoring

exporters:
  otlp:
    endpoint: oap:11800
    tls:
      insecure: true

processors:
  batch:
service:
  pipelines:
    metrics:
      receivers:
        - prometheus
      processors:
        - batch
      exporters:
        - otlp

Warning:
Please do not edit the value of the job_name configuration, otherwise SkyWalking will not handle these data.
oap means the address of your SkyWalking oap address,please replace it accordingly.
Since the original Flink metrics contain the job_name labels, and SkyWalking relies on the job_name label to handle OpenTelemetry data, to avoid conflicts, we use metric_relabel_configs to rename the original job_name label to flink_job_name.

Metrics Definition

Monitoring metrics involve in Cluster Metrics, TaskManager Metrics, and Job Metrics.

Cluster Metrics

Cluster Metrics mainly focuses on statistics from the perspective of the entire cluster, as well as displaying JVM-related metrics of the JobManager, such as:

Running Jobs：The number of currently running jobs.
TaskManagers：The number of TaskManagers.
Task Managers Slots Total：The total number of TaskManager slots.
Task Managers Slots Available：The number of available TaskManager slots.
JVM CPU Load：The CPU load of the JobManager’s JVM.

TaskManager Metrics

TaskManager Metrics mainly focuses on statistics from the perspective of individual TaskManager nodes, such as:

JVM Memory Heap Used：The amount of JVM heap memory used on the TaskManager node.
JVM Memory Heap Available：The amount of JVM heap memory available on the TaskManager node.
NumRecordsIn：The number of records received per minute by the TaskManager.
NumBytesInPerSecond：The number of bytes received per second by the TaskManager.
IsBackPressured：Indicates whether the TaskManager node is under backpressure.
IdleTimeMsPerSecond：The idle time per second of the TaskManager node.

Job Metrics

Job Metricsmainly focuses on statistics from the perspective of running jobs, such as:

Job RunningTime：The duration for which the job has been running.
Job Restart Number：The number of times the job has been restarted.
Checkpoints Failed：The number of failed checkpoints.
NumBytesInPerSecond：The number of bytes received per second by the job.

You can find explanations for each metric in the tip of the corresponding chart.

References

Blog: New Features of SkyWalking Go 0.2.0

Mon, 31 Jul 2023 00:00:00 +0000

Announcing Apache SkyWalking Go 0.2.0

I’m excited to announce the release of Apache SkyWalking Go 0.2.0! This version packs several awesome new features that I’ll overview below.

Log Reporting

The log reporting feature allows the Go agent to automatically collect log content from supported logging frameworks like logrus and zap. The logs are organized and sent to the SkyWalking backend for visualization. You can see how the logs appear for each service in the SkyWalking UI:

Making Logs Searchable

You can configure certain log fields to make them searchable in SkyWalking. Set the SW_AGENT_LOG_REPORTER_LABEL_KEYS environment variable to include additional fields beyond the default log level.

For example, with logrus:

# define log with fields
logrus.WithField("module", "test-service").Info("test log")

Metrics Reporting

The agent can now collect and report custom metrics data from runtime/metrics to the backend. Supported metrics are documented here.

Automatic Instrumentation

In 0.1.0, you had to manually integrate the agent into your apps. Now, the new commands can automatically analyze and instrument projects at a specified path, no code changes needed! Try using the following command to import skywalking-go into your project:

# inject to project at current path
skywalking-go-agent -inject=./ -all

Or you can still use the original manual approach if preferred.

Get It Now!

Check out the CHANGELOG for the full list of additions and fixes. I encourage you to try out SkyWalking Go 0.2.0 today! Let me know if you have any feedback.

Blog: Build Grafana dashboards for Apache SkyWalking -- Native PromQL Support

Thu, 16 Mar 2023 00:00:00 +0000

Background

As an application performance monitoring tool for distributed systems, Apache SkyWalking provides monitoring, tracing, diagnosing capabilities for distributed system in Cloud Native architecture. Prometheus is an open-source systems monitoring and alerting toolkit with an active ecosystem. Especially Prometheus metrics receive widespread support through exporters and integrations. PromQL as Prometheus Querying Language containing a set of expressions and expose HTTP APIs to read metrics.

SkyWalking supports to ingest Prometheus metrics through OpenTelemetry collector and through the aggregate calculation of these metrics to provide a variety of systems monitoring, such as Linux Monitoring and Kubernetes monitoring. SkyWalking already provides native UI and GraphQL API for users. But as designed to provide wider ecological integration capabilities, since 9.4.0, it provides PromQL Service, the third-party systems or visualization platforms that already support PromQL (such as Grafana), could obtain metrics through it. SkyWalking users will benefit from it when they integrate with different systems.

What is PromQL Service in SkyWalking?

PromQL Service is a query engine on the top of SkyWalking native GraphQL query, with additional query stage calculation capabilities powered by Prometheus expressions. It can accept PromQL HTTP API requests, parse Prometheus expressions, and transform between Prometheus metrics and SkyWalking metrics.

The PromQL Service follows all PromQL’s protocols and grammar and users can use it as they would with PromQL. As SkyWalking is fundamentally different from Prometheus in terms of metric classification, format, storage, etc. PromQL Service doesn’t have to implement the full PromQL feature. Refer to the documentation for the detail.

SkyWalking Basic Concepts

Here are some basic concepts and differences from Prometheus that users need to understand in order to use the PromQL service: Prometheus metrics specify the naming format and structure, the actual metric names and labels are determined by the client provider, and the details are stored. The user aggregates and calculates the metrics using the expression in PromQL. Unlike Prometheus, SkyWalking’s metric mechanism is built around the following core concepts with a hierarchical structure:

Layer: represents an abstract framework in computer science, such as Operating System(OS_LINUX layer), Kubernetes(k8s layer). This layer would be the owner of different services detected from different technologies. All Layers definitions can be found here.
Service: Represents a set/group of workloads which provides the same behaviors for incoming requests.
Service Instance: An individual workload in the Service group.
Endpoint: A path in a service for incoming requests.
Process: An operating system process. In some scenarios, a service instance is not a process, such as a pod Kubernetes could contain multiple processes.

The metric name and properties (labels) are configured by the SkyWalking OAP server based on the data source as well as OAL and MAL. SkyWalking provides the ability to down-sampling time series metrics, and generate different time bucket data (minute, hour, day).

The SkyWalking metric stream is as follows:

Traffic

The metadata of the Service/ServiceRelation/Instance/ServiceInstanceRelation/Endpoint/EndpointRelation/Process/ProcessRelation. Include names, layers, properties, relations between them, etc.

Metric

Name: metric name, configuration from OAL and MAL.
Entity: represents the metrics’ belonging and used for the query. An Entity will contain the following information depending on the Scope： Scope represents the metrics level and in query stage represents the Scope catalog, Scope catalog provides high-dimension classifications for all scopes as a hierarchy structure.

Scope	Entity Info
Service	Service(include layer info)
ServiceInstance	Service, ServiceInstance
Endpoint	Service, Endpoint
ServiceRelation	Service, DestService
ServiceInstanceRelation	ServiceInstance, DestServiceInstance
EndpointRelation	Endpoint, DestEndpoint
Process	Service, ServiceInstance, Process
ProcessRelation	Process, ServiceInstance, DestProcess

Value:

single value: long.
labeled value: text, label1,value1|label2,value2|..., such as L2 aggregation,5000 | L1 aggregation,8000.

TimeBucket: the time is accurate to minute, hour, day.

How to use PromQL Service

Setup

PromQL Service is enabled by default after v9.4.0, so no additional configuration is required. The default ports, for example, can be configured by using OAP environment variables:

restHost: ${SW_PROMQL_REST_HOST:0.0.0.0}
restPort: ${SW_PROMQL_REST_PORT:9090}
restContextPath: ${SW_PROMQL_REST_CONTEXT_PATH:/}
restMaxThreads: ${SW_PROMQL_REST_MAX_THREADS:200}
restIdleTimeOut: ${SW_PROMQL_REST_IDLE_TIMEOUT:30000}
restAcceptQueueSize: ${SW_PROMQL_REST_QUEUE_SIZE:0}

Use Prometheus expression

PromQL matches metric through the Prometheus expression. Here is a typical Prometheus metric.

To match the metric, the Prometheus expression is as follows:

In the PromQL Service, these reserved labels would be parsed as the metric name and entity info fields with other labels for the query. The mappings are as follows.

SkyWalking Concepts	Prometheus expression
Metric name	Metric name
Layer	Label
Service	Label
ServiceInstance	Label<service_instance>
Endpoint	Label
…	…

For example, the following expressions are used to match query metrics: service_cpm, service_instance_cpm, endpoint_cpm

service_cpm{service='agent::songs', layer='GENERAL'}
service_instance_cpm{service='agent::songs', service_instance='agent::songs_instance_1', layer='GENERAL'}
endpoint_cpm{service='agent::songs', endpoint='GET:/songs', layer='GENERAL'}

Typical Query Example

At here, we take the SkyWalking Showcase deployment as the playground to demonstrate how to use PromQL for SkyWalking metrics.

The following examples can be used to query the metadata and metrics of services through PromQL Service.

Get metrics names

Query:

http://localhost:9099/api/v1/label/__name__/values

Result:

{
    "status": "success",
    "data": [
        "meter_mysql_instance_qps",
        "service_cpm",
        "envoy_cluster_up_rq_active",
        "instance_jvm_class_loaded_class_count",
        "k8s_cluster_memory_requests",
        "meter_vm_memory_used",
        "meter_apisix_sv_bandwidth_unmatched",
        "meter_vm_memory_total",
        ...
    ]
}

Select a metric and get the labels

Query:

http://localhost:9099/api/v1/labels?match[]=service_cpm

Result:

{
  "status": "success",
  "data": [
    "layer",
    "service",
    "top_n",
    "order"
  ]
}

Get services from a specific layer

Query:

http://127.0.0.1:9099/api/v1/series?match[]=service_traffic{layer='GENERAL'}&start=1677479336&end=1677479636

Result:

{
    "status": "success",
    "data": [
        {
            "__name__": "service_traffic",
            "service": "agent::songs",
            "scope": "Service",
            "layer": "GENERAL"
        },
        {
            "__name__": "service_traffic",
            "service": "agent::recommendation",
            "scope": "Service",
            "layer": "GENERAL"
        },
        {
            "__name__": "service_traffic",
            "service": "agent::app",
            "scope": "Service",
            "layer": "GENERAL"
        },
        {
            "__name__": "service_traffic",
            "service": "agent::gateway",
            "scope": "Service",
            "layer": "GENERAL"
        },
        {
            "__name__": "service_traffic",
            "service": "agent::frontend",
            "scope": "Service",
            "layer": "GENERAL"
        }
    ]
}

Query specific metric for a service

Query:

http://127.0.0.1:9099/api/v1/query?query=service_cpm{service='agent::songs', layer='GENERAL'}

Result:

{
  "status": "success",
  "data": {
    "resultType": "vector",
    "result": [
      {
        "metric": {
          "__name__": "service_cpm",
          "layer": "GENERAL",
          "scope": "Service",
          "service": "agent::songs"
        },
        "value": [
          1679559960,
          "6"
        ]
      }
    ]
  }
}

About the range query and different metrics type for query can refer to the document here.

Build Grafana Dashboard

From the above, we know the mechanism and how to query from PromQL Service, now we can build the Grafana Dashboard for the above service example. Note: All the following configurations are based on Grafana version 9.1.0.

SkyWalking Showcase provides dashboards files such as services of General and Service Mesh layers， we can quickly create a dashboard for the General layer service by importing the dashboard JSON file.

After the Grafana application is deployed, follow the steps below:

Configure Data Source

First, we need to create a data source: In the data source config panel, chose Prometheus and set the URL to the OAP server address, the default port is 9090. Here set the data source name SkyWalking in case there are multiple Prometheus data sources.

Import Dashboard File

Create a dashboard folder named SkyWalking.
Import the dashboard file into Grafana, there are two ways to get the file:
1. From SkyWalking Showcase.
2. Go to SkyWaking Demo: Preview metrics on Grafana, and export it from the General Service dashboard.
Done! Now we can see the dashboard is working, the services are in the drop-down list and the metrics are displayed on the panels.

This is an easy way to build, but we need to know how it works if we want to customize it.

How the dashboard works

Dashboard Settings

Open the Settings-Variables we can see the following variables:

Let’s look at what each variable does:

$DS_SkyWalking

This is a data source ty variable that specifies the Prometheus data source which was defined earlier as SkyWalking.
$layer

This is a constant type because in the ‘General Service’ dashboard, all services belong to the ‘GENERAL’ layer, so they can be used directly in each query Note When you customize other layers, this value must be defined in the Layer mentioned above.
$service

Query type variable, to get all service names under this layer for the drop-down list.

Query expression:
```
label_values(service_traffic{layer='$layer'}, service)
```
The query expression will query HTTP API /api/v1/series for service metadata in $layer and fetch the service name according to the label(service).
$service_instance

Same as the $service is a query variable that is used to select all instances of the service in the drop-down list.

Query expression:
```
label_values(instance_traffic{layer='$layer', service='$service'}, service_instance)
```
The query expression here not only specifies the $layer but also contains the variable $service, which is used to correlate with the services for the drop-down list.
$endpoint

Same as the $service is a query variable that is used to select all endpoints of the service in the drop-down list.

Query expression:
```
label_values(endpoint_traffic{layer='$layer', service='$service', keyword='$endpoint_keyword', limit='$endpoint_limit'}, endpoint)
```
The query expression here specifies the $layer and $service which are used to correlate with the services for the drop-down list. And also accept variables $endpoint_keyword and $endpoint_limit as filtering condition.
$endpoint_keyword

A text type variable that the user can input to filter the return value of $endpoint.
$endpoint_limit

Custom type, which the user can select to limit the maximum number of returned endpoints.

Panel Configurations

There are several typical metrics panels on this dashboard, let’s see how it’s configured.

Common Value Metrics

Select Time series chart panel Service Apdex and click edit.

Query expression
```
service_apdex{service='$service', layer='$layer'} / 10000
```
The metric scope is Service, add labels service and layer for the match, and the label value used the variables configured above. The calculation Divided by 10000 is used for matching the result units. The document for the query can refer to here.
Set Query options --> Min interval = 1m, because the metrics min time bucket in SkyWalking is 1m.
Set Connect null values --> Always and Show points --> Always because when the query interval > 1 hour or 1 day SkyWalking returns the hour/day step metrics values.

Labeled Value Metrics

Select Time series chart panel Service Response Time Percentile and click edit.

Query expression
```
service_percentile{service='$service', layer='$layer', labels='0,1,2,3,4', relabels='P50,P75,P90,P95,P99'}
```
The metric scope is Service, add labels service and layer for the match, and the label value used the variables configured above. Add labels='0,1,2,3,4' filter the result label, and addrelabels='P50,P75,P90,P95,P99' rename the result label. The document for the query can refer to here.
Set Query options --> Min interval = 1m, because the metrics min time bucket in SkyWalking is 1m.
Set Connect null values --> Always and Show points --> Always because when the query interval > 1 hour or 1 day SkyWalking returns the hour/day step metrics values.
Set Legend to {{label}} for show up.

Sort Metrics

Select Time series chart panel Service Response Time Percentile and click edit.

Query expression
```
service_instance_cpm{parent_service='$service', layer='$layer', top_n='10', order='DES'}
```
The expression is used for query the sore metrics under service, so add labels parent_service and layer for the match. Add top_n='10' and order='DES' filter the result. The document for the query can refer to here.
Set Query options --> Min interval = 1m, because the metrics min time bucket in SkyWalking is 1m.
Set the Calculation --> Latest*.
Set Legend to {{service_instance}} for show up.

Conclusion

In this article, we introduced what is the PromQL Service in SkyWalking and its background. Detailed how to use PromQL Service and the basic concepts related to SkyWalking, and show how to use PromQL Service to build Grafana dashboards for SkyWalking.

In the future, there will be more integrations by leveraging this protocol, such as CI/CD, HPA (scaling), etc.

Zh: 为 Apache SkyWalking 构建 Grafana Dashboard —— 原生 PromQL 支持

Thu, 16 Mar 2023 00:00:00 +0000

背景

Apache SkyWalking 作为分布式系统的应用性能监控工具，提供了对云原生架构下的分布式系统的监控、跟踪、诊断能力。Prometheus 是一个开源系统监控和警报工具包，具有活跃的生态系统。特别是 Prometheus 指标通过导出器和集成得到广泛支持。 PromQL 作为 Prometheus 查询语言，包含一组表达式并公开 HTTP API 以读取指标。

SkyWalking 支持通过 OpenTelemetry 收集器摄取 Prometheus 指标，并通过这些指标的聚合计算提供多种系统监控，例如 Linux 监控和 Kubernetes 监控。SkyWalking 已经为用户提供了原生 UI 和 GraphQL API。但为了提供更广泛的生态整合能力，从 9.4.0 开始，它提供了 PromQL 服务，已经支持 PromQL 的第三方系统或可视化平台（如 Grafana），可以通过它获取指标。SkyWalking 用户在与不同系统集成时将从中受益。

SkyWalking 中的 PromQL 服务是什么？

PromQL 服务是 SkyWalking 原生 GraphQL 查询之上的查询引擎，具有由 Prometheus 表达式提供支持的附加查询阶段计算能力。它可以接受 PromQL HTTP API 请求，解析 Prometheus 表达式，并在 Prometheus 指标和 SkyWalking 指标之间进行转换。

PromQL 服务遵循 PromQL 的所有协议和语法，用户可以像使用 PromQL 一样使用它。由于 SkyWalking 在度量分类、格式、存储等方面与 Prometheus 有根本不同，因此 PromQL 服务不必实现完整的 PromQL 功能。有关详细信息，请参阅文档。

SkyWalking 基本概念

以下是用户使用 PromQL 服务需要了解的一些基本概念和与 Prometheus 的区别： Prometheus 指标指定命名格式和结构，实际指标名称和标签由客户端提供商确定，并存储详细信息。用户使用 PromQL 中的表达式聚合和计算指标。与 Prometheus 不同，SkyWalking 的度量机制是围绕以下具有层次结构的核心概念构建的：

层（Layer）：表示计算机科学中的一个抽象框架，如 Operating System（OS_LINUX 层）、Kubernetes（k8s 层）。该层将是从不同技术检测到的不同服务的所有者。可以在此处

找到所有层定义。
服务：表示一组 / 一组工作负载，它为传入请求提供相同的行为。
服务实例：服务组中的单个工作负载。
端点：传入请求的服务路径。
进程：操作系统进程。在某些场景下，service instance 不是一个进程，比如一个 Kubernetes Pod 可能包含多个进程。

Metric 名称和属性（标签）由 SkyWalking OAP 服务器根据数据源以及 OAL 和 MAL 配置。SkyWalking 提供了对时间序列指标进行下采样（down-sampling），并生成不同时间段数据（分钟、小时、天）的能力。

SkyWalking 指标流如下：

流量

Service/ServiceRelation/Instance/ServiceInstanceRelation/Endpoint/EndpointRelation/Process/ProcessRelation 的元数据。包括名称、层、属性、它们之间的关系等。

指标

名称（Name）：指标名称，来自 OAL 和 MAL 的配置。
实体（Entity）：表示指标的归属，用于查询。一个 Entity 根据 Scope 不同会包含如下信息： Scope 代表指标级别，在查询阶段代表 Scope catalog，Scope catalog 为所有的 scope 提供了高维的分类，层次结构。

Scope	实体信息
Service	服务（包括图层信息）
ServiceInstance	服务、服务实例
Endpoint	服务、端点
ServiceRelation	服务，目标服务
ServiceInstanceRelation	服务实例、目标服务实例
EndpointRelation	端点、目标端点
Process	服务、服务实例、流程
ProcessRelation	进程、服务实例、DestProcess

值：

单值：long
标签值：文本，label1,value1|label2,value2|... ，例如 L2 aggregation,5000 | L1 aggregation,8000

TimeBucket：时间精确到分钟、小时、天

如何使用 PromQL 服务

设置

PromQL 服务在 v9.4.0 之后默认开启，不需要额外配置。例如，可以使用 OAP 环境变量配置默认端口：

restHost: ${SW_PROMQL_REST_HOST:0.0.0.0}
restPort: ${SW_PROMQL_REST_PORT:9090}
restContextPath: ${SW_PROMQL_REST_CONTEXT_PATH:/}
restMaxThreads: ${SW_PROMQL_REST_MAX_THREADS:200}
restIdleTimeOut: ${SW_PROMQL_REST_IDLE_TIMEOUT:30000}
restAcceptQueueSize: ${SW_PROMQL_REST_QUEUE_SIZE:0}

使用 Prometheus 表达式

PromQL 通过 Prometheus 表达式匹配指标。这是一个典型的 Prometheus 指标。

为了匹配指标，Prometheus 表达式如下：

在 PromQL 服务中，这些保留的标签将被解析为度量名称和实体信息字段以及用于查询的其他标签。映射如下。

SkyWalking 概念	Prometheus 表达
指标名称	指标名称
层	标签
服务	标签
服务实例	标签 <服务实例>
端点	标签
……	……

例如，以下表达式用于匹配查询指标：service_cpm、service_instance_cpm、endpoint_cpm

service_cpm {service='agent::songs', layer='GENERAL'}
service_instance_cpm {service='agent::songs', service_instance='agent::songs_instance_1', layer='GENERAL'}
endpoint_cpm {service='agent::songs', endpoint='GET:/songs', layer='GENERAL'}

典型查询示例

在这里，我们将 SkyWalking Showcase 部署作为 Playground 来演示如何使用 PromQL 获取 SkyWalking 指标。

以下示例可用于通过 PromQL 服务查询服务的元数据和指标。

获取指标名称

查询：

http://localhost:9099/api/v1/label/__name__/values

结果：

{
    "status": "success",
    "data": [
        "meter_mysql_instance_qps",
        "service_cpm",
        "envoy_cluster_up_rq_active",
        "instance_jvm_class_loaded_class_count",
        "k8s_cluster_memory_requests",
        "meter_vm_memory_used",
        "meter_apisix_sv_bandwidth_unmatched",
        "meter_vm_memory_total",
        ...
    ]
}

选择一个指标并获取标签

查询：

http://localhost:9099/api/v1/labels?match []=service_cpm

结果：

{
  "status": "success",
  "data": [
    "layer",
    "service",
    "top_n",
    "order"
  ]
}

从特定层获取服务

查询：

http://127.0.0.1:9099/api/v1/series?match []=service_traffic {layer='GENERAL'}&start=1677479336&end=1677479636

结果：

{
    "status": "success",
    "data": [
        {"__name__": "service_traffic",
            "service": "agent::songs",
            "scope": "Service",
            "layer": "GENERAL"
        },
        {"__name__": "service_traffic",
            "service": "agent::recommendation",
            "scope": "Service",
            "layer": "GENERAL"
        },
        {"__name__": "service_traffic",
            "service": "agent::app",
            "scope": "Service",
            "layer": "GENERAL"
        },
        {"__name__": "service_traffic",
            "service": "agent::gateway",
            "scope": "Service",
            "layer": "GENERAL"
        },
        {"__name__": "service_traffic",
            "service": "agent::frontend",
            "scope": "Service",
            "layer": "GENERAL"
        }
    ]
}

查询服务的特定指标

查询：

http://127.0.0.1:9099/api/v1/query?query=service_cpm {service='agent::songs', layer='GENERAL'}

结果：

{
  "status": "success",
  "data": {
    "resultType": "vector",
    "result": [
      {"metric": {
          "__name__": "service_cpm",
          "layer": "GENERAL",
          "scope": "Service",
          "service": "agent::songs"
        },"value": [
          1679559960,
          "6"
        ]
      }
    ]
  }
}

关于range query和不同的metrics type for query 可以参考这里的文档。

构建 Grafana Dashboard

从上面我们知道了 PromQL 服务的机制和查询方式，现在我们可以为上面的服务示例构建 Grafana Dashboard。注：以下所有配置均基于 Grafana 9.1.0 版本。

SkyWalking Showcase 提供了 General Service 和 Service Mesh 层等 Dashboard 文件，我们可以通过导入 Dashboard JSON 文件快速为层服务创建 Dashboard。

部署 Grafana 应用程序后，请按照以下步骤操作：

配置数据源

首先，我们需要创建一个数据源：在数据源配置面板中，选择 Prometheus 并设置 URL 为 OAP 服务器地址，默认端口为 9090。 SkyWalking 如果有多个 Prometheus 数据源，请在此处设置数据源名称。

导入 Dashboard 文件

创建一个名为 SkyWalking 的 Dashboard 文件夹。
将 Dashboard 文件导入到 Grafana 中，有两种获取文件的方式：
1. 来自 SkyWalking Showcase
2. 转到 SkyWaking Demo：在 Grafana 上预览指标，并将其从 General Service Dashboard 导出。
完毕！现在我们可以看到 Dashboard 正在运行，服务位于下拉列表中，指标显示在面板上。

这是一种简单的构建方式，但是如果我们想要自定义它，我们需要知道它是如何工作的。

Dashboard 的工作原理

Dashboard 设置

打开 Settings-Variables 我们可以看到如下变量：

让我们看看每个变量的作用：

$DS_SkyWalking

这是一个数据源 ty 变量，它指定了之前定义为 SkyWalking 的 Prometheus 数据源。
$layer

这是一个常量类型，因为在 ‘General Service’ Dashboard 中，所有服务都属于 ‘GENERAL’ 层，因此可以在每个查询中直接使用它们。注意，当您自定义其他层时，必须在 Layer 上面定义该值。
$service

查询类型变量，为下拉列表获取该层下的所有服务名称。

查询表达式：
```
label_values (service_traffic {layer='$layer'}, service)
```
查询表达式将查询 HTTP API /api/v1/series，以获取 $layer 中服务元数据，并根据标签（服务）提取服务名称。
$service_instance

与 $service 一样，是一个查询变量，用于在下拉列表中选择服务的所有实例。

查询表达式：
```
label_values (instance_traffic {layer='$layer', service='$service'}, service_instance)
```
这里的查询表达式不仅指定了 $layer 还包含 $service 变量，用于关联下拉列表的服务。
$endpoint

与 $service 一样，是一个查询变量，用于在下拉列表中选择服务的所有端点。

查询表达式：
```
label_values (endpoint_traffic {layer='$layer', service='$service', keyword='$endpoint_keyword', limit='$endpoint_limit'}, endpoint)
```
此处的查询表达式指定 $layer 和 $service 用于与下拉列表的服务相关联的。并且还接受 $endpoint_keyword 和 $endpoint_limit 变量作为过滤条件。
$endpoint_keyword

一个文本类型的变量，用户可以输入它来过滤 $endpoint 的返回值。
$endpoint_limit

自定义类型，用户可以选择它以限制返回端点的最大数量。

Dashboard 配置

这个 Dashboard 上有几个典型的指标面板，让我们看看它是如何配置的。

普通值指标

选择 Time series chart 面板 Service Apdex 并单击 edit。

查询表达式
```
service_apdex {service='$service', layer='$layer'} / 10000
```
指标范围为 Service，添加 service 和 layer 标签用于匹配，label 值使用上面配置的变量。该计算 Divided by 10000 用于匹配结果单位。查询文档可以参考这里。
设置 Query options --> Min interval = 1m，因为 SkyWalking 中的指标最小时间段是 1m。
设置 Connect null values --> AlwaysShow points --> Always，因为当查询间隔大于 1 小时或 1 天时，SkyWalking 返回小时 / 天步长指标值。

标签值指标

选择 Time series chart 面板 Service Response Time Percentile 并单击 edit。

查询表达式
```
service_percentile {service='$service', layer='$layer', labels='0,1,2,3,4', relabels='P50,P75,P90,P95,P99'}
```
指标范围为 Service，添加 service 和 layer 标签用于匹配，label 值使用上面配置的变量。添加 labels='0,1,2,3,4' 过滤结果标签，并添加 relabels='P50,P75,P90,P95,P99' 重命名结果标签。查询文档可以参考这里。
设置 Query options --> Min interval = 1m，因为 SkyWalking 中的指标最小时间段是 1m。
设置 Connect null values --> AlwaysShow points --> Always，因为当查询间隔 > 1 小时或 1 天时，SkyWalking 返回小时 / 天步长指标值。
设置 Legend 为 {{label}} 来展示。

排序指标

选择 Time series chart 面板 Service Response Time Percentile 并单击 edit。

查询表达式
```
service_instance_cpm {parent_service='$service', layer='$layer', top_n='10', order='DES'}
```
该表达式用于查询服务下的排序指标，因此添加标签 parent_service 和 layer 进行匹配。添加 top_n='10' 和 order='DES' 过滤结果。查询文档可以参考这里。
设置 Query options --> Min interval = 1m，因为 SkyWalking 中的指标最小时间段是 1m。
设置 Calculation --> Latest*。
设置 Legend 为 {{service_instance}} 来展示。

结论

在这篇文章中，我们介绍了 SkyWalking 中的 PromQL 服务是什么以及它的背景。详细介绍了 PromQL 服务的使用方法和 SkyWalking 相关的基本概念，展示了如何使用 PromQL 服务为 SkyWalking 构建 Grafana Dashboard。

未来，将会有更多的集成利用这个协议，比如 CI/CD、HPA（缩放）等。

Blog: eBPF enhanced HTTP observability - L7 metrics and tracing

Thu, 12 Jan 2023 00:00:00 +0000

Background

Apache SkyWalking is an open-source Application Performance Management system that helps users collect and aggregate logs, traces, metrics, and events for display on a UI. In the previous article, we introduced how to use Apache SkyWalking Rover to analyze the network performance issue in the service mesh environment. However, in business scenarios, users often rely on mature layer 7 protocols, such as HTTP, for interactions between systems. In this article, we will discuss how to use eBPF techniques to analyze performance bottlenecks of layer 7 protocols and how to enhance the tracing system using network sampling.

This article will show how to use Apache SkyWalking with eBPF to enhance metrics and traces in HTTP observability.

HTTP Protocol Analysis

HTTP is one of the most common Layer 7 protocols and is usually used to provide services to external parties and for inter-system communication. In the following sections, we will show how to identify and analyze HTTP/1.x protocols.

Protocol Identification

In HTTP/1.x, the client and server communicate through a single file descriptor (FD) on each side. Figure 1 shows the process of communication involving the following steps:

Connect/accept: The client establishes a connection with the HTTP server, or the server accepts a connection from the client.
Read/write (multiple times): The client or server reads and writes HTTPS requests and responses. A single request-response pair occurs within the same connection on each side.
Close: The client and server close the connection.

To obtain HTTP content, it’s necessary to read it from the second step of this process. As defined in the RFC, the content is contained within the data of the Layer 4 protocol and can be obtained by parsing the data. The request and response pair can be correlated because they both occur within the same connection on each side.

Figure 1: HTTP communication timeline.

HTTP Pipeline

HTTP pipelining is a feature of HTTP/1.1 that enables multiple HTTP requests to be sent over a single TCP connection without waiting for the corresponding responses. This feature is important because it ensures that the order of the responses on the server side matches the order of the requests.

Figure 2 illustrates how this works. Consider the following scenario: an HTTP client sends multiple requests to a server, and the server responds by sending the HTTP responses in the same order as the requests. This means that the first request sent by the client will receive the first response from the server, the second request will receive the second response, and so on.

When designing HTTP parsing, we should follow this principle by adding request data to a list and removing the first item when parsing a response. This ensures that the responses are processed in the correct order.

Figure 2: HTTP/1.1 pipeline.

Metrics

Based on the identification of the HTTP content and process topology diagram mentioned in the previous article, we can combine these two to generate process-to-process metrics data.

Figure 3 shows the metrics that currently support the analysis between the two processes. Based on the HTTP request and response data, we can analyze the following data:

Metrics Name	Type	Unit	Description
Request CPM(Call Per Minute)	Counter	count	The HTTP request count
Response Status CPM(Call Per Minute)	Counter	count	The count of per HTTP response status code
Request Package Size	Counter/Histogram	Byte	The request package size
Response Package Size	Counter/Histogram	Byte	The response package size
Client Duration	Counter/Histogram	Millisecond	The duration of single HTTP response on the client side
Server Duration	Counter/Histogram	Millisecond	The duration of single HTTP response on the server side

Figure 3: Process-to-process metrics.

HTTP and Trace

During the HTTP process, if we unpack the HTTP requests and responses from raw data, we can use this data to correlate with the existing tracing system.

Trace Context Identification

In order to track the flow of requests between multiple services, the trace system usually creates a trace context when a request enters a service and passes it along to other services during the request-response process. For example, when an HTTP request is sent to another server, the trace context is included in the request header.

Figure 4 displays the raw content of an HTTP request intercepted by Wireshark. The trace context information generated by the Zipkin Tracing system can be identified by the “X-B3” prefix in the header. By using eBPF to intercept the trace context in the HTTP header, we can connect the current request with the trace system.

Figure 4: View of HTTP headers in Wireshark.

Trace Event

We have added the concept of an event to traces. An event can be attached to a span and consists of start and end times, tags, and summaries, allowing us to attach any desired information to the Trace.

When performing eBPF network profiling, two events can be generated based on the request-response data. Figure 5 illustrates what happens when a service performs an HTTP request with profiling. The trace system generates trace context information and sends it in the request. When the service executes in the kernel, we can generate an event for the corresponding trace span by interacting with the request-response data and execution time in the kernel space.

Previously, we could only observe the execution status in the user space. However, by combining traces and eBPF technologies, we can now also get more information about the current trace in the kernel space, which would impact less performance for the target service if we do similar things in the tracing SDK and agent.

Figure 5: Logical view of profiling an HTTP request and response.

Sampling

To ensure efficient data storage and minimize unnecessary data sampling, we use a sampling mechanism for traces in our system. This mechanism triggers sampling only when certain conditions are met. We also provide a list of the top N traces, which allows users to quickly access the relevant request information for a specific trace.

To help users easily identify and analyze relevant events, we offer three different sampling rules:

Slow Traces: Sampling is triggered when the response time for a request exceeds a specified threshold.
Response Status [400, 500): Sampling is triggered when the response status code is greater than or equal to 400 and less than 500.
Response Status [500, 600): Sampling is triggered when the response status code is greater than or equal to 500 and less than 600.

In addition, we recognize that not all request or response raw data may be necessary for analysis. For example, users may be more interested in requesting data when trying to identify performance issues, while they may be more interested in response data when troubleshooting errors. As such, we also provide configuration options for request or response events to allow users to specify which type of data they would like to sample.

Profiling in a Service Mesh

The SkyWalking and SkyWalking Rover projects have already implemented the HTTP protocol analyze and trace associations. How do they perform when running in a service mesh environment?

Deployment

Figure 6 demonstrates the deployment of SkyWalking and SkyWalking Rover in a service mesh environment. SkyWalking Rover is deployed as a DaemonSet on each machine where a service is located and communicates with the SkyWalking backend cluster. It automatically recognizes the services on the machine and reports metadata information to the SkyWalking backend cluster. When a new network profiling task arises, SkyWalking Rover senses the task and analyzes the designated processes, collecting and aggregating network data before ultimately reporting it back to the SkyWalking backend service.

Figure 6: SkyWalking rover deployment topology in a service mesh.

Tracing Systems

Starting from version 9.3.0, the SkyWalking backend fully supports all functions in the Zipkin server. Therefore, the SkyWalking backend can collect traces from both the SkyWalking and Zipkin protocols. Similarly, SkyWalking Rover can identify and analyze trace context in both the SkyWalking and Zipkin trace systems. In the following two sections, network analysis results will be displayed in the SkyWalking and Zipkin UI respectively.

SkyWalking

When SkyWalking performs network profiling, similar to the TCP metrics in the previous article, the SkyWalking UI will first display the topology between processes. When you open the dashboard of the line representing the traffic metrics between processes, you can see the metrics of HTTP traffic from the “HTTP/1.x” tab and the sampled HTTP requests with tracing in the “HTTP Requests” tab.

As shown in Figure 7, there are three lists in the tab, each corresponding to a condition in the event sampling rules. Each list displays the traces that meet the pre-specified conditions. When you click on an item in the trace list, you can view the complete trace.

Figure 7: Sampled HTTP requests within tracing context.

When you click on an item in the trace list, you can quickly view the specified trace. In Figure 8, we can see that in the current service-related span, there is a tag with a number indicating how many HTTP events are related to that trace span.

Since we are in a service mesh environment, each service involves interacting with Envoy. Therefore, the current span includes Envoy’s request and response information. Additionally, since the current service has both incoming and outgoing requests, there are events in the corresponding span.

Figure 8: Events in the trace detail.

When the span is clicked, the details of the span will be displayed. If there are events in the current span, the relevant event information will be displayed on a time axis. As shown in Figure 9, there are a total of 6 related events in the current Span. Each event represents a data sample of an HTTP request/response. One of the events spans multiple time ranges, indicating a longer system call time. It may be due to a blocked system call, depending on the implementation details of the HTTP request in different languages. This can also help us query the possible causes of errors.

Figure 9: Events in one trace span.

Finally, we can click on a specific event to see its complete information. As shown in Figure 10, it displays the sampling information of a request, including the SkyWalking trace context protocol contained in the request header from the HTTP raw data. The raw request data allows you to quickly re-request the request to solve any issues.

Figure 10: The detail of the event.

Zipkin

Zipkin is one of the most widely used distributed tracing systems in the world. SkyWalking can function as an alternative server to provide advanced features for Zipkin users. Here, we use this way to bring the feature into the Zipkin ecosystem out-of-box. The new events would also be treated as a kind of Zipkin’s tags and annotations.

To add events to a Zipkin span, we need to do the following:

Split the start and end times of each event into two annotations with a canonical name.
Add the sampled HTTP raw data from the event to the Zipkin span tags, using the same event name for corresponding purposes.

Figures 11 and 12 show annotations and tags in the same span. In these figures, we can see that the span includes at least two events with the same event name and sequence suffix (e.g., “Start/Finished HTTP Request/Response Sampling-x” in the figure). Both events have separate timestamps to represent their relative times within the span. In the tags, the data content of the corresponding event is represented by the event name and sequence number, respectively.

Figure 11: Event timestamp in the Zipkin span annotation.

Figure 12: Event raw data in the Zipkin span tag.

Demo

In this section, we demonstrate how to perform network profiling in a service mesh and complete metrics collection and HTTP raw data sampling. To follow along, you will need a running Kubernetes environment.

Deploy SkyWalking Showcase

SkyWalking Showcase contains a complete set of example services and can be monitored using SkyWalking. For more information, please check the official documentation.

In this demo, we only deploy service, the latest released SkyWalking OAP, and UI.

export SW_OAP_IMAGE=apache/skywalking-oap-server:9.3.0
export SW_UI_IMAGE=apache/skywalking-ui:9.3.0
export SW_ROVER_IMAGE=apache/skywalking-rover:0.4.0

export FEATURE_FLAGS=mesh-with-agent,single-node,elasticsearch,rover
make deploy.kubernetes

After deployment is complete, please run the following script to open SkyWalking UI: http://localhost:8080/.

kubectl port-forward svc/ui 8080:8080 --namespace default

Start Network Profiling Task

Currently, we can select the specific instances that we wish to monitor by clicking the Data Plane item in the Service Mesh panel and the Service item in the Kubernetes panel.

In figure 13, we have selected an instance with a list of tasks in the network profiling tab.

Figure 13: Network Profiling tab in the Data Plane.

When we click the Start button, as shown in Figure 14, we need to specify the sampling rules for the profiling task. The sampling rules consist of one or more rules, each of which is distinguished by a different URI regular expression. When the HTTP request URI matches the regular expression, the rule is used. If the URI regular expression is empty, the default rule is used. Using multiple rules can help us make different sampling configurations for different requests.

Each rule has three parameters to determine if sampling is needed:

Minimal Request Duration (ms): requests with a response time exceeding the specified time will be sampled.
Sampling response status code between 400 and 499: all status codes in the range [400-499) will be sampled.
Sampling response status code between 500 and 599: all status codes in the range [500-599) will be sampled.

Once the sampling configuration is complete, we can create the task.

Figure 14: Create network profiling task page.

Done!

After a few seconds, you will see the process topology appear on the right side of the page.

When you click on the line between processes, you can view the data between the two processes, which is divided into three tabs:

TCP: displays TCP-related metrics.
HTTP/1.x: displays metrics in the HTTP 1 protocol.
HTTP Requests: displays the analyzed request and saves it to a list according to the sampling rule.

Figure 16: TCP metrics in a network profiling task.

Figure 17: HTTP/1.x metrics in a network profiling task.

Figure 18: HTTP sampled requests in a network profiling task.

Conclusion

In this article, we detailed the overview of how to analyze the Layer 7 HTTP/1.x protocol in network analysis, and how to associate it with existing trace systems. This allows us to extend the scope of data we can observe from just user space to also include kernel-space data.

In the future, we will delve further into the analysis of kernel data, such as collecting information on TCP packet size, transmission frequency, network card, and help on enhancing distributed tracing from another perspective.

Additional Resources

Zh: 使用 eBPF 提升 HTTP 可观测性 - L7 指标和追踪

Thu, 12 Jan 2023 00:00:00 +0000

背景

Apache SkyWalking 是一个开源应用性能管理系统，帮助用户收集和聚合日志、追踪、指标和事件，并在 UI 上显示。在上一篇文章中，我们介绍了如何使用 Apache SkyWalking Rover 分析服务网格环境中的网络性能问题。但是，在商业场景中，用户通常依靠成熟的第 7 层协议（如 HTTP）来进行系统之间的交互。在本文中，我们将讨论如何使用 eBPF 技术来分析第 7 层协议的性能瓶颈，以及如何使用网络采样来增强追踪系统。

本文将演示如何使用 Apache SkyWalking 与 eBPF 来增强 HTTP 可观察性中的指标和追踪。

HTTP 协议分析

HTTP 是最常用的 7 层协议之一，通常用于为外部方提供服务和进行系统间通信。在下面的章节中，我们将展示如何识别和分析 HTTP/1.x 协议。

协议识别

在 HTTP/1.x 中，客户端和服务器通过两端的单个文件描述符（File Descriptor）进行通信。图 1 显示了涉及以下步骤的通信过程：

Connect/Accept：客户端与 HTTP 服务器建立连接，或者服务器接受客户端的连接。
Read/Write（多次）：客户端或服务器读取和写入 HTTPS 请求和响应。单个请求 - 响应对在每边的同一连接内发生。
Close：客户端和服务器关闭连接。

为了获取 HTTP 内容，必须从此过程的第二步读取它。根据 RFC 定义，内容包含在 4 层协议的数据中，可以通过解析数据来获取。请求和响应对可以相关联，因为它们都在两端的同一连接内发生。

图 1：HTTP 通信时间线。

HTTP 管线化

HTTP 管线化（Pipelining）是 HTTP/1.1 的一个特性，允许在等待对应的响应的情况下在单个 TCP 连接上发送多个 HTTP 请求。这个特性很重要，因为它确保了服务器端的响应顺序必须与请求的顺序匹配。

图 2 说明了这是如何工作的，考虑以下情况：HTTP 客户端向服务器发送多个请求，服务器通过按照请求的顺序发送 HTTP 响应来响应。这意味着客户端发送的第一个请求将收到服务器的第一个响应，第二个请求将收到第二个响应，以此类推。

在设计 HTTP 解析时，我们应该遵循这个原则，将请求数据添加到列表中，并在解析响应时删除第一个项目。这可以确保响应按正确的顺序处理。

图 2： HTTP/1.1 管道。

指标

根据前文提到的 HTTP 内容和流程拓扑图的识别，我们可以将这两者结合起来生成进程间的指标数据。

图 3 显示了目前支持两个进程间分析的指标。基于 HTTP 请求和响应数据，可以分析以下数据：

指标名称	类型	单位	描述
请求 CPM（Call Per Minute）	计数器	计数	HTTP 请求计数
响应状态 CPM (Call Per Minute)	计数器	计数	每个 HTTP 响应状态码的计数
请求包大小	计数器 / 直方图	字节	请求包大小
响应包大小	计数器 / 直方图	字节	响应包大小
客户端持续时间	计数器 / 直方图	毫秒	客户端单个 HTTP 响应的持续时间
服务器持续时间	计数器 / 直方图	毫秒	服务器端单个 HTTP 响应的持续时间

图 3：进程到进程指标。

HTTP 和追踪

在 HTTP 过程中，如果我们能够从原始数据中解包 HTTP 请求和响应，就可以使用这些数据与现有的追踪系统进行关联。

追踪上下文标识

为了追踪多个服务之间的请求流，追踪系统通常在请求进入服务时创建追踪上下文，并在请求 - 响应过程中将其传递给其他服务。例如，当 HTTP 请求发送到另一个服务器时，追踪上下文包含在请求头中。

图 4 显示了 Wireshark 拦截的 HTTP 请求的原始内容。由 Zipkin Tracing 系统生成的追踪上下文信息可以通过头中的 “X-B3” 前缀进行标识。通过使用 eBPF 拦截 HTTP 头中的追踪上下文，可以将当前请求与追踪系统连接起来。

图 4：Wireshark 中的 HTTP Header 视图。

Trace 事件

我们已经将事件这个概念加入了追踪中。事件可以附加到跨度上，并包含起始和结束时间、标签和摘要，允许我们将任何所需的信息附加到追踪中。

在执行 eBPF 网络分析时，可以根据请求 - 响应数据生成两个事件。图 5 说明了在带分析的情况下执行 HTTP 请求时发生的情况。追踪系统生成追踪上下文信息并将其发送到请求中。当服务在内核中执行时，我们可以通过与内核空间中的请求 - 响应数据和执行时间交互，为相应的追踪跨度生成事件。

以前，我们只能观察用户空间的执行状态。现在，通过结合追踪和 eBPF 技术，我们还可以在内核空间获取更多关于当前追踪的信息，如果我们在追踪 SDK 和代理中执行类似的操作，将对目标服务的性能产生较小的影响。

图 5：分析 HTTP 请求和响应的逻辑视图。

抽样

该机制仅在满足特定条件时触发抽样。我们还提供了前 N 条追踪的列表，允许用户快速访问特定追踪的相关请求信息。为了帮助用户轻松识别和分析相关事件，我们提供了三种不同的抽样规则：

慢速追踪：当请求的响应时间超过指定阈值时触发抽样。
响应状态 [400,500)：当响应状态代码大于或等于 400 且小于 500 时触发抽样。
响应状态 [500,600)：当响应状态代码大于或等于 500 且小于 600 时触发抽样。

此外，我们认识到分析时可能并不需要所有请求或响应的原始数据。例如，当试图识别性能问题时，用户可能更感兴趣于请求数据，而在解决错误时，他们可能更感兴趣于响应数据。因此，我们还提供了请求或响应事件的配置选项，允许用户指定要抽样的数据类型。

服务网格中的分析

SkyWalking Rover 项目已经实现了 HTTP 协议的分析和追踪关联。当在服务网格环境中运行时它们的表现如何？

部署

图 6 演示了 SkyWalking 和 SkyWalking Rover 在服务网格环境中的部署方式。SkyWalking Rover 作为一个 DaemonSet 部署在每台服务所在的机器上，并与 SkyWalking 后端集群通信。它会自动识别机器上的服务并向 SkyWalking 后端集群报告元数据信息。当出现新的网络分析任务时，SkyWalking Rover 会感知该任务并对指定的进程进行分析，在最终将数据报告回 SkyWalking 后端服务之前，收集和聚合网络数据。

图 6：服务网格中的 SkyWalking rover 部署拓扑。

追踪系统

从版本 9.3.0 开始，SkyWalking 后端完全支持 Zipkin 服务器中的所有功能。因此，SkyWalking 后端可以收集来自 SkyWalking 和 Zipkin 协议的追踪。同样，SkyWalking Rover 可以在 SkyWalking 和 Zipkin 追踪系统中识别和分析追踪上下文。在接下来的两节中，网络分析结果将分别在 SkyWalking 和 Zipkin UI 中显示。

SkyWalking

当 SkyWalking 执行网络分析时，与前文中的 TCP 指标类似，SkyWalking UI 会首先显示进程间的拓扑图。当打开代表进程间流量指标的线的仪表板时，您可以在 “HTTP/1.x” 选项卡中看到 HTTP 流量的指标，并在 “HTTP Requests” 选项卡中看到带追踪的抽样的 HTTP 请求。

如图 7 所示，选项卡中有三个列表，每个列表对应事件抽样规则中的一个条件。每个列表显示符合预先规定条件的追踪。当您单击追踪列表中的一个项目时，就可以查看完整的追踪。

图 7：Tracing 上下文中的采样 HTTP 请求。

当您单击追踪列表中的一个项目时，就可以快速查看指定的追踪。在图 8 中，我们可以看到在当前的服务相关的跨度中，有一个带有数字的标签，表示与该追踪跨度相关的 HTTP 事件数。

由于我们在服务网格环境中，每个服务都涉及与 Envoy 交互。因此，当前的跨度包括 Envoy 的请求和响应信息。此外，由于当前的服务有传入和传出的请求，因此相应的跨度中有事件。

图 8：Tracing 详细信息中的事件。

当单击跨度时，将显示跨度的详细信息。如果当前跨度中有事件，则相关事件信息将在时间轴上显示。如图 9 所示，当前跨度中一共有 6 个相关事件。每个事件代表一个 HTTP 请求 / 响应的数据样本。其中一个事件跨越多个时间范围，表示较长的系统调用时间。这可能是由于系统调用被阻塞，具体取决于不同语言中的 HTTP 请求的实现细节。这也可以帮助我们查询错误的可能原因。

图 9：一个 Tracing 范围内的事件。

最后，我们可以单击特定的事件查看它的完整信息。如图 10 所示，它显示了一个请求的抽样信息，包括从 HTTP 原始数据中的请求头中包含的 SkyWalking 追踪上下文协议。原始请求数据允许您快速重新请求以解决任何问题。

图 10：事件的详细信息。

Zipkin

Zipkin 是世界上广泛使用的分布式追踪系统。SkyWalking 可以作为替代服务器，提供高级功能。在这里，我们使用这种方式将功能无缝集成到 Zipkin 生态系统中。新事件也将被视为 Zipkin 的标签和注释的一种。

为 Zipkin 跨度添加事件，需要执行以下操作：

将每个事件的开始时间和结束时间分别拆分为两个具有规范名称的注释。
将抽样的 HTTP 原始数据从事件添加到 Zipkin 跨度标签中，使用相同的事件名称用于相应的目的。

图 11 和图 12 显示了同一跨度中的注释和标签。在这些图中，我们可以看到跨度包含至少两个具有相同事件名称和序列后缀的事件（例如，图中的 “Start/Finished HTTP Request/Response Sampling-x”）。这两个事件均具有单独的时间戳，用于表示其在跨度内的相对时间。在标签中，对应事件的数据内容分别由事件名称和序列号表示。

图 11：Zipkin span 注释中的事件时间戳。

图 12：Zipkin span 标签中的事件原始数据。

演示

在本节中，我们将演示如何在服务网格中执行网络分析，并完成指标收集和 HTTP 原始数据抽样。要进行操作，您需要一个运行中的 Kubernetes 环境。

部署 SkyWalking Showcase

SkyWalking Showcase 包含一套完整的示例服务，可以使用 SkyWalking 进行监控。有关详细信息，请参阅官方文档。

在本演示中，我们只部署了服务、最新发布的 SkyWalking OAP 和 UI。

export SW_OAP_IMAGE=apache/skywalking-oap-server:9.3.0
export SW_UI_IMAGE=apache/skywalking-ui:9.3.0
export SW_ROVER_IMAGE=apache/skywalking-rover:0.4.0

export FEATURE_FLAGS=mesh-with-agent,single-node,elasticsearch,rover
make deploy.kubernetes

部署完成后，运行下面的脚本启动 SkyWalking UI：http://localhost:8080/。

kubectl port-forward svc/ui 8080:8080 --namespace default

启动网络分析任务

目前，我们可以通过单击服务网格面板中的 Data Plane 项和 Kubernetes 面板中的 Service 项来选择要监视的特定实例。

在图 13 中，我们已在网络分析选项卡中选择了一个具有任务列表的实例。

图 13：数据平面中的网络分析选项卡。

当我们单击 “开始” 按钮时，如图 14 所示，我们需要为分析任务指定抽样规则。抽样规则由一个或多个规则组成，每个规则都由不同的 URI 正则表达式区分。当 HTTP 请求的 URI 与正则表达式匹配时，将使用该规则。如果 URI 正则表达式为空，则使用默认规则。使用多个规则可以帮助我们为不同的请求配置不同的抽样配置。

每个规则都有三个参数来确定是否需要抽样：

最小请求持续时间（毫秒）：响应时间超过指定时间的请求将被抽样。
在 400 和 499 之间的抽样响应状态代码：范围 [400-499) 中的所有状态代码将被抽样。
在 500 和 599 之间的抽样响应状态代码：范围 [500-599) 中的所有状态码将被抽样。

抽样配置完成后，我们就可以创建任务了。

图 14：创建网络分析任务页面。

完成

几秒钟后，你会看到页面的右侧出现进程拓扑结构。

图 15：网络分析任务中的流程拓扑。

当您单击进程之间的线时，您可以查看两个过程之间的数据，它被分为三个选项卡：

TCP：显示与 TCP 相关的指标。
HTTP/1.x：显示 HTTP 1 协议中的指标。
HTTP 请求：显示已分析的请求，并根据抽样规则保存到列表中。

图 16：网络分析任务中的 TCP 指标。

图 17：网络分析任务中的 HTTP/1.x 指标。

图 18：网络分析任务中的 HTTP 采样请求。

总结

在本文中，我们详细介绍了如何在网络分析中分析 7 层 HTTP/1.x 协议，以及如何将其与现有追踪系统相关联。这使我们能够将我们能够观察到的数据从用户空间扩展到内核空间数据。

在未来，我们将进一步探究内核数据的分析，例如收集 TCP 包大小、传输频率、网卡等信息，并从另一个角度提升分布式追踪。

其他资源

Blog: Boost Root Cause Analysis Quickly With SkyWalking’s New Trace-Metrics Association Feature

Mon, 19 Dec 2022 00:00:00 +0000

Observability for modern distributed applications work is critical for understanding how they behave under a variety of conditions and for troubleshooting and resolving issues when they arise. Traces, metrics, and logs are regarded as fundamental parts of the observability stack. Traces are the footprints of distributed system executions, meanwhile, metrics measure system performance with numbers in the timeline. Essentially, they measure the performance from two dimensions. Being able to quickly visualize the connection between traces and corresponding metrics makes it possible to quickly diagnose which process flows are correlated to potentially pathological behavior. This powerful new capability is now available in SkyWalking 9.3.0.

The SkyWalking project started only with tracing, with a focus on 100% sampling-based metrics and topology analysis since 2018. When users face anomaly trends of time-series metrics, like a peak on the line chart, or histogram shows a larger gap between p95 and p95, the immediate question is, why is this happening? One of SkyWalking’s latest features, the trace-metric association, makes it much easier to answer that question and to address the root cause.

How Are Metrics Generated?

SkyWalking provides three ways to calculate metrics:

Metrics built from trace spans, depending on the span’s layer, kind, and tags.
Metrics extracted from logs—a kind of keyword and tags-based metrics extraction.
Metrics reported from mature and mainstream metrics/meter systems, such as OpenTelemetry, Prometheus, and Zabbix.

Tracing tracks the processes of requests between an application’s services. Most systems that generate traffic and performance-related metrics also generate tracing data, either from server-side trace-based aggregations or through client SDKs.

Use SkyWalking to Reduce the Traditional Cost of Trace Indexing

Tracing data and visualization are critical troubleshooting tools for both developers and operators alike because of how helpful they are in locating issue boundaries. But, because it has traditionally been difficult to find associations between metrics and traces, teams have added increasingly more tags into the spans, and search through various combinations. This trend of increased instrumentation and searching has required increased infrastructure investment to support this kind of search. SkyWalking’s metrics and tracing association capabilities can help reduce the cost of indexing and searching that data.

Find the Associated Trace

When looking for association between metrics and traces, the kind of metrics we’re dealing with determines their relationships to traces. Let’s review the standard request rate, error, and duration (RED) metrics to see how it works.

Success Rate Metrics

The success rate is determined by the return code, RPC response code, or exceptions of the process. When the success rate decreases, looking for errors in the traces of this service or pod are the first place to look to find clues.

Figure 1: The success rate graph from SkyWalking’s 9.3.0 dashboard with the option to view related traces at a particular time.

Drilling down from the peak of the success rate, SkyWalking lists all traces and their error status that were collected in this particular minute (Figure 2):

Figure 2: SkyWalking shows related traces with an error status.

Requests to /test can be located from the trace, and the span’s tag indicates a 404 response code of the HTTP request.

Figure 3: A detail view of a request to http://frontend/test showing that the URI doesn’t exist.

By looking at the trace data, it becomes immediately clear that the drop in success rate is caused by requests to a nonexistent URI.

Average Response Time

The average response time metric provides a general overview of service performance. When average response time is unstable, this usually means that the system is facing serious performance impacts.

Figure 4: SkyWalking’s query UI for searching for related traces showing traces for requests that exceed a particular duration threshold.

When you drill down from this metric, this query condition (Figure 4) will reveal the slowest traces of the service in this specific minute. Notice, at least 168ms is added as a condition automatically, to avoid scanning a large number of rows in the Database.

Apdex

Apdex—the Application Performance Index—is a measure of response time based against a set threshold. It measures the ratio of satisfactory response times to unsatisfactory response times (Figure 5). The response time is measured from an asset request to completed delivery back to the requestor.

Figure 5: The Apdex formula

A user defines a response time tolerating threshold T. All responses handled in T or less time satisfy the user.

For example, if T is 1.2 seconds and a response completes in 0.5 seconds, then the user is satisfied. All responses greater than 1.2 seconds dissatisfy the user. Responses greater than 4.8 seconds frustrate the user.

When the Apdex score decreases, we need to find related traces from two perspectives: slow traces and error status traces. SkyWalking’s new related tracing features offers a quick way to view both (Figure 6) directly from the Apdex graph.

Figure 6: Show slow trace and error status traces from the Apdex graph

Service Response Time

Percentile MetricThe percentile graph (Figure 7) provides p50, p75, p90, p95, and p99 latency ranks to measure the long-tail issues of service performance.

Figure 7: The service response time percentile graph helps to highlight long-tail issues of service performance.

This percentile graph shows a typical long-tail issue. P99 latency is four times slower than the P95. When we use the association, we see the traces with latency between P95 - P99 and P99 - Infinity.

The traces of requests causing this kind of long-tail phenomena are automatically listing from there.

Figure 8: Query parameters to search for traces based on latency.

Are More Associations Available?

SkyWalking provides more than just associations between between traces and metrics to help you find possible causal relationships and to avoid looking for the proverbial needle in a haystack.

Currently, SkyWalking 9.3.0 offers two more associations: metric-to-metric associations and event-to-metric associations.

Metric-to-metric Associations

There are dozens of metrics on the dashboard—which is great for getting a complete picture of application behavior. During a typical performance issue, the peaks of multiple metrics are affected simultaneously. But, trying to correlate peaks across all of these graphs can be difficult…

Now in SkyWalking 9.3.0, when you click the peak of one graph, the pop-out box lets you see associated metrics.

Figure 9: SkyWalking’s option to view associated metrics.

When you choose that option, all associated metrics graphs will show axis pointers (the dotted vertical lines) in all associated graphs like in Figure 10. This makes it easier to correlate the peaks in different graphs with each other. Often, these correlated peaks with have the same root cause.

Figure 10: Axis pointers (vertical dotted lines) show associations between peaks across multiple metrics graphs.

Event-to-Metric Associations

SkyWalking provides the event concept to associate possible service performance impacted by the infrastructure, such as new deployment even from k8s. Or, the anomaly had been detected by alerting or integrated AIOps engine.

The event to metrics association is also automatically, it could cover the time range of the event on the metric graphs(blue areas). If the area of event and peaks are matched, most likely this event covered this anomaly.

Figure 11: SkyWalking’s event to metric association view.

SkyWalking Makes it Easier and Faster to Find Root Causes

SkyWalking now makes it easy to find associations between metrics, events, and traces, ultimately making it possible to identify root causes and fix problems fast. The associations we’ve discussed in this article are available out-of-box in the SkyWalking 9.3.0 release.

Figure 12: Just click on the dots to see related traces and metrics associations.

Click the dots on any metric graph, and you will see a View Related Traces item pop-out if this metric has logical mapping traces.

Conclusion

In this blog, we took a look at the newly-added association feature between metrics and traces. With this new visualization, it’s now much easier to find key traces to identify root cause of issues.Associations in SkyWalking can go even deeper. Associations from metrics to traces is not the end of diagnosing system bottleneck. In the next post, we will introduce an eBPF powered trace enhancement where you’ll be able to see HTTP request and response details associated with tracing spans from network profiling. Stay tuned.

Zh: SkyWalking 推出 trace-metric 关联功能助力快速根源问题排查

Mon, 19 Dec 2022 00:00:00 +0000

现代分布式应用程序工作的可观测性对于了解它们在各种条件下的行为方式以及在出现问题时进行故障排除和解决至关重要。追踪、指标和日志被视为可观测性堆栈的基本部分。Trace 是分布式系统执行的足迹，而 metric 则是用时间轴上的数字衡量系统性能。本质上，它们从两个维度衡量性能。能够快速可视化追踪和相应指标之间的联系，可以快速诊断哪些流程与潜在的异常相关。SkyWalking 9.3.0 现在提供了这一强大的新功能。

SkyWalking 项目从 tracing 开始，从 2018 年开始专注于 100% 基于采样的指标和拓扑分析。当用户面对时间序列指标的异常趋势时，比如折线图上的峰值，或者直方图显示 p95 和 p95 之间的差距较大，直接的问题是，为什么会出现这种情况？SkyWalking 的最新功能之一，trace 与 metric 关联，使得回答这个问题和解决根本原因更加容易。

指标是如何生成的？

SkyWalking 提供了三种计算指标的方式：

根据追踪跨度构建的指标，具体取决于跨度的层、种类和标签。
从日志中提取指标—— 一种基于关键词和标签的指标提取。
从成熟和主流的指标 / 仪表系统报告的指标，例如 OpenTelemetry、Prometheus 和 Zabbix。

Tracing 追踪应用程序服务之间的请求过程。大多数生成流量和性能相关指标的系统也会生成追踪数据，这些数据来自服务器端基于追踪的聚合或通过客户端 SDK。

使用 SkyWalking 降低追踪索引的传统成本

Trace 数据和可视化对于开发人员和运维人员来说都是至关重要的故障排除工具，因为它们在定位问题边界方面非常有帮助。但是，由于传统上很难找到指标和痕迹之间的关联，团队已经将越来越多的标签添加到跨度中，并搜索各种组合。这种增加仪器和搜索的趋势需要增加基础设施投资来支持这种搜索。SkyWalking 的指标和追踪关联功能有助于降低索引和搜索该数据的成本。

查找关联的 trace

在寻找 metric 和 trace 之间的关联时，我们处理的指标类型决定了它们与 trace 的关系。让我们回顾一下标准请求*率、错误和持续时间（RED）*指标，看看它是如何工作的。

成功率指标

成功率由返回码、RPC 响应码或进程异常决定。当成功率下降时，在这个服务或 Pod 的 trace 中寻找错误是第一个寻找线索的地方。

图 1：SkyWalking 9.3.0 仪表板的成功率图表，带有在特定时间查看相关 trace 的选项。

从成功率的峰值向下探索，SkyWalking 列出了在这一特定分钟内收集的所有 trace 及其错误状态（图 2）：

图 2：SkyWalking 显示具有错误状态的相关追踪。

可以从 trace 中找到对 /test 的请求，并且 span 的标记指示 HTTP 请求的 404 响应代码。

图 3：显示 URI 不存在的 http://frontend/test 请求的详细视图。

通过查看 trace 数据，很明显成功率的下降是由对不存在的 URI 的请求引起的。

平均响应时间

平均响应时间指标提供了服务性能的一般概览。当平均响应时间不稳定时，这通常意味着系统面临严重的性能影响。

图 4：SkyWalking 用于搜索相关 trace 的查询 UI，显示超过特定持续时间阈值的请求的 trace。

当您从该指标向下探索时，该查询条件（图 4）将揭示该特定分钟内服务的最慢 trace。请注意，至少 168ms 作为条件自动添加，以避免扫描数据库中的大量行。

Apdex

Apdex（应用程序性能指数）是根据设定的阈值衡量响应时间的指标。它测量令人满意的响应时间与不令人满意的响应时间的比率（图 5）。响应时间是从资产请求到完成交付回请求者的时间。

图 5：Apdex 公式

用户定义响应时间容忍阈值 T。在 T 或更短时间内处理的所有响应都使用户满意。

例如，如果 T 为 1.2 秒，响应在 0.5 秒内完成，则用户会感到满意。所有大于 1.2 秒的响应都会让用户不满意。超过 4.8 秒的响应会让用户感到沮丧。

当 Apdex 分数下降时，我们需要从两个角度寻找相关的 trace：慢速和错误状态的 trace。SkyWalking 的新相关追踪功能提供了一种直接从 Apdex 图表查看两者（图 6）的快速方法。

图 6：显示 Apdex 图中的慢速 trace 和错误状态 trace

服务响应时间

百分位指标百分位图（图 7）提供 p50、p75、p90、p95 和 p99 延迟排名，以衡量服务性能的长尾问题。

图 7：服务响应时间百分位图有助于突出服务性能的长尾问题。

这个百分位数图显示了一个典型的长尾问题。P99 延迟比 P95 慢四倍。当我们使用关联时，我们会看到 P95 - P99 和 P99 - Infinity 之间具有延迟的 trace。

造成这种长尾现象的请求 trace，就是从那里自动列出来的。

图 8：用于根据延迟搜索 trace 的查询参数。

是否有更多关联可用？

SkyWalking 提供的不仅仅是 trace 和 metric 之间的关联，还可以帮助您找到可能的因果关系，避免大海捞针。

目前，SkyWalking 9.3.0 提供了两种关联：metric-to-metric 关联和 event-to-metric 关联。

Metric-to-metric 关联

仪表板上有许多指标 —— 这对于全面了解应用程序行为非常有用。在典型的性能问题中，多个指标的峰值会同时受到影响。但是，尝试关联所有这些图表中的峰值可能很困难……

现在在 SkyWalking 9.3.0 中，当你点击一个图表的峰值时，弹出框可以让你看到相关的指标。

图 9：SkyWalking 用于查看相关指标的选项。

当您选择该选项时，所有关联的指标图表将在所有关联的图表中显示轴指针（垂直虚线），如图 10 所示。这使得将不同图表中的峰值相互关联起来变得更加容易。通常，这些相关的峰值具有相同的根本原因。

图 10：轴指针（垂直虚线）显示多个指标图中峰值之间的关联。

Event-to-metric 关联

SkyWalking 提供了事件概念来关联可能受基础设施影响的服务性能，例如来自 Kubernetes 的新部署。或者，已通过警报或集成 AIOps 引擎检测到异常。

事件到指标的关联也是自动的，它可以覆盖指标图上事件的时间范围（蓝色区域）。如果事件区域和峰值匹配，则很可能该事件覆盖了该异常。

图 11：SkyWalking 的事件与指标关联视图。

SkyWalking 使查找根本原因变得更加容易和快速

SkyWalking 现在可以轻松找到指标、事件和追踪之间的关联，最终可以确定根本原因并快速解决问题。我们在本文中讨论的关联在 SkyWalking 9.3.0 版本中开箱即用。

图 12：只需单击圆点即可查看相关 trace 和 metric 关联。

单击任何指标图上的点，如果该指标具有逻辑映射，您将看到一个查看相关 trace 弹出窗口。

结论

在这篇博客中，我们了解了 metric 和 trace 之间新增的关联功能。有了这个新的可视化，现在可以更容易地找到关键 trace 来识别问题的根本原因。SkyWalking 中的关联可以更深入。从 metric 到 trace 的关联并不是诊断系统瓶颈的终点。在下一篇文章中，我们将介绍 eBPF 支持的追踪增强功能，您将看到与网络分析中的追踪跨度相关的 HTTP 请求和响应详细信息。敬请关注。

Blog: Apache ShenYu(incubating) plugin implementation principles and observability practices

Sun, 08 May 2022 00:00:00 +0000

Content

1. Introduction of SkyWalking and ShenYu

1.1 SkyWalking

SkyWalking is an Application Performance Monitoring (APM) and Observability Analysis Platform (OAP) for microservices, distributed systems, and cloud natives, Has powerful features that provide a multi-dimensional means of application performance analysis, including distributed topology diagrams, application performance metrics, distributed link tracing, log correlation analysis and alerts. Also has a very rich ecology. Widely used in various companies and open source projects.

1.2 Apache ShenYu (incubating)

Apache ShenYu (incubating) High-performance,multi-protocol,extensible,responsive API Gateway. Compatible with a variety of mainstream framework systems, support hot plug, users can customize the development, meet the current situation and future needs of users in a variety of scenarios, experienced the temper of large-scale scenes. Rich protocol support: Http, Spring Cloud, gRPC, Dubbo, SOFARPC, Motan, Tars, etc.

2. Apache ShenYu plugin implementation principle

ShenYu’s asynchrony is a little different from previous exposure to asynchrony, it is a full-link asynchrony, the execution of each plug-in is asynchronous, and thread switching is not a single fixed situation (and the individual plug-in implementation is related). The gateway initiates service calls of various protocol types, and the existing SkyWalking plugins create ExitSpan (synchronous or asynchronous) when they initiate service calls. The gateway receives the request and creates an asynchronous EntrySpan. The asynchronous EntrySpan needs to be concatenated with the synchronous or asynchronous ExitSpan, otherwise the link will be broken.

There are 2 types of tandem solutions：

Snapshot Delivery:
Pass the snapshot after creating the EntrySpan to the thread that created the ExitSpan in some way.
Currently this approach is used in the asynchronous WebClient plugin, which can receive asynchronous snapshots. shenYu proxy Http service or SpringCloud service is to achieve span concatenation through snapshot passing.
LocalSpan transit:
Other RPC class plugins do not receive snapshots for concatenation like Asynchronous WebClient. Although you can modify other RPC plugins to receive snapshots for concatenation, it is not recommended or necessary to do so. This can be achieved by creating a LocalSpan in the thread where the ExitSpan is created, and then connecting the asynchronous EntrySpan and LocalSpan by snapshot passing. This can be done without changing the original plugin code.

The span connection is shown below:

You may ask if it is possible to create LocalSpan inside a generic plugin, instead of creating one separately for ShenYu RPC plugin? The answer is no, because you need to ensure that LocalSpan and ExitSpan are in the same thread, and ShenYu is fully linked asynchronously. The code to create LocalSpan is reused in the implementation.

3. Adding generalized call tracking to the gRPC plugin and keeping it compatible

The existing SkyWalking gRPC plugin only supports calls initiated by way of stubs. For the gateway there is no proto file, the gateway takes generalized calls (not through stubs), so tracing RPC requests, you will find that the link will break at the gateway node. In this case, it is necessary to make the gRPC plugin support generalized calls, while at the same time needing to remain compatible and not affect the original tracing method. This is achieved by determining whether the request parameter is a DynamicMessage, and if it is not, then the original tracing logic through the stub is used. If not, then the original tracing logic via stubs is used, and if not, then the generalized call tracing logic is used. The other compatibility is the difference between the old and new versions of gRPC, as well as the compatibility of various cases of obtaining server-side IP, for those interested in the source code.

4. ShenYu Gateway Observability Practice

The above explains the principle of SkyWalking ShenYu plug-in implementation, the following deployment application to see the effect. SkyWalking powerful, in addition to the link tracking requires the development of plug-ins, other powerful features out of the box. Here only describe the link tracking and application performance analysis part, if you want to experience the power of SkyWalking features, please refer to the SkyWalking official documentation.
Version description:

skywalking-java: 8.11.0-SNAPSHOT source code build. Note: The shenyu plugin will be released in version 8.11.0, and will probably release it initially in May or June. the Java agent is in the regular release phase.
skywalking: 9.0.0 V9 version

Usage instructions:
SkyWalking is designed to be very easy to use. Please refer to the official documentation for configuring and activating the shenyu plugin.

4.1 Sending requests to the gateway

Initiate various service requests to the gateway via the postman client or other means.

4.2 Request Topology Diagram

4.3 Request Trace (in the case of gRPC)

Normal Trace：

Abnormal Trace：

Click on the link node to see the corresponding node information and exception information

Service Provider Span

Gateway request span

4.4 Service Metrics Monitoring

4.5 Gateway background metrics monitoring

Database Monitoring:

Thread pool and connection pool monitoring:

4.6 JVM Monitoring

4.7 Endpoint Analysis

4.8 Exception log and exception link analysis

See official documentation for log configuration

Log monitoring

Distributed link trace details corresponding to exception logs

5. Summary

SkyWalking has very comprehensive support for metrics, link tracing, and logging in observability, and is powerful, easy to use, and designed for large distributed systems, microservices, cloud-native, container architectures, and has a rich ecosystem. Using SkyWalking to provide powerful observability support for Apache ShenYu (incubating) gives ShenYu a boost. Finally, if you are interested in high-performance responsive gateways, you can follow Apache ShenYu (incubating). Also, thanks to SkyWalking such an excellent open source software to the industry contributions.

Blog: SkyWalking 8.4 provides infrastructure monitoring

Mon, 08 Feb 2021 00:00:00 +0000

Origin: Tetrate.io blog

Background

Apache SkyWalking– the APM tool for distributed systems– has historically focused on providing observability around tracing and metrics, but service performance is often affected by the host. The newest release, SkyWalking 8.4.0, introduces a new feature for monitoring virtual machines. Users can easily detect possible problems from the dashboard– for example, when CPU usage is overloaded, when there’s not enough memory or disk space, or when the network status is unhealthy, etc.

How it works

SkyWalking leverages Prometheus and OpenTelemetry for collecting metrics data as we did for Istio control panel metrics; Prometheus is mature and widely used, and we expect to see increased adoption of the new CNCF project, OpenTelemetry. The SkyWalking OAP Server receives these metrics data of OpenCensus format from OpenTelemetry. The process is as follows:

Prometheus Node Exporter collects metrics data from the VMs.
OpenTelemetry Collector fetches metrics from Node Exporters via Prometheus Receiver, and pushes metrics to SkyWalking OAP Server via the OpenCensus GRPC Exporter.
The SkyWalking OAP Server parses the expression with MAL to filter/calculate/aggregate and store the results. The expression rules are in /config/otel-oc-rules/vm.yaml.
We can now see the data on the SkyWalking WebUI dashboard.

What to monitor

SkyWalking provides default monitoring metrics including:

CPU Usage (%)
Memory RAM Usage (MB)
Memory Swap Usage (MB)
CPU Average Used
CPU Load
Memory RAM (total/available/used MB)
Memory Swap (total/free MB)
File System Mount point Usage (%)
Disk R/W (KB/s)
Network Bandwidth Usage (receive/transmit KB/s)
Network Status (tcp_curr_estab/tcp_tw/tcp_alloc/sockets_used/udp_inuse)
File fd Allocated

The following is how it looks when we monitor Linux:

How to use

To enable this feature, we need to install Prometheus Node Exporter and OpenTelemetry Collector and activate the VM monitoring rules in SkyWalking OAP Server.

Install Prometheus Node Exporter

wget https://github.com/prometheus/node_exporter/releases/download/v1.0.1/node_exporter-1.0.1.linux-amd64.tar.gz
tar xvfz node_exporter-1.0.1.linux-amd64.tar.gz
cd node_exporter-1.0.1.linux-amd64
./node_exporter

In linux Node Exporter exposes metrics on port 9100 by default. When it is running, we can get the metrics from the /metrics endpoint. Use a web browser or command curl to verify.

curl http://localhost:9100/metrics

We should see all the metrics from the output like:

# HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 7.7777e-05
go_gc_duration_seconds{quantile="0.25"} 0.000113756
go_gc_duration_seconds{quantile="0.5"} 0.000127199
go_gc_duration_seconds{quantile="0.75"} 0.000147778
go_gc_duration_seconds{quantile="1"} 0.000371894
go_gc_duration_seconds_sum 0.292994058
go_gc_duration_seconds_count 2029
...

Note: We only need to install Node Exporter, rather than Prometheus server. If you want to get more information about Prometheus Node Exporter see: https://prometheus.io/docs/guides/node-exporter/

Install OpenTelemetry Collector

We can quickly install a OpenTelemetry Collector instance by using docker-compose with the following steps:

Create a directory to store the configuration files, like /usr/local/otel.
Create docker-compose.yaml and otel-collector-config.yaml in this directory represented below:

docker-compose.yaml

version: "2"
services:
  # Collector
  otel-collector:
    # Specify the image to start the container from
    image: otel/opentelemetry-collector:0.19.0
    # Set the  otel-collector configfile 
    command: ["--config=/etc/otel-collector-config.yaml"]
    # Mapping the configfile to host directory
    volumes:
      - ./otel-collector-config.yaml:/etc/otel-collector-config.yaml
    ports:
      - "13133:13133" # health_check extension
      - "55678"       # OpenCensus receiver

otel-collector-config.yaml

extensions:
  health_check:
# A receiver is how data gets into the OpenTelemetry Collector
receivers:
  # Set Prometheus Receiver to collects metrics from targets
  # It’s supports the full set of Prometheus configuration
  prometheus:
    config:
      scrape_configs:
        - job_name: 'otel-collector'
          scrape_interval: 10s
          static_configs: 
              # Replace the IP to your VMs‘s IP which has installed Node Exporter
            - targets: [ 'vm1:9100' ] 
            - targets: [ 'vm2:9100' ]
            - targets: [ ‘vm3:9100' ]

processors:
  batch:

# An exporter is how data gets sent to different systems/back-ends
exporters:
  # Exports metrics via gRPC using OpenCensus format
  opencensus:
    endpoint: "docker.for.mac.host.internal:11800" # The OAP Server address
    insecure: true
  logging:
    logLevel: debug

service:
  pipelines:
    metrics:
      receivers: [prometheus]
      processors: [batch]
      exporters: [logging, opencensus]

  extensions: [health_check]

In this directory use command docker-compose to start up the container:

docker-compose up -d

After the container is up and running, you should see metrics already exported in the logs:

...
Metric #165
Descriptor:
-> Name: node_network_receive_compressed_total
-> Description: Network device statistic receive_compressed.
-> Unit:
-> DataType: DoubleSum
-> IsMonotonic: true
-> AggregationTemporality: AGGREGATION_TEMPORALITY_CUMULATIVE
DoubleDataPoints #0
Data point labels:
-> device: ens4
StartTime: 1612234754364000000
Timestamp: 1612235563448000000
Value: 0.000000
DoubleDataPoints #1
Data point labels:
-> device: lo
StartTime: 1612234754364000000
Timestamp: 1612235563448000000
Value: 0.000000
...

If you want to get more information about OpenTelemetry Collector see: https://opentelemetry.io/docs/collector/

Set up SkyWalking OAP Server

To activate the oc handler and vm relevant rules, set your environment variables:

SW_OTEL_RECEIVER=default
SW_OTEL_RECEIVER_ENABLED_OC_RULES=vm

Note: If there are other rules already activated , you can add vm with use , as a separator.

SW_OTEL_RECEIVER_ENABLED_OC_RULES=vm,oap

Start the SkyWalking OAP Server.

Done!

After all of the above steps are completed, check out the SkyWalking WebUI. Dashboard VM provides the default metrics of all observed virtual machines. Note: Clear the browser local cache if you used it to access deployments of previous SkyWalking versions.

Additional Resources

Read more about the SkyWalking 8.4 release highlights.
Get more SkyWalking updates on Twitter.

Apache SkyWalking – Metrics

Blog: Monitoring WeChat and Alipay Mini Programs with SkyWalking

Data path

Two platforms, two independent Layers and dashboards

Asymmetric metric semantics

What the SDK does

What OAP needs

Compatibility

What’s next

Blog: Query SkyWalking and Zipkin Traces with TraceQL and Visualize in Grafana

Query SkyWalking and Zipkin Traces with TraceQL and Visualize in Grafana

Architecture Overview

Limitations and Supported TraceQL Features

Trace Format Conversion

SkyWalking Native Trace

Trace ID Encoding

Span Kind Mapping

Status Mapping

SpanAttachedEvents

Zipkin Trace

Span Kind Mapping

Status Mapping

Endpoint and Annotation Mapping

How to Enable TraceQL

Step 1: Enable the TraceQL Module

Step 2: Enable the Zipkin Receiver (for Zipkin traces only)

Full Configuration Reference

Configuring Grafana Tempo Data Source

Context Paths

Setting Up the SkyWalking Data Source

Setting Up the Zipkin Data Source

Configuring Trace List Result Tags

Building a Trace Dashboard in Grafana

SkyWalking Native Trace Dashboard

Step 1: Explore and Save

Step 2: Configure Variables

Step 3: Add a Trace Panel

Step 4: View Trace Details

Zipkin Trace Dashboard

Summary

Zh: 使用 TraceQL 查询 SkyWalking 和 Zipkin 链路追踪数据并在 Grafana 中可视化

使用 TraceQL 查询 SkyWalking 和 Zipkin 链路追踪数据并在 Grafana 中可视化

架构概览

支持的 TraceQL 特性与限制

Trace 格式转换

SkyWalking 原生链路追踪

Trace ID 编码

Span Kind 映射

状态映射

SpanAttachedEvents

Zipkin 链路追踪

Span Kind 映射

状态映射

Endpoint 与 Annotation 映射

如何启用 TraceQL

步骤 1：启用 TraceQL 模块

步骤 2：启用 Zipkin 接收器（仅用于 Zipkin 链路追踪）

完整配置参考

配置 Grafana Tempo 数据源

上下文路径

配置 SkyWalking 数据源

配置 Zipkin 数据源

配置链路追踪列表结果标签

在 Grafana 中构建链路追踪仪表板

SkyWalking 原生追踪仪表板

步骤 1：探索并保存

步骤 2：配置变量

步骤 3：添加追踪面板

步骤 4：查看追踪详情

Zipkin 追踪仪表板

总结

Blog: Monitoring LLM Applications with SkyWalking 10.4: Insights into Performance and Cost

The Problem: As Applications “Consume” LLMs, Monitoring Leaves a Blind Spot

1. The “Black Box” of Cost and Performance: Is the Expensive Model Worth It?

2. The Vanishing “Golden Timeout”

3. The Overlooked Experience Killer: TTFT

Virtual GenAI Observability

How It Works

Installation & Configuration

Requirements