Apache SkyWalking – Community

Blog: Monitoring WeChat and Alipay Mini Programs with SkyWalking

Thu, 30 Apr 2026 00:00:00 +0000

Mini programs are a major part of the mobile experience in China, but the open-source observability ecosystem has long focused on web browsers and native apps. SkyWalking already covers browser (client-js), iOS, and the server side; mini programs and Android were the remaining gaps. With SkyAPM/mini-program-monitor joining the SkyWalking ecosystem, the mini-program half of that gap is closed — one SDK supports both WeChat and Alipay, and the matching OAP-side component IDs, MAL rules, and UI templates are merged on main and will ship with 10.5.0.

This post is for teams that already run a SkyWalking backend and want to bring their mini programs into the same observability stack. The interesting parts aren’t that the project exists — they are how the data flows from a mini program to a SkyWalking dashboard, how the two platforms coexist, and what design trade-offs you should know about before rolling this out.

Data path

The SDK uses two protocols:

OTLP HTTP (error logs, performance metrics, request metrics) → OAP /v1/logs, /v1/metrics
SkyWalking native (distributed tracing segments, optional) → OAP /v3/segments

Why not a single protocol? OTLP already covers logs and metrics, so there’s no point reinventing native endpoints for those. But for tracing, OAP’s native SegmentObject maps more cleanly onto SkyWalking’s trace model, and sw8 header propagation to the backend works without any conversion. So traces go native, everything else goes OTLP, and neither side has to translate.

OTLP defaults to protobuf; JSON is available for debugging. The SDK has zero runtime dependencies.

Two platforms, two independent Layers and dashboards

Many teams maintain a WeChat mini program and an Alipay mini program against a shared backend. Rather than collapsing them into a single tagged service, the design promotes each platform to its own Layer — WECHAT_MINI_PROGRAM and ALIPAY_MINI_PROGRAM — with its own dashboard set. The SDK tags every signal with a resource attribute miniprogram.platform = wechat | alipay and assigns each platform its own component ID (WeChat = 10002, Alipay = 10003).

On the OAP side, the MAL rule’s filter routes data into the right Layer at ingest:

metricPrefix: meter_wechat_mp
filter: "{ tags -> tags.miniprogram_platform == 'wechat' }"

The Alipay rule mirrors this with 'alipay'. The two rules are mutually exclusive — no double counting — and produce distinct metric prefixes (meter_wechat_mp_* vs meter_alipay_mp_*) that feed each Layer’s dashboards. Even when both platforms use the same service.name (e.g. mini-program-demo), the UI exposes two completely separate entry points.

Asymmetric metric semantics

This is the design choice I want to highlight. WeChat’s base library exposes PerformanceObserver, which gives you renderer-authoritative timings: app launch, first render, route navigation, script execution, sub-package load — all real measurements. Alipay’s base library doesn’t offer an equivalent, so the SDK falls back to lifecycle hooks: the App.onLaunch → App.onShow delta is used as an approximation of launch time, and renderer-level timings simply aren’t available.

So the two MAL rule sets are deliberately not the same:

WeChat: app_launch_duration, first_render_duration, route_duration, script_duration, package_load_duration, request_duration_percentile, request_cpm
Alipay: app_launch_duration, first_render_duration, request_duration_percentile, request_cpm

The Alipay app_launch_duration is a lifecycle approximation and is not directly comparable to WeChat’s renderer timing — the dashboard tooltip says so explicitly. Putting the two numbers side by side is comparing two different measurement definitions.

What the SDK does

Four signals:

Errors — JS exceptions, unhandled promise rejections, and pageNotFound go out as OTLP logs, following the OTel exception.* semantic conventions (exception.type, exception.stacktrace). Anything downstream that speaks OTLP — SkyWalking, OTel Collector, Grafana — recognizes them.
Performance — the metrics listed above. OTLP gauge.
Requests — wx.request / my.request / downloadFile / uploadFile are reported as OTLP delta histograms, one batch per flushInterval (default 5s). The le bucket labels are already in milliseconds, and the MAL rule explicitly declares MILLISECONDS to disable the default SECONDS→MS rescale. Failed requests (4xx / 5xx / timeout) additionally emit an error log so you can pivot from a dashboard to a concrete failure.
Tracing (opt-in) — when enabled, outbound requests get sw8 header injection, and the resulting segments stitch together with backend traces into one end-to-end view. Trace data goes out as SkyWalking SegmentObject, not OTLP traces.

Two reliability and cardinality details worth calling out:

Persisting events on app hide. Mini programs get killed by the framework after some time in background, and weak networks make in-flight events easy to lose. The SDK writes unsent events to wx.setStorage / my.setStorage on onAppHide and restores them on the next launch.

Avoiding cardinality explosions. Set serviceInstance to the app version (e.g. 1.4.2), not a device ID — at a million DAU the device-ID dimension blows up the OAP instance index. For request paths, the SDK exposes urlGroupRules regex patterns to fold parameterized URLs like /api/user/12345 into /api/user/{id} so the endpoint dimension doesn’t blow up either.

What OAP needs

If you’re on main or a release ≥ 10.5.0, the following are already shipped:

config/component-libraries.yml registers WeChat-MiniProgram: 10002 and AliPay-MiniProgram: 10003
config/otel-rules/miniprogram/ holds four MAL rules — service-scoped and instance-scoped for each platform
config/ui-initialized-templates/wechat_mini_program/ and alipay_mini_program/ carry root / service / instance / endpoint dashboards
config/ui-initialized-templates/menu.yaml registers both layers under the Mobile menu group

The only thing left is enabling the OTel receiver and giving the SDK an OTLP HTTP port it can reach. SkyWalking OAP binds its OTLP HTTP handler onto the receiver-sharing-server port, and that port defaults to 0 — meaning it’s folded into the core REST port (12800). If you want the SDK to use the standard OTLP HTTP port 4318, set the sharing port to 4318:

docker run -d --name sw-oap \
  -p 11800:11800 -p 12800:12800 -p 4318:4318 \
  -e SW_STORAGE=banyandb \
  -e SW_STORAGE_BANYANDB_TARGETS=banyandb:17912 \
  -e SW_OTEL_RECEIVER=default \
  -e SW_RECEIVER_SHARING_REST_PORT=4318 \
  apache/skywalking-oap-server:latest

All receivers (OTLP, native segment, browser perf, log report) move to 4318 together, while GraphQL stays on 12800 for the UI.

Minimal SDK config:

import MiniProgramMonitor from 'mini-program-monitor';

MiniProgramMonitor.init({
  service: 'mini-program-demo',
  serviceInstance: '1.4.2',          // Recommended: app version
  collector: 'http://your-oap:4318',
  enable: {
    error: true,
    perf: true,
    request: true,
    tracing: false,                  // Off by default; enable as needed
  },
});

WeChat and Alipay use the same config — the SDK detects the platform at runtime and tags the data accordingly.

Compatibility

WeChat base library ≥ 2.11
Alipay base library ≥ 2.0
Apache SkyWalking OAP main or ≥ 10.5.0, with the OTLP HTTP receiver enabled
Any other OTLP-compatible backend (OpenTelemetry Collector, Grafana, etc.) also works, but you won’t get the SkyWalking-specific cross-platform dashboards

What’s next

To get involved, head over to SkyAPM/mini-program-monitor and open an issue or PR. The repo also ships a make preview target that boots OAP, the UI, and both platform simulators locally — handy if you want to play with it end-to-end.

Android end-user experience monitoring is still a gap in the SkyWalking ecosystem; contributors interested in closing that one are very welcome.

Zh: 用 SkyWalking 监控微信和支付宝小程序

Thu, 30 Apr 2026 00:00:00 +0000

小程序是国内移动端体验里绕不过去的一块，但开源监控生态长期偏向 Web 浏览器和原生 App。SkyWalking 自身已经覆盖了浏览器（client-js）、iOS、服务端，缺口主要在小程序和 Android。SkyAPM/mini-program-monitor 加入 SkyWalking 生态后，把这块缺口的小程序部分补上了 —— 一份 SDK 同时支持微信和支付宝，OAP 端的 component、MAL 规则、UI 模板已经合进 main 分支，会随 10.5.0 一起发布。

这篇博客面向已经跑着 SkyWalking 后端、希望把小程序也接进来的团队。重点不是"项目存在"这件事，而是数据从小程序到 SkyWalking dashboard 走的是哪条路、双平台是怎么共存的、以及上线之前需要知道哪些设计取舍。

数据通路

SDK 走两条腿：

OTLP HTTP（错误日志、性能指标、请求指标）→ OAP 的 /v1/logs、/v1/metrics
SkyWalking 原生协议（链路追踪 segment，可选）→ OAP 的 /v3/segments

为什么不是单协议？OTLP 已经覆盖了 logs 和 metrics 两类信号，没必要再造一份原生 endpoint；但分布式追踪上 OAP 的原生 SegmentObject 比 OTLP traces 表达力更贴 SkyWalking 自己的 trace 模型，且与服务端通过 sw8 header 透传时无需任何转换。所以追踪走原生，其它走 OTLP，两边都不绕路。

OTLP 默认用 protobuf，调试时可切成 JSON。SDK 没有任何运行时依赖。

双平台对应两个独立的 Layer 与监控面板

很多团队会同时维护一个微信小程序和一个支付宝小程序，业务逻辑共享一个后端。这套设计没有把它们塞进同一个 service 用 tag 区分，而是直接做成两个独立的 Layer：WECHAT_MINI_PROGRAM 和 ALIPAY_MINI_PROGRAM，对应两套独立的监控面板。SDK 在每个信号上打 resource 属性 miniprogram.platform = wechat | alipay，并给两端各分配独立的 component ID（微信 10002、支付宝 10003）。

OAP 这一头是用 MAL 规则的 filter 把数据在 ingest 阶段就分流到对应 Layer 的：

metricPrefix: meter_wechat_mp
filter: "{ tags -> tags.miniprogram_platform == 'wechat' }"

支付宝那份规则同理过滤 alipay。两份规则互斥，不会重复计数；输出的 metric 前缀也不一样（meter_wechat_mp_* vs meter_alipay_mp_*），各自落在对应 Layer 的 dashboard 上。即使两端用同一个 service.name（比如都叫 mini-program-demo），UI 里也是两套完全独立的入口。

不对等的指标语义

这是这套设计里我特别想强调的一处诚实选择。微信的基础库提供 PerformanceObserver，能拿到来自渲染层的权威时序：app launch、first render、route navigation、script execution、sub-package load 都是真实指标。支付宝的基础库不提供等价 API，SDK 只能用生命周期回退做近似：App.onLaunch → App.onShow 的 delta 当作启动时间，渲染相关的拿不到。

所以两份 OAP 规则里的 metric 集合不对等：

微信：app_launch_duration、first_render_duration、route_duration、script_duration、package_load_duration、request_duration_percentile、request_cpm
支付宝：app_launch_duration、first_render_duration、request_duration_percentile、request_cpm

支付宝侧的 app_launch_duration 是生命周期近似值，与微信的渲染层数值不可直接对比，这一点在 dashboard 的字段提示里也写明了。把两个数字放一起做横评等于在比较两种不同测量定义。

SDK 端做了什么

四类信号：

错误：JS 异常 / unhandled promise rejection / pageNotFound 走 OTLP logs，按 OTel exception.* 语义约定（exception.type、exception.stacktrace），下游不光 SkyWalking，OTel Collector / Grafana 也都认。
性能：上面那张表里那些。OTLP gauge。
请求：wx.request / my.request / downloadFile / uploadFile 都走 OTLP delta histogram，每个 flush 间隔（默认 5s）发一次增量。le 桶标签直接用 ms，OAP MAL 里显式声明 MILLISECONDS 阻止默认的 SECONDS→MS 缩放。失败请求（4xx/5xx/超时）额外发一条错误日志，方便从 dashboard 跳到具体错误。
追踪（可选）：开启后给出站请求注入 sw8 头，落到 OAP 后能与服务端 trace 拼成一条完整链路。trace 段以 SkyWalking SegmentObject 形式发出，不走 OTLP traces。

可靠性和基数控制的两个细节值得一提：

App hide 时落本地存储。小程序后台一段时间会被框架杀掉，弱网时也容易丢包。SDK 在 onAppHide 时把未发送的事件写到 wx.setStorage / my.setStorage，下次启动恢复并继续上报。

反基数膨胀。强烈建议把 serviceInstance 设成应用版本号（如 1.4.2），不要用设备 ID —— 小程序日活百万级时设备 ID 维度直接把 OAP 的 instance 索引打爆。请求路径方面 SDK 提供 urlGroupRules 正则把 /api/user/12345 这类参数化路径归并到 /api/user/{id}，避免 endpoint 维度也膨胀。

OAP 端要做什么

如果你用的是 main 分支或者 10.5.0 之后的发布版，下面这些已经内置：

config/component-libraries.yml：注册了 WeChat-MiniProgram: 10002 和 AliPay-MiniProgram: 10003
config/otel-rules/miniprogram/：四份 MAL 规则，按 service / instance 维度分别定义
config/ui-initialized-templates/wechat_mini_program/ 和 alipay_mini_program/：root / service / instance / endpoint 四张 dashboard
config/ui-initialized-templates/menu.yaml：把两个 layer 注册到 Mobile 菜单组下

唯一需要做的就是确认 OTel receiver 启用、给 OTLP HTTP 一个 SDK 能直连的端口。SkyWalking OAP 的 OTLP HTTP handler 默认绑在 receiver-sharing-server 的端口上，而该端口默认值是 0（即复用 core REST 端口 12800）。如果想让 SDK 用标准 OTLP HTTP 端口 4318，把 sharing 端口设到 4318：

docker run -d --name sw-oap \
  -p 11800:11800 -p 12800:12800 -p 4318:4318 \
  -e SW_STORAGE=banyandb \
  -e SW_STORAGE_BANYANDB_TARGETS=banyandb:17912 \
  -e SW_OTEL_RECEIVER=default \
  -e SW_RECEIVER_SHARING_REST_PORT=4318 \
  apache/skywalking-oap-server:latest

这样所有 receiver（OTLP + native segment + browser perf + log report）一起搬到 4318，GraphQL 仍在 12800 给 UI 用。

SDK 端配置最小集：

import MiniProgramMonitor from 'mini-program-monitor';

MiniProgramMonitor.init({
  service: 'mini-program-demo',
  serviceInstance: '1.4.2',          // 推荐：应用版本号
  collector: 'http://your-oap:4318',
  enable: {
    error: true,
    perf: true,
    request: true,
    tracing: false,                  // 默认关，按需开
  },
});

微信和支付宝两端配置一模一样，平台标签由 SDK 在运行时自动判定。

兼容性

微信基础库 ≥ 2.11
支付宝基础库 ≥ 2.0
Apache SkyWalking OAP main 分支或 ≥ 10.5.0；OTLP HTTP receiver 启用即可
也可对接任意 OTLP 后端（OpenTelemetry Collector、Grafana 等），但那条路上拿不到 SkyWalking 专属的双平台 dashboard

后续

参与方式直接去 SkyAPM/mini-program-monitor 提 issue / PR。仓库里有一个 make preview 一键拉起 OAP、UI、两端模拟器的本地 demo 环境，想看效果可以直接跑。

Android 端的端用户体验监控目前还是 SkyWalking 生态的空白，欢迎对这块感兴趣的同学一起补齐。

Blog: Agentic Vibe Coding in a Mature OSS Project: What Worked, What Didn't

Sun, 08 Mar 2026 00:00:00 +0000

Most “vibe coding” stories start with a greenfield project. This one doesn’t.

Apache SkyWalking is a 9-year-old observability platform with hundreds of production deployments, a complex DSL stack, and an external API surface that users have built dashboards, alerting rules, and automation scripts against. When I decided to replace the core scripting engine — purging the Groovy runtime from four DSL compilers — the constraint wasn’t “can AI write the code?” It was: “can AI write the code without breaking anything for existing users?”

The answer turned out to be yes — ~77,000 lines changed across 10 major PRs in about 5 weeks — but only because the AI was tightly guided by a human who understood the project’s architecture, its compatibility contracts, and its users. This post is about the methodology: what worked, what didn’t, and what mature open-source maintainers should know before handing their codebase to AI agents.

The Project in Brief

The task was to replace SkyWalking’s Groovy-based scripting engines (MAL, LAL, Hierarchy) with a unified ANTLR4 + Javassist bytecode compilation pipeline, matching the architecture already proven by the OAL compiler. The internal tech stack was completely overhauled; the external interface had to remain identical.

Beyond the compiler rewrites, the scope included a new queue infrastructure (threads dropped from 36 to 15), virtual thread support for JDK 25+, and E2E test modernization. By conventional estimates, this was 5-8 months of senior engineer work.

For the full technical details on the compiler architecture, see the Groovy elimination discussion.

What is Agentic Vibe Coding?

“Vibe coding” — a term coined by Andrej Karpathy — describes a style of programming where you describe intent and let AI write the code. It’s powerful for prototyping, but on its own, it’s risky for production systems.

Agentic vibe coding takes this further: instead of a single AI autocomplete, you orchestrate multiple AI agents — each with different strengths — under your architectural direction, with automated tests as the safety net. In my workflow:

Claude Code (plan mode): Primary coding agent. Plan mode lets me review the approach before any code is generated. This is critical for architectural decisions — I steer the design, Claude handles the implementation.
Gemini: Code review, concurrency analysis, and verification reports. Gemini reviewed every major PR for thread-safety, feature parity, and edge cases.
Codex: Autonomous task execution for well-defined, bounded work items.

The key insight: AI writes the code, but the architect owns the design. Without deep domain knowledge of SkyWalking’s internals, no AI could have planned these changes. Without AI, I couldn’t have executed them in 5 weeks.

How TDD Made AI Coding Safe

The reason I could move this fast without breaking things comes down to one principle: never let AI code without a test harness.

My workflow for each major change:

Plan mode first: Describe the goal to Claude, review the plan, iterate on architecture before any code is written.
Write the test contract: Define what “correct” means — for the compiler rewrites, this meant cross-version comparison tests that run every expression through both the old and new engines, asserting identical results across 1,290+ expressions.
Let AI implement: With the test contract in place, Claude can write thousands of lines of implementation code. If it’s wrong, the tests catch it immediately.
E2E as the final gate: Every PR must pass the full E2E test suite — Docker-based integration tests that boot the entire server with real storage backends.
AI code review: Gemini reviewed each PR for concurrency issues, thread-safety, and feature parity — catching things that unit tests alone wouldn’t find.

This is the opposite of “hope it works” vibe coding. The AI writes fast, the tests verify fast, and I steer the architecture. The feedback loop is tight enough that I can iterate on complex compiler code in minutes instead of days.

Lessons Learned

AI is a force multiplier, not a replacement. Before any AI agent wrote a single line, a human had to define the replacement solution: what gets replaced, how it gets replaced, and — critically — where the boundaries are. Which APIs could break? The internal compilation pipeline was fair game for a complete overhaul. Which APIs must stay aligned? Every external-facing DSL syntax, every YAML configuration key, every metrics name and tag structure had to remain byte-for-byte identical — because hundreds of deployed dashboards, alerting rules, and user scripts depend on them. Drawing these boundaries required deep knowledge of the codebase and its users. AI executed the plan at extraordinary speed, but the plan itself — the scope, the invariants, the compatibility contract — had to come from a human who understood the blast radius of every change.

Plan mode is non-negotiable for architectural work. Letting AI jump straight to code on a compiler rewrite would be a disaster. Plan mode’s strength is that it collects code context — scanning imports, tracing call chains, mapping class hierarchies — and uses that context to help me fill in implementation details I’d otherwise have to look up manually. But it can’t tell you the design principles. That direction had to come from me, stated clearly upfront, so the AI’s planning stayed on the right track instead of optimizing toward a locally reasonable but architecturally wrong solution.

Know when to hit ESC. Claude has a clear tendency to dive deep into solution code writing once it starts — and it won’t stop on its own when it encounters something that conflicts with the original plan’s concept. Instead of pausing to flag the conflict, it will push forward, improvising around the obstacle in ways that silently violate the design intent. I had to learn to watch for this: when Claude’s output started drifting from the plan, I’d manually cancel the task (ESC), call it off, identify where the plan and reality diverged, adjust the plan, and restart. This interrupt-replan cycle was a regular part of the workflow, not an exception. The architect has to stay in the loop — not just at planning time, but during execution — because AI agents don’t yet know when to stop and ask.

Spec-driven testing is necessary but not sufficient — the logic workflow matters more. It’s tempting to think that if you define the input/output spec clearly enough, AI can fill in the implementation and tests will catch any mistakes. I tried this. It doesn’t work for anything non-trivial. During the expression compiler rewrite, Claude would sometimes change code in unreasonable ways just to make the spec tests pass — the inputs went in, the expected outputs came out, and everything looked green. But the internal logic was wrong: inconsistent with the design patterns the rest of the codebase relied on, impossible to extend, or solving the specific test case through a hack rather than a general mechanism. A spec only checks what the code produces; it says nothing about how the code produces it. For a mature project, the “how” matters enormously — the solution needs to be consistent with the existing architecture, widely adoptable by contributors, and maintainable long-term. That’s why I needed cross-version testing and human review of the implementation path, not just the results.

Testing at two levels kept the rewrite honest. Cross-version testing was part of my design plan from the start — I architected the dual-path comparison framework so that every production DSL expression runs through both the old and new engines, asserting identical results across 1,290+ expressions. This gave me confidence no human review could match, and it was a deliberate planning decision: I knew AI-generated compiler code needed a mechanical proof of behavioral equivalence, not just eyeball review. On top of that, E2E tests served as the project’s existing infrastructure safety net — Docker-based integration tests that boot the entire server with real storage backends. Unit tests and cross-version tests verify logic in isolation; E2E tests verify the system actually works end-to-end. For infrastructure-level changes like queue replacement and thread model changes, E2E is the only gate that truly matters. Together, the two layers — designed-for-this-rewrite cross-version tests and pre-existing E2E infrastructure — caught different classes of bugs and made shipping with confidence possible.

Multiple AIs have different strengths. Claude excels at large-scale code generation with plan mode. Gemini is exceptional at logic review — it can mentally trace code branches with given input data, simulating execution without actually running the code. This is significant for reviewing AI-generated code: Gemini would walk through a generated compiler method step by step, flagging where a null check was missing or where a branch would produce wrong output for a specific edge case. Codex proved most valuable as a test reviewer and honesty checker. AI-generated code has a subtle failure mode: the coding agent can make wrong assumptions and then write tests that pass by setting expected values to match the wrong behavior — effectively bypassing the test safety net. Codex caught cases where Claude had set unreasonable expected values that happened to make tests green, masking logic errors that would have surfaced in production. Using all three as checks on each other was far more effective than relying on any single one.

The Mythical Man-Month still applies — and so does the Mythical Token-Month. Brooks taught us that a task requiring 12 person-months does not mean 12 people can finish it in one month. The same law applies to AI: you cannot simply throw more tokens, more agents, or more parallel sessions at a problem and expect it to converge faster. Communication costs, coordination overhead, requirements analysis, and conceptual integrity — these software engineering fundamentals do not disappear just because your workforce is artificial. Worse, when the direction is wrong — when there’s a conceptual error in the design or an unreasonable architectural choice — AI will not recognize it. It will charge down the wrong path at extraordinary speed, burning tokens furiously while trapped in a vortex of self-justification: patching code to make failing tests pass, adjusting expected values to match wrong behavior, adding workarounds on top of workarounds — each iteration making the codebase look more “complete” while drifting further from correctness. AI vibe coding cannot break out of this spiral on its own. Only a human who understands the domain can recognize “this is fundamentally wrong, stop,” discard the work, and redirect. Speed without direction is just expensive chaos.

The Bigger Picture

The agentic vibe coding approach worked because it combined AI’s speed with human architectural judgment and automated test discipline. It’s not magic — it’s engineering, accelerated.

Brooks also gave us “No Silver Bullet,” and its core distinction matters more than ever: software complexity comes in two kinds. Essential complexity comes from the problem itself — the domain semantics, the behavioral contracts, the concurrency invariants. No tool can eliminate this; it must be understood, modeled, and reasoned about by someone who knows the domain. Accidental complexity comes from the tools and implementation — boilerplate code, manual refactoring across hundreds of files, the mechanical work of translating a design into compilable source. This is exactly where AI excels. What made this project work was recognizing which complexity was which: I owned the essential complexity (architecture, API boundaries, correctness invariants), and AI demolished the accidental complexity (generating 77K lines of implementation, scaffolding test harnesses, rewriting repetitive patterns across dozens of config files). Confuse the two — let AI make essential decisions, or waste human time on accidental work — and you get the worst of both worlds.

Qian Xuesen(Tsien Hsue-shen)’s Engineering Cybernetics offers another lens that proved surprisingly relevant. His core framework — feedback, control, optimization — describes how to keep complex systems running toward their target. AI vibe coding at full speed is like a hypersonic missile: extraordinarily fast, but without a guidance system it just creates a bigger crater in the wrong place. The feedback loop in my workflow was the test harness — cross-version tests and E2E tests providing continuous signal on whether the system was still on course. Control was the human architect deciding when to intervene: reviewing plans before execution, hitting ESC when the direction drifted, choosing which AI to trust for which task. Optimization was iterative: each interrupt-replan cycle refined the approach, each Gemini review tightened the logic, each Codex audit caught assumptions the coding agent had smuggled past the tests. Without all three — feedback to detect deviation, control to correct course, optimization to converge — the speed of AI coding would be not an advantage but a liability. The faster the missile, the more precise the guidance must be.

For more details or to share your own experience with agentic coding on production systems, feel free to reach me on GitHub.

Zh: 在成熟开源大型项目中实践 Agentic Vibe Coding：软件工程与工程控制论还在延续

Sun, 08 Mar 2026 00:00:00 +0000

大多数"vibe coding"的故事都从一个全新项目开始，讲述一个快速构建原型或者可运行项目的过程，但这篇不是。

Apache SkyWalking 是一个有 9 年历史的Apache顶级项目，线上数以千计的集群部署，内部有一套复杂的 DSL 编译栈，对外暴露的 API 上承载着用户构建的仪表盘、告警规则和自动化脚本。当我决定替换核心脚本引擎——从四个 DSL 编译器中彻底移除 Groovy 运行时——面临的问题不是"AI 能不能写出代码"，而是"也许只有AI能完成如此大规模的一致性迭代"，以及"AI 能不能在不破坏系统的前提下写出完整且高效的代码"。

答案是可以——约 7.7 万行代码变更，10 个主要 PR，历时约 5 周——但前提是 AI 始终在一个深刻理解项目架构、兼容性要求和用户场景的人的引导下工作。这篇文章分享了我在过去几个月的实践体验，以及成熟开源项目的维护者在把代码库交给 AI 智能体之前应该知道什么。

项目概况

这次的任务是将 SkyWalking 基于 Groovy 的脚本引擎（MAL、LAL、Hierarchy）替换为统一的 ANTLR4 + Javassist 字节码编译管线，对齐 OAL 编译器已经验证过的架构。内部技术栈彻底重构，但对外接口必须保持完全一致。

除了编译器重写，范围还包括新的线程管理策略（线程数从 36 降到 15）、JDK 25+ 虚拟线程支持，以及端到端测试的现代化改造。按传统估算，这是 5-8 个月的资深工程师（以我自己为例）工作量。

编译器架构的完整技术细节，参见 Groovy 移除讨论。

什么是 Agentic Vibe Coding？

“Vibe coding”——Andrej Karpathy 提出的概念——描述的是一种你表达意图、让 AI 来写代码的编程风格。整个AI编程过程，一直以来都是用来做原型，效果强大且速度迅猛，但单独用于生产系统是有风险的。

Agentic vibe coding 更进一步：不是单一的 AI 自动补全，而是在你的架构指导下编排多个 AI 智能体——各有所长——以自动化测试作为安全网。我的工作流是这样的：

Claude Code（plan 模式）：主力编码智能体。Plan 模式让我在生成任何代码之前先审查方案。这对架构决策至关重要——我把控设计方向，Claude 负责实现。
Gemini：代码审查、并发分析和验证报告。每个主要 PR 都经过 Gemini 审查线程安全性、功能对等性和边界情况。
Codex：对定义明确、边界清晰的工作项进行自主任务执行。

核心洞察：AI 写代码，但架构师掌控设计。 没有对 SkyWalking 内部机制的深入领域知识，任何 AI 都无法规划这些变更。没有 AI，我也不可能在 5 周内完成执行。

TDD 如何让 AI 编程变得安全

我能以这样的速度推进而不搞砸，归结为一个原则：绝不让 AI 在没有测试保护的情况下写代码。

每次重大变更的工作流：

先进 plan 模式：向 Claude 描述目标，审查方案，在写任何代码之前先在架构层面迭代。
编写测试契约：定义"正确"意味着什么——对于编译器重写，这意味着交叉版本对比测试，让每个表达式同时通过新旧两个引擎运行，在 1290+ 个表达式上断言结果完全一致。
让 AI 实现：有了测试契约，Claude 可以写出数千行实现代码。如果写错了，测试会立即捕获。
端到端测试作为最终关卡：每个 PR 都必须通过完整的端到端测试套件——基于 Docker 的集成测试，启动整个服务器并连接真实存储后端。
AI 代码审查：Gemini 审查每个 PR 的并发问题、线程安全性和功能对等性——捕获单元测试无法发现的问题。

这和"写完祈祷能跑"的 vibe coding 完全相反。AI 写得快，测试验证得快，我把控架构方向。反馈循环足够紧凑，让我能在几分钟而不是几天内迭代复杂的编译器代码。

经验教训

AI 是力量倍增器，不是替代品。 在任何 AI 智能体写下第一行代码之前，必须由人来定义替换方案：替换什么、怎么替换，以及——至关重要的——边界在哪里。哪些 API 可以破坏性变更？内部编译管线可以彻底重构。哪些 API 必须保持对齐？每一个对外的 DSL 语法、每一个 YAML 配置键、每一个指标名称和标签结构都必须逐字节保持一致——因为数百个已部署的仪表盘、告警规则和用户脚本依赖于它们。划定这些边界需要对代码库及其用户的深入了解。AI 以惊人的速度执行了计划，但计划本身——范围、不变量、兼容性契约——必须来自一个理解每次变更影响半径的人。

架构级工作，plan 模式不可妥协。 让 AI 在编译器重写上直接跳到写代码，那是灾难。Plan 模式的价值在于它会收集代码上下文——扫描 import、追踪调用链、映射类继承关系——并利用这些上下文帮我补全那些我本来需要手动查找的实现细节。但它无法告诉你设计原则。方向必须由我在前期明确给出，这样 AI 的规划才能沿着正确的轨道走，而不是朝着一个局部合理但架构上错误的方案去优化。

要知道什么时候该按 ESC。 Claude 有一个明显的倾向：一旦开始写解决方案代码就会一头扎进去——当遇到与原始计划概念冲突的东西时，它不会自己停下来。它不会暂停来标记冲突，而是会继续推进，用即兴的方式绕过障碍，悄无声息地违背设计意图。我必须学会观察这个信号：当 Claude 的输出开始偏离计划时，我会手动取消任务（ESC），叫停它，找出计划和现实的分歧点，调整计划，然后重新开始。这种中断-重新规划的循环是工作流的常态，而非例外。架构师必须始终在环路中——不仅是在规划阶段，执行阶段也是——因为 AI 智能体还不知道什么时候该停下来问一句。

Spec-Driven 更多的运用于测试，而非开发。它只是一个必要的但不充分条件，而逻辑工作流更重要。 很容易产生一种想法：只要把输入/输出规格定义得足够清楚，AI 就能填充实现，测试会捕获任何错误。我试过。对于任何复杂的生产场景，这行不通。在表达式编译器重写过程中，Claude 有时会以不合理的方式修改代码，仅仅为了让规格测试通过——输入进去了，预期输出出来了，一切看起来都是正常的。但内部逻辑是错的：与代码库其他部分依赖的设计模式不一致，无法扩展，或者通过 hack （代码反射、字段名称静态比较等不可接受的工程方法）而非通用机制来解决特定测试用例。规格只检查代码产出了什么；它对代码如何产出一无所知。对于成熟项目，“如何"极其重要——解决方案需要与现有架构一致，能被贡献者广泛采用，并且长期可维护可扩展。这就是为什么我需要交叉版本测试加上对实现路径的人工审查，而不仅仅是审查结果。

两个层次的测试让重写的代码验证更有保障。 交叉版本测试从一开始就是我设计方案的一部分——我架构了双路径对比框架，让每个生产环境的 DSL 表达式同时通过新旧两个引擎运行，在 1290+ 个表达式上断言结果完全一致。这给了我任何人工审查都无法匹敌的信心，而且这是一个刻意的规划决策：我知道 AI 生成的编译器代码需要行为等价性的机械证明，而不仅仅是肉眼审查。在此之上，端到端测试作为项目已有的基础设施安全网——基于 Docker/K8s 的集成测试，启动整个服务器并连接真实存储后端。单元测试和交叉版本测试在隔离环境中验证逻辑；端到端测试验证系统真正能端到端地工作。对于队列替换和线程模型变更这样的基础设施级变更，端到端测试是唯一真正重要的关卡。两个层次——为本次重写专门设计的交叉版本测试和预先存在的端到端基础设施——捕获了不同类别的 bug，使得有信心地发布成为可能。

多个 AI 各有所长。 Claude 擅长配合 plan 模式进行大规模代码生成。Gemini 在逻辑审查方面表现出色——它能在给定输入数据的情况下在脑中追踪代码分支，模拟执行而无需实际运行代码。这对审查 AI 生成的代码意义重大：Gemini 会逐步走查一个编译器生成的方法，标记出哪里缺少空值检查，或者哪个分支在特定边界情况下会产生错误输出。Codex 作为测试审查者和诚实性检查者最有价值。AI 生成的代码有一种微妙的失败模式：编码智能体可能做出错误假设，然后编写测试时将期望值设置为匹配错误行为——实际上绕过了测试安全网。Codex 捕获了 Claude 设置不合理期望值使测试变绿的情况，掩盖了本会在生产环境中暴露的逻辑错误。将三者互相校验，远比依赖其中任何一个更有效。

人月神话依然适用——基于Token的AI月神话同样如此。 Brooks 告诉我们，一个需要 12 人月的任务不意味着 12 个人能在一个月内完成。同样的定律适用于 AI：你不能简单地投入更多 token、更多智能体或更多并行会话，就指望问题更快收敛。沟通成本、协调开销、需求分析和概念完整性——这些软件工程的基本规律不会因为你的劳动力是人工智能就消失。更糟糕的是，当方向错误时——当设计中存在概念性错误或不合理的架构选择时——AI 不会识别出来。它会以惊人的速度冲向错误的方向，疯狂消耗 token，同时陷入自我辩护的漩涡：修补代码让失败的测试通过，调整期望值去匹配错误行为，在变通方案上叠加变通方案——每次迭代都让代码库看起来更"完整”，实际上却离正确越来越远。AI vibe coding 无法自行跳出这个螺旋。只有理解领域的人才能认识到"这从根本上就是错的，停下来"，丢弃这些工作，重新引导方向。没有方向的速度，只是昂贵的混乱。

更大的图景

Agentic vibe coding 之所以有效，是因为它将 AI 的速度与人的架构判断力和自动化测试纪律结合在了一起。这不是魔法——这是被加速的工程。

Brooks 还给了我们《没有银弹》，其核心区分在今天比以往任何时候都更重要：软件复杂性分为两种。本质复杂性来自问题本身——领域语义、行为契约、并发不变量。没有任何工具能消除它；它必须由理解领域的人去理解、建模和推理。偶然复杂性来自工具和实现——样板代码、跨数百个文件的手动重构、将设计翻译成可编译源码的机械工作。这恰恰是 AI 擅长的地方。这个项目之所以成功，在于认清了哪种复杂性是哪种：我掌控本质复杂性（架构、API 边界、正确性不变量），AI 消灭偶然复杂性（生成 7.7 万行实现代码、搭建测试框架、跨数十个配置文件重写重复模式）。搞混这两者——让 AI 做本质决策，或者让人浪费时间在偶然工作上——你会得到两个世界中最差的结果。

钱学森的《工程控制论》提供了另一个视角，在实践中出人意料地切题。他的核心框架——反馈、控制、优化——描述的是如何让复杂系统持续朝目标运行。全速运转的 AI vibe coding 就像一枚高超音速导弹：速度惊人，但没有制导系统只会在错误的地方炸出一个更大的坑。我工作流中的反馈回路是测试体系——交叉版本测试和端到端测试持续提供系统是否仍在航线上的信号。控制是人类架构师决定何时介入：在执行前审查方案，在方向偏移时按 ESC，选择哪个 AI 负责哪项任务。优化是迭代式的：每次中断-重新规划的循环都在精炼方法，每次 Gemini 审查都在收紧逻辑，每次 Codex 审计都在捕获编码智能体偷偷绕过测试的假设。缺少其中任何一个——检测偏差的反馈、纠正航向的控制、趋向收敛的优化——AI 编程的速度就不是优势而是负债。导弹越快，制导就必须越精确。

AI Vibe Coding以及它的迭代，正在快速地走进每一个开发者，也正在广泛地融入开源和商业软件。我们都在见证这种新的开发模式，以及AI Vibe Coding和软件工程理论的融合。如果你想和我探讨更多的AI + OSS话题，欢迎在 GitHub 上联系我。

Blog: Apache SkyWalking 2025 in Review: Making BanyanDB Ready for Production

Thu, 01 Jan 2026 00:00:00 +0000

2025 was a very focused year for the Apache SkyWalking community: moving BanyanDB from “native storage” to a “production-ready default”, and making SkyWalking APM fully benefit from that foundation.

This post summarizes the key milestones, with an emphasis on BanyanDB.

Storage strategy: saying goodbye to H2

We started 2025 with a clear direction: the H2 storage option is permanently removed. This change reduced long-term maintenance burden and removed a storage choice that was not aligned with production and cloud-native deployments.

BanyanDB: from 0.8.0 foundations to 0.9.0 production features

BanyanDB 0.8.0 delivered the “day-2 operations” foundation that a default storage backend needs. The community put a lot of effort into making queries faster and more predictable (for example index_mode, numeric index types, and multiple query-path optimizations), while also making the system safer under real production pressure. Disk-usage thresholds and a query memory protector were added as guardrails, and the operational toolbox matured with snapshot/backup/restore utilities and improved metadata synchronization.

Just as importantly, 0.8.0 started filling in the missing pieces of a full platform: native property storage and lifecycle-related capabilities that later enabled stronger HA and stage-based deployment patterns.

BanyanDB 0.9.0 was the “production features” milestone. It introduced the Trace data model as a first-class citizen, which unlocked much deeper trace storage and query capabilities. On the reliability and scaling side, the release brought configurable replicas, liaison-side improvements (including load balancing and moving some TopN flow), and broader correctness work such as migrations, version compatibility checks, and access logs.

It also made long-term operations more cloud-friendly with backup/restore support for AWS S3, GCS, and Azure Blob Storage, and added authentication primitives needed in shared environments. In short, 0.9.0 is where BanyanDB clearly moved beyond a “fast storage engine” into a “production platform”.

SkyWalking APM: BanyanDB becomes the default path

With APM 10.2.0, the project made the strategic shift official: H2 was removed permanently, and BanyanDB 0.8.0 became the default path that SkyWalking invests in. A lot of the work here was not flashy, but essential — refining OAP behavior (group settings, index model changes, Progressive TTL, query limits, and more) so running BanyanDB in production felt stable and predictable.

With APM 10.3.0, SkyWalking and BanyanDB moved forward together: BanyanDB 0.9.0’s new trace model was adopted end-to-end, reducing inefficient query round-trips and enabling new query views that significantly lowered page latency. The integration also expanded into lifecycle-aware operations with hot/warm/cold stage configuration (including TTL and query support), and added BanyanDB self-monitoring through OAP and the UI — the kind of end-to-end polish that turns a storage backend into a truly native solution.

If you’d like this review to cover APM 10.4.x as well, please point me to the corresponding release content in this repo (I didn’t find an APM 10.4.0 release announcement in the current checkout).

Packaging and deployment ecosystem (Helm)

BanyanDB’s production readiness is not only server features — it also depends on deployment maturity.

Helm charts:
- SkyWalking Kubernetes Helm Chart 4.8.0 improved BanyanDB deployment defaults by updating the bundled BanyanDB Helm dependency, fixing an init-job volume-mount mismatch, and aligning OAP/UI images with the APM 10.3.0 line.
- BanyanDB Helm 0.4.0 added backup/restore sidecars and a default volume for property storage.
- BanyanDB Helm 0.5.0 introduced stage-aware patterns (hot/warm/cold), improved lifecycle-sidecar scheduling, moved liaison to StatefulSet, refined internal networking, and expanded configuration options.
- BanyanDB Helm 0.5.1 refined liaison configuration and fixed restore-init environment issues.
- BanyanDB Helm 0.5.3 fixed a liaison/data-node port issue.

The rest of the community: agents and tooling kept moving

While storage was the “main storyline”, the community shipped releases across agents, clients, and surrounding components throughout 2025.

Below is a consolidated view of the other releases, grouped by project, with the most important notes.

SkyWalking Java Agent
- 9.4.0: agent self-observability; async-profiler support; broader plugin improvements.
- 9.5.0: virtual thread executor plugin; compatibility and stability fixes; dependency upgrades.
SkyWalking Go
- 0.6.0: richer manual APIs (events/logs/metrics, set span error); goframev2 plugin; bug fixes including Redis cluster mode.
SkyWalking for NodeJS
- 0.8.0: Express 4/5 compatibility, keep-alive HTTP trace fix, and test/dependency maintenance.
SkyWalking Python
- 1.2.0: sampling service, sw_grpc plugin, async/profiling stability fixes, Python 3.13 support, and dropping Python 3.7.
SkyWalking PHP
- 1.0.0: reach 1.0; add PSR-3 log reporting; upgrade toolchain/dependencies.
SkyWalking Rust
- 0.9.0: migrate to Rust edition 2024 and upgrade dependencies.
- 0.10.0: Kafka client configuration refactor, rdkafka upgrade, CI maintenance.
SkyWalking Ruby
- 0.1.0: initialize agent core and e2e tests; add plugins for Sinatra, redis-rb, net-http, memcached, and Elasticsearch.
SkyWalking Client JS
- 1.0.0: add Core Web Vitals and static resource metrics; fix fetch/resource error handling; dependency and e2e/test improvements.
SkyWalking Satellite
- 1.3.0: support native eBPF Access Log protocol and async-profiler protocol; upgrade Go toolchain.
SkyWalking Eyes
- 0.7.0: improve installation/docs, respect gitignore behavior, upgrade Go, and simplify release steps.
- 0.8.0: add Elixir support and stronger dependency-license scanning (notably Ruby via Gemfile.lock), plus stability fixes.

Looking ahead: possible directions in 2026

2025 was about making BanyanDB ready for production. In 2026, the community is exploring the next set of improvements that could make the whole stack simpler to operate, more stable under stress, and easier to integrate into broader observability ecosystems.

Possible areas include:

BanyanDB: remove the etcd dependency: the direction under discussion is to move away from etcd (given ecosystem activity and maintenance concerns) and rely more on DNS-based discovery plus BanyanDB’s native property capabilities.
BanyanDB: stronger stability testing: more systematic testing, including chaos testing, to validate behavior under failures and noisy conditions.
BanyanDB: better observability export: introducing First Occurrence Data Collection (FODC) as a sidecar and proxy server to provide a unified stream of observability data to third-party systems.
SkyWalking APM: broader runtime and query capabilities: cold-stage data query support, a newer Java runtime (Java 25), and consideration of TraceQL protocol (Temper) support.

Closing

Thanks to everyone who contributed to SkyWalking in 2025. Every contribution is high-value — code, documentation, reviews, testing, issue triage, and operational experience — and each of them helped move the project forward.

We also want to say a special thank you to the countless end users across global companies. Many of the most valuable improvements don’t start from a pull request: they start from real-world use cases, performance investigations, production feedback, bug reports, and the patience to help us reproduce and validate fixes.

As another milestone, SkyWalking reached 968 GitHub contributors globally, and we expect the 1000th contributor milestone to arrive soon in 2026. But the community is much larger than the number suggests, and SkyWalking’s progress has always been driven by collaboration between contributors, adopters, and maintainers.

Apache SkyWalking was originally created by Sheng Wu as a personal project in May 2015. It would never have grown into what it is today without the whole community — and it will keep moving forward because of the community.

Blog: Improving Alert Accuracy with Dynamic Baselines

Mon, 24 Feb 2025 00:00:00 +0000

Background

Apache SkyWalking is an open-source application performance monitoring (APM) system that collects various data from business applications, including metrics, logs, and distributed tracing information, and visualizes them through its UI. It also allows users to configure alerting rules by setting threshold values for specific metrics in the configuration file. When a metric associated with a particular service exceeds the predefined threshold within a given period, an alert is triggered.

However, in real-world scenarios, traffic patterns and invocation behaviors vary across different time periods. For example, in a shopping system, the number of purchases is significantly lower during late-night hours compared to daytime. As a result, system metrics fluctuate within different ranges depending on the time of day. This makes it challenging to rely solely on static threshold values for accurate alerting.

Therefore, dynamically generating thresholds for each time period based on historical data becomes crucial.

Introduce SkyAPM SkyPredictor

Based on the above scenario, we developed the SkyAPM SkyPredictor project to fix this issue. SkyAPM SkyPredictor periodically collects data from SkyWalking and generates dynamic baselines. Meanwhile, SkyWalking queries from SkyPredictor to obtain predicted metric values for the recent period, enabling more precise and adaptive alerting.

NOTE: SkyWalking does not have a hard dependency on the SkyPredictor service. If SkyPredictor is not configured, no predicted values would be retrieved, and not cause any failures in SkyWalking. Additionally, you can use your own AI engine to build a custom prediction system. Simply implement the required protocol as outlined in the official documentation: https://skywalking.apache.org/docs/main/next/en/setup/ai-pipeline/metrics-baseline-integration/

Architecture diagram

As shown in the diagram, the process consists of two steps:

Data Collection & Prediction: The Predictor queries history metrics from SkyWalking’s OAP via its HTTP service. Then processes this data to generate dynamic predicted values for a future time period.
Baseline Query & Alerting: The OAP periodically sends queries to the Predictor to fetch the predicted dynamic baseline. Then evaluates the current metric values with prediction result using MQE. If the deviation exceeds a certain threshold, an alert is triggered.

Data Collection

The Predictor utilizes the following three APIs to query data:

Status API: Retrieves the TTL (Time-to-Live) of history data stored in OAP, helping to determine the available time range for exporting all history metrics.
Metadata API: Fetches the list of services within a specified Layer from OAP, providing insights into which services are generating data.
MQE API: Iterates through the required metrics and the list of services to fetch all history metrics values for each metric associated with each service.

These APIs collectively enable the Predictor to gather history metrics data, which is then used to compute dynamic baselines for future alerting.

Prediction

Once the Prediction service collects data from OAP, it proceeds with forecasting using the open-source Prophet library. The prediction process consists of the following steps:

Data Preparation: The collected metric data is split into multiple DataFrames, each corresponding to a unique combination of service + metric name.
Data Sufficiency Check: If a DataFrame contains less than two days (configurable) of data, the prediction is skipped. This is to ensure accuracy, as an insufficient data volume may lead to unreliable forecasts.
Forecasting: Using Prophet, the Predictor estimates the metric values for each hour over the next day (configurable).
Result Storage: The generated predictions are stored in local files, enabling querying from external services.

Predicted Value

The Prediction service supports calculating the following two types of values:

Predicted Value: Computes the expected metric value for the next hour based on history metrics data.
Prediction Range: Determines the possible upper and lower bounds for the metric in the next hour, representing its expected fluctuation range.

These values help establish a dynamic baseline, allowing the alerting system to account for natural variations while accurately detecting anomalies.

Baseline MQE with Alarm

In OAP, predicted values can be queried directly using an MQE within MQE operation. This operation enables retrieving forecasted values for a future time period.

Since SkyWalking’s alerting system already supports query through MQE expressions, users can configure alerts directly in the alerting configuration file using MQE.

For more details, please refer to the official documentation.

Impact of Data Collection on Prediction Accuracy

The Predictor service supports two different data collection and prediction granularity, each with its own trade-offs in accuracy and resource consumption.

Minute Level: Collects minutes level metrics data.
1. More effective for metrics with high fluctuations, as it captures finer details.
2. Consumes more resources (OAP, DB CPU and System Load resources, Predictor CPU and Memory resources).
3. Alerts are configured based on current value comparisons.
Hour Level: Collects hourly metrics data.
1. Less resource-intensive compared to minute-level collection.
2. Less data volume, resource, and processing cost.
3. Alerts are configured based on predicted range values.

Granularity	Data Fluctuation	Data Volume	Current Value Prediction Accuracy	Range Prediction Accuracy	Best Use Case
Minute	Higher fluctuation	Large	Less accurate	More accurate	Ideal for highly fluctuating metrics, using range-based alerting rules
Hour	Lower fluctuation	Small	More accurate	Relatively accurate	Suitable for stable metrics, using current value-based predictions

Choosing the appropriate granularity depends on the nature of the metric and the desired alerting method. For metrics with high volatility, minute-level collection provides better accuracy when using range-based alerts. For stable metrics, hourly aggregation is sufficient and allows for efficient predictions using current-value comparisons.

Predict use Hourly level by default.

OAP and Predictor Scheduling & Caching

Both SkyWalking OAP and SkyAPM Predictor implement caching strategies to prevent excessive execution and optimize resource usage.

By default, Predictor runs at 00:10, 08:10, and 16:10 every day. It forecasts the next 24 hours and stores the results locally. Updating predictions every 8 hours balances resource efficiency and real-time accuracy. The 10-minute delay (instead of running at exactly 00:00, 08:00, etc.) ensures historical data is fully written to the database before querying.

OAP queries Predictor for all required predicted metrics of a single service. The query covers a ±24-hour time range from the current moment. Results are cached for one hour to reduce redundant queries and improve efficiency.

These mechanisms ensure that predictions remain up-to-date, while minimizing unnecessary processing and system load.

Demo

In this section, I will demonstrate how to preview the predicted values of a metric by deploying a SkyWalking cluster along with the Predictor service in a Kubernetes cluster. This hands-on example will help you understand how to use these components effectively.

Deploy SkyWalking Showcase

SkyWalking Showcase contains a complete set of example services and can be monitored using SkyWalking. For more information, please check the official documentation.

In this demo, we only deploy the predictor service, SkyWalking OAP, and UI.

export FEATURE_FLAGS=single-node,banyandb,baseline
make deploy.kubernetes

Import History Data

Since a newly deployed cluster does not contain history data, I have created a Python script to simulate data. This allows the Predictor service to import data and generate baseline predictions for a future period.

Before importing data, you must expose the 11800 port of the OAP service in your Kubernetes cluster. You can achieve this using kubectl by running the following command:

kubectl port-forward -n skywalking-showcase   service/demo-oap 11800:11800

Then, you can download and run the demo script using the following command:

# clone and get into the demo repository
git clone https://github.com/mrproliu/SkyPredictorDemo && cd SkyPredictorDemo
# installing dependencies
make install
# import data(7 days)
python3 -m client.generate localhost:11800 7

Finally, you can see the output in the console: Metrics send success!.

Prediction metrics

Since the Predictor service runs based on a cron schedule, it does not automatically execute immediately after data import. To force it to collect data and perform a prediction, you can manually delete the Predictor pod, prompting Kubernetes to restart it:

kubectl delete pod -n skywalking-showcase $(kubectl get pods -n skywalking-showcase --no-headers -o custom-columns=":metadata.name" | grep "skywalking-predictor")

Once the Predictor pod restarts, you can check its logs to confirm that the prediction process has been completed.

Predicted for e2e-test-dest-service of service_xxx to xxxx-xx-xx xx:xx:xx.

View in SkyWalking UI

Once the prediction process is complete, you can visualize the predicted values in the SkyWalking UI by configuring the appropriate metric widgets.

First, Run the following command to forward the UI service port to your local machine:

kubectl port-forward svc/demo-ui 8080:80 --namespace skywalking-showcase

Then, you can access this page to view the service traffic that was generated using the Python script earlier: http://localhost:8080/dashboard/MESH/Service/ZTJlLXRlc3QtZGVzdC1zZXJ2aWNl.1/Mesh-Service

To display predicted values, edit the Service Avg Resp Time Widget and add the following MQE:

# The maximum predicted response time.
baseline(service_resp_time, upper)
# The predicted response time.
baseline(service_resp_time, value)
# The minimum predicted response time.
baseline(service_resp_time, lower)

Finally, you can see the predicted values displayed in the widget.

Since the default data collection is hourly and the metric has significant fluctuations, the predicted values are derived from hourly averages rather than minute-level granularity. This approach smooths out fluctuations and provides a more stable baseline for monitoring.

Now, you should see the predicted response times visualized alongside actual values, helping you analyze trends and configure dynamic alerting thresholds effectively.

Conclusion

SkyAPM SkyPredictor enhances alert accuracy by using dynamic baselines instead of static thresholds. It collects history metrics data, forecasts future values with Prophet, and supports minute or hour-level collection for better precision. By integrating predictions into SkyWalking UI, users can optimize alerting and improve system observability.

By integrating dynamic thresholds, SkyWalking can adapt to traffic patterns and detect anomalies more effectively, reducing false positives and improving system observability.

Zh: 开源之夏 2024 SkyWalking 社区项目情况公示

Tue, 10 Dec 2024 00:00:00 +0000

Aapche SkyWalking PMC 和 committer团队参加了"开源之夏 2024"活动，作为导师，共获得了5个官方赞助名额。最终对学生开放如下任务

BanyanDB支持自定义插入/更新触发器
在SkyWalking Go的toolkit中支持完整trace, log和meter APIs
在SkyWalking Java中集成JFR性能剖析功能
SWCK 支持注入 skywalking Python agent
BanyanDB支持数据聚合

经过3个月的开发，上游评审，PMC成员评议，PMC Chair复议，OSPP官方委员会评审多个步骤，现公布项目参与人员与最终结果

通过评审项目（共3个）

在SkyWalking Java中集成JFR性能剖析功能

学生：郑子熠
学校：南京邮电大学本科
官方文档Profiling - Async Profiler详细介绍了此功能。
此功能将作为SkyWalking 10.2的主要新功能之一发布，预计发布时间 2025年2月（以官方Release为准）。
官网发布了blog - 使用 SkyWalking中的 async-profiler 对 Java 应用进行性能分析介绍此功能

2024年12月9日，郑子熠因此项目在结项优秀学生评比中，获得突出贡献奖。

在SkyWalking Go的toolkit中支持完整trace, log和meter APIs

学生：李天源
学校：广东东软学院本科
相关Pull Requests
相关官方文档包括
此功能将包含在SkyWalking Go Agent 0.6 release中发布，以及发布时间2025年年初（以官方Release为准）。

BanyanDB 支持自定义插入/更新触发器

学生：谢李奥
学校：中科院软件所硕士
相关官方文档包括
2024年11月，在社区关于BanyanDB未来路线图的讨论决议中，服务端相关聚合功能暂时不作为BanyanDB规划功能，[Feature] Hand over downsampling(hour/day) metric processes to BanyanDB 已被关闭。此项目合并代码，不会发布，将在0.8 release中被移除。注，此代码移除与学生代码质量和完成度无关，系项目目标变更导致。

We have an agreement, this feature benefit is too limited.

# Summary of downsides for server-side aggregation

1. Removing L2 would impact the alerting. Because minute dimension eventual(aggregated) data will be lost from OAP.
2. Keeping minute dimension aggregation makes the delta data lost, and causes another extra round delta data in minute dimension flushing. 
3. Clearly, this server-side aggregation increases the server-side payload(CPU cost), but wouldn't tradeoff for IOPS.
4. The process pipeline would be more complex.
____
The only positive part is the cache of hour/day dimension metrics could be removed.

未通过评审/未启动项目（2个）

下列项目因为质量无法达到社区要求，无学生报名等原因，将被标定为失败。

SWCK 支持注入 skywalking Python agent。无学生申报
BanyanDB支持数据聚合。未通过

结语

2024年，由于开源之夏提供的支持名额降低到5个，SkyWalking选题的难度都有显著提升。我们很高兴的看到，依然有3位学生较好的完成了相关课题。感觉开源之夏和各位同学们对Apache SkyWalking的支持和热情参与。

Zh: 引入 R3 识别 RESTFul URI

Thu, 17 Oct 2024 00:00:00 +0000

背景

在现代应用中，服务通常通过 RESTFul HTTP URI 提供。使用 RESTFul HTTP URI 作为唯一的资源标识符，不仅具备良好的可读性，还能让客户端和服务器更容易理解请求。然而，在可观测性领域，这种方式也带来了一些挑战：

大量的端点（HTTP URI）：浏览所有对外提供的端点变得更加困难，因此很难识别出存在问题的端点。
指标收集困难：尤其难以对类似的端点进行归类并生成可观测性指标。

现有解决方案通常采用以下应用级别的方式来解决此问题：

代理检测：在某些框架中，通常会声明规则来处理 RESTFul 请求。例如，在 Java 的 Spring Web 中可以使用 @GET 等注解，然后可以通过 Java Agent 将其与当前请求关联起来。
OpenAPI：应用可以关联预定义文件，使可观测性系统知晓可能使用的 URI。

这两种解决方案都与应用设置紧密耦合，这对于未知应用或者无法监控代理的应用来说是一个局限。因此，我们需要考虑是否有一种更通用的解决方案来识别 URI，并合并来自类似 URI 生成的指标，以便更好地展示数据。

R3

R3（RESTFul 模式识别）是一个高性能的 RESTFul URI 识别工具，其灵感来自 Drain3。它可以作为独立应用部署在可观测性服务器上，并与 SkyWalking OAP 进行通信。

R3 可以通过 gRPC 协议接收 URI 列表，并将类似的 URI 聚合为特定格式。聚合后的（格式化）URI 列表也可以通过 gRPC 协议进行查询。

数据交互流程

OAP 接收并缓存未格式化的 URI 列表：OAP 通过不同协议接收可观测性数据，并识别出所有未格式化的 URI。然后将这些 URI 按照所属服务暂时存储在列表中。
OAP 将待格式化的 URI 发送给 R3：OAP 定期将需要格式化的 URI 批量发送到 R3 服务。
R3 接收并解析 URI 列表：R3 异步分析接收到的 URI 相似性，并将结果存储（持久化）在本地磁盘中，以便在重启后进行恢复。
OAP 查询 R3 中的格式化 URI 列表：OAP 定期查询 R3 中检测到的格式化 URI，并将结果保存在内存中。
OAP 格式化 URI：当 OAP 接收到新的可观测性数据时，会将 URI 与从 R3 获取的格式化 URI 进行匹配。如果匹配成功，后续的指标计算将使用格式化后的 URI。

场景

在 R3 中，主要解决以下场景。对于识别为重复的 URI，R3 将用 {var} 替换变量部分以标准化 URI。

ID 匹配

在 RESTFul API 中，常见做法是将各种 ID 包含在 URI 路径中，这导致了大量唯一的 URI 端点。例如，以下路径将被 R3 聚合为标准化格式 /api/users/{var}。

/api/users/cbf11b02ea464447b507e8852c32190a
/api/users/5e363a4a18b7464b8cbff1a7ee4c91ca
/api/users/44cf77fc351f4c6c9c4f1448f2f12800
/api/users/38d3be5f9bd44f7f98906ea049694511
/api/users/5ad14302e7924f4aa1d60e58d65b3dd2

词语检测

在 RESTFul URI 中，实体的操作通常通过 HTTP 方法指定，但有时还需要在路径中添加特定名词。为此，R3 实现了词语解析：当 R3 检测到路径中的特定词语时，将不会格式化该部分。例如，以下 URI 将不会被视为相似，因此不会被合并：

/api/sale
/api/product_sale
/api/ProductSale

样本不足

为防止由于样本不足而导致错误判断，R3 允许在配置文件中配置最小 URI 合并计数参数。

例如，当阈值为 3 时，以下 URI 将保持原样，不会被参数化。

/api/fetch1
/api/fetch2

但是以下 URI 将被参数化为 /api/{var}，因为样本数大于阈值。

/api/fetch1
/api/fetch2
/api/fetch3

版本 API

在实际场景中，常会遇到包含多个版本的 URI。R3 通过确保指定路径中包含 v\\d+ 参数（表示版本信息）来解决这个问题，该部分将不会被参数化。例如，以下 URI 将分别解析为 /test/v1/{var} 和 /test/v999/{var}。

/test/v1/cbf11b02ea464447b507e8852c32190a
/test/v1/5e363a4a18b7464b8cbff1a7ee4c91ca
/test/v1/38d3be5f9bd44f7f98906ea049694511
/test/v999/1
/test/v999/2
/test/v999/3

演示

接下来我们快速演示如何使用 R3 格式化观察到的端点，帮助你更具体地理解它的功能。

部署 SkyWalking Showcase

SkyWalking Showcase 包含一整套示例服务，可以通过 SkyWalking 进行监控。更多信息请查看官方文档。

在此演示中，我们仅部署服务、最新发布的 SkyWalking OAP、R3 服务和 UI。

export FEATURE_FLAGS=java-agent-injector,single-node,elasticsearch,r3
make deploy.kubernetes

部署完成后，请运行以下脚本打开 SkyWalking UI：http://localhost:8080/。

kubectl port-forward svc/ui 8080:8080 --namespace default

触发 RESTFul 请求

在 R3 中，默认启动了定时任务以定期生成 RESTFul 流量。不过，你也可以使用以下命令手动触发此过程：

kubectl exec -n sample-services $(kubectl get pod -n sample-services --selector=app=gateway -o jsonpath='{.items[0].metadata.name}') -- /bin/bash -c 'for i in $(seq 1 200); do curl http://rating/songs/$i/reviews/$((i+1)); sleep 1; done'

在上述命令中，R3 将自动定位网关节点，并以 RESTFul 格式向该节点内的 rating 服务发送请求。此操作允许 R3 生成并测试模拟实际 RESTFul 请求的流量模式。

查看格式化 URI

一旦触发 RESTFul 请求，你可以在 UI 中查看聚合后的端点。

注意：由于格式化端点是异步生成的，因此一些较早的请求可能尚未被格式化。你可能需要等待一段时间，UI 才会仅显示格式化后的地址。

结论

在本文中，我们详细讨论了 SkyWalking 如何利用 R3 服务来格式化 RESTFul URI，并在接收后聚合相关的指标。目前，它适用于大多数 RESTFul 场景，如果需要支持更多情况，我们可以根据需求进一步扩展。

Blog: [Video] Coffee + Software with Josh Long - Apache SkyWalking with Sheng Wu and Apache ShardingSphere with Trista Pan

Mon, 04 Dec 2023 00:00:00 +0000

🚀 Dive into the World of Cutting-Edge Technology with Apache’s Finest! 🌐 Join me today as we embark on an exhilarating journey with two of Apache’s most brilliant minds - Sheng Wu and Trista Pan. We’re exploring the realms of Apache SkyWalking and Apache ShardingSphere, two groundbreaking initiatives that are reshaping the landscape of open-source technology. 🌟

In this exclusive session, we delve deep into Apache SkyWalking - an innovative observability platform that’s revolutionizing how we monitor and manage distributed systems in the cloud. Witness firsthand how SkyWalking is empowering developers and organizations to gain unparalleled insights into their applications, ensuring performance, reliability, and efficient troubleshooting. 🛰️🔍

But there’s more! We’re also unveiling the secrets of Apache ShardingSphere, a dynamic distributed database ecosystem. Learn how ShardingSphere is making waves in the world of big data, offering scalable, high-performance solutions for data sharding, encryption, and more. This is your gateway to understanding how these technologies are pivotal in handling massive data sets across various industries. 🌐💾

Whether you’re a developer, tech enthusiast, or just curious about the future of open-source technology, this is a conversation you don’t want to miss! Get ready to be inspired and informed as we unlock new possibilities and applications of Apache SkyWalking and ShardingSphere. 🚀🌟

Join us, and let’s decode the future together!

Please join and follow Josh’s 龙之春 Youtube Coffee + Software with Josh Long Channel to learn more about technology and open source from telanted engineers and industry leads.

Zh: 开源之夏 2023 SkyWalking 社区项目情况公示

Thu, 09 Nov 2023 00:00:00 +0000

Aapche SkyWalking PMC 和 committer团队参加了"开源之夏 2023"活动，作为导师，共获得了9个官方赞助名额。最终对学生开放如下任务

SkyWalking 支持 GraalVM
Skywalking Infra E2E 自测试
监控Apache Pulsar
统一BanyanDB的查询计划和查询执行器
使用Helm部署BanyanDB
编写go agent的gRPC插件
监控Kafka
集成SkyWalking PHP到SkyWalking E2E 测试
在线黄金指标异常检测

经过3个月的开发，上游评审，PMC成员评议，PMC Chair复议，OSPP官方委员会评审多个步骤，现公布项目参与人员与最终结果

通过评审项目（共6个）

SkyWalking 支持 GraalVM

学生：张跃骎
学校：辽宁大学本科
合并PR：11354
后续情况说明：GraalVM因为复杂的生态，替代的代码将被分离到SkyWalking GraalVM Distro, 相关讨论，请参见Issue 11518

Skywalking Infra E2E 自测试

学生：王子忱
学校：华中师范大学本科
合并PR：115, 116, 117, 118, 119
后续情况说明：此特性已经包含在发行版skywalking-infra-e2e v1.3.0中

统一BanyanDB的查询计划和查询执行器

学生：曾家华
学校：电子科技大学本科
合并PR：343

使用Helm部署BanyanDB

学生：黄友亮
学校：北京邮电大学硕士研究生
合并PR：1
情况说明：因为BanyanDB Helm为新项目，学生承接了项目初始化、功能提交、自动化测试，发布准备等多项任务。所参与功能包含在skywalking-banyandb-helm v0.1.0中

编写go agent的gRPC插件

学生：胡宇腾
学校：西安邮电大学
合并PR：88, 94
后续情况说明：该学生在开源之夏相关项目外，完成了feature: add support for iris #99和Go agent APIs功能开发。并发表文章SkyWalking Go Toolkit Trace 详解以及英文译本Detailed explanation of SkyWalking Go Toolkit Trace

监控Kafka

学生：王竹
学校：美国东北大学 ( Northeastern University)
合并PR：11282, UI 318

未通过评审项目（3个）

下列项目因为质量无法达到社区要求，违规等原因，将被标定为失败。 注：在开源之夏中失败的项目，其Pull Reqeust可能因为符合社区功能要求，也被接受合并。

监控Apache Pulsar

学生：孟祥迎
学校：重庆邮电大学本科
合并PR：11339
失败原因：项目申请成员，作为ASF Pulsar项目的Committer，在担任Pulsar开源之夏项目导师期间，但依然申请了学生参与项目。属于违规行为。SkyWalking PMC审查了此行为并通报开源之夏组委会。开源之夏组委会依据活动规则取消其结项奖金。

集成SkyWalking PHP到SkyWalking E2E 测试

学生：罗文
学校：San Jose State University B.S.
合并PR：11330
失败原因：根据pull reqeust中的提交记录，SkyWalking PMC Chair审查了提交明细，学生参与代码数量大幅度小于导师的提交代码。并在考虑到这个项目难度以及明显低于SkyWalking 开源之夏项目的平均水平的情况下，通报给开源之夏组委会。经过组委会综合评定，项目不合格。

在线黄金指标异常检测

学生：黄颖
学校：同济大学研究生
合并PR：无
失败原因：项目在进度延迟后实现较为简单且粗糙，并且没有提供算法评估结果和文档等。在 PR 开启后的为期一个月审核合并期间，学生并未能成功按预定计划改善实现的质量和文档。和导师以及 SkyWalking 社区缺少沟通。

结语

SkyWalking社区每年都有近10位PMC成员或Committer参与开源之夏中，帮助在校学生了解顶级开源项目、开源社区的运作方式。我们希望大家在每年经过3个月的时间，能够真正的帮助在校学生了解开源和参与开源。因为，社区即使在考虑到学生能力的情况下，不会明显的降低pull request的接受标准。希望今后的学生，能够在早期，积极、主动和导师，社区其他成员保持高频率的沟通，对参与的项目有更深入、准确的了解。

Blog: The Statement for SkyWalking users on HashiCorp license changes

Sun, 13 Aug 2023 00:00:00 +0000

On Aug. 10th, 2023, HashiCorp announced to adopt the Business Source License (BSL) from Mozilla Public License v2.0 (MPL 2.0), here is their post. They officially annouced they have changed the license for the ALL of their open-source products from the previous MPL 2.0 to a source-available license, BSL 1.1. Meanwhile, HashiCorp APIs, SDKs, and almost all other libraries will remain MPL 2.0.

HashiCorp Inc. is one of the most important vendors in the cloud-native landscape, as well as Golang ecosystem. This kind of changes would have potential implications for SkyWalking, which is closely integrated with cloud-native technology stacks.

Conclusion First

What does that mean for SkyWalking users?

SkyWalking community has evaluated our dependencies from HashiCorp products and libraries, the current conclusion is

SkyWalking users would NOT suffer any implication. All components of SkyWalking don’t have hard-dependency on BSL license affected codes.

SkyWalking community have found out all following dependencies of all relative repositories, all licenses are TRUELY stayed unchanged, and compatible with Apache 2.0 License.

OAP Server @kezhenxu94 @wu-sheng
- consul-client Apache 2.0 Repo archived on Jul 27, 2023
BanyanDB @hanahmily @lujiajing1126
- Server @hanahmily
  - hashicorp/golang-lru MPL-2.0
  - hashicorp/hcl MPL-2.0
- CLI @hanahmily No HashiCorp Dependency
SkyWalking OAP CLI @kezhenxu94
- github.com/hashicorp/hcl v1.0.0 MPL-2.0
- All under swck as transitive dependencies
SWCK @hanahmily
- hashicorp/consul/api MPL-2.0
- hashicorp/consul/sdk MPL-2.0
- hashicorp/errwrap MPL-2.0
- hashicorp/go-cleanhttp MPL-2.0
- hashicorp/go-immutable-radix MPL-2.0
- hashicorp/go-msgpack MIT
- hashicorp/go-multierror MPL-2.0
- hashicorp/go-rootcerts MPL-2.0
- hashicorp/go-sockaddr MPL-2.0
- hashicorp/go-syslog MIT
- hashicorp/go-uuid MPL-2.0
- hashicorp/go.net BSD-3
- hashicorp/golang-lru MPL-2.0
- hashicorp/hcl MPL-2.0
- hashicorp/logutils MPL-2.0
- hashicorp/mdns MIT
- hashicorp/memberlist MPL-2.0
- hashicorp/serf MPL-2.0
Go agent @mrproliu
- hashicorp/consul/api MPL-2.0
- hashicorp/consul/sdk MPL-2.0
- hashicorp/errwrap MPL-2.0
- hashicorp/go-cleanhttp MPL-2.0
- hashicorp/go-hclog MIT
- hashicorp/go-immutable-radix MPL-2.0
- hashicorp/go-kms-wrapping/entropy MPL-2.0
- hashicorp/go-kms-wrapping/entropy/v2 MPL-2.0
- hashicorp/go-msgpack MIT
- hashicorp/go-multierror MPL-2.0
- hashicorp/go-plugin MPL-2.0
- hashicorp/go-retryablehttp MPL-2.0
- hashicorp/go-rootcerts MPL-2.0
- hashicorp/go-secure-stdlib/base62 MPL-2.0
- hashicorp/go-secure-stdlib/mlock MPL-2.0
- hashicorp/go-secure-stdlib/parseutil MPL-2.0
- hashicorp/go-secure-stdlib/password MPL-2.0
- hashicorp/go-secure-stdlib/tlsutil MPL-2.0
- hashicorp/go-sockaddr MPL-2.0
- hashicorp/go-syslog MIT
- hashicorp/go-uuid MPL-2.0
- hashicorp/go-version MPL-2.0
- hashicorp/go.net BSD-3-Clause
- hashicorp/golang-lru MPL-2.0
- hashicorp/logutils MPL-2.0
- hashicorp/mdns MIT
- hashicorp/memberlist MPL-2.0
- hashicorp/serf MPL-2.0
- hashicorp/vault/api MPL-2.0
- hashicorp/vault/sdk MPL-2.0
- hashicorp/yamux MPL-2.0
SkyWalking eyes @kezhenxu94
- none
SkyWalking Infra e2e @kezhenxu94
- all under swck as transitive dependencies
SkyWalking rover(ebpf agent) @mrproliu
- hashicorp/consul/api MPL-2.0
- hashicorp/consul/sdk MPL-2.0
- hashicorp/errwrap MPL-2.0
- hashicorp/go-cleanhttp MPL-2.0
- hashicorp/go-hclog MIT
- hashicorp/go-immutable-radix MPL-2.0
- hashicorp/go-msgpack MIT
- hashicorp/go-multierror MPL-2.0
- hashicorp/go-retryablehttp MPL-2.0
- hashicorp/go-rootcerts MPL-2.0
- hashicorp/go-sockaddr MPL-2.0
- hashicorp/go-syslog MIT
- hashicorp/go-uuid MPL-2.0
- hashicorp/golang-lru MPL-2.0
- hashicorp/hcl MPL-2.0
- hashicorp/logutils MPL-2.0
- hashicorp/mdns MIT
- hashicorp/memberlist MPL-2.0
- hashicorp/serf MPL-2.0
SkyWalking satellite @mrproliu
- hashicorp/consul/api MPL-2.0
- hashicorp/consul/sdk MPL-2.0
- hashicorp/errwrap MPL-2.0
- hashicorp/go-cleanhttp MPL-2.0
- hashicorp/go-immutable-radix MPL-2.0
- hashicorp/go-msgpack MIT
- hashicorp/go-multierror MPL-2.0
- hashicorp/go-rootcerts MPL-2.0
- hashicorp/go-sockaddr MPL-2.0
- hashicorp/go-syslog MIT
- hashicorp/go-uuid MPL-2.0
- hashicorp/go.net BSD-3-Clause
- hashicorp/golang-lru MPL-2.0
- hashicorp/hcl MPL-2.0
- hashicorp/logutils MPL-2.0
- hashicorp/mdns MIT
- hashicorp/memberlist MPL-2.0
- hashicorp/serf MPL-2.0
SkyWalking Terraform (scripts) @kezhenxu94
- No HashiCorp Dependency
- The scripts for Terraform users only. No hard requirement.

The GitHub ID is listed about the PMC members did the evaluations.

FAQ

If I am using Consul to manage SkyWalking Cluster or configurations, does this license change bring an implication?

YES, anyone using their server sides would be affected once you upgrade to later released versions after Aug. 10th, 2023.

This is HashiCorp’s statement

End users can continue to copy, modify, and redistribute the code for all non-commercial and commercial use, except where providing a competitive offering to HashiCorp. Partners can continue to build integrations for our joint customers. We will continue to work closely with the cloud service providers to ensure deep support for our mutual technologies. Customers of enterprise and cloud-managed HashiCorp products will see no change as well. Vendors who provide competitive services built on our community products will no longer be able to incorporate future releases, bug fixes, or security patches contributed to our products.

So, notice that, the implication about whether voilating BSL 1.1 is determined by the HashiCorp Inc about the status of the identified competitive relationship. We can’t provide any suggestions. Please refer to FAQs and contacts for the official explanations.

Will SkyWalking continoue to use HashiCorp Consul as an optional cluster coordinator and/or an optional dynamic configuration server?

For short term, YES, we will keep that part of codes, as the licenses of the SDK and the APIs are still in the MPL 2.0.

But, during the evaluation, we noticed the consul client we are using is rickfast/consul-client which had been archived by the owner on Jul 27, 2023. So, we are facing the issues that no maintaining and no version to upgrade. If there is not a new consul Java client lib available, we may have to remove this to avoid CVEs or version incompatible with new released servers.

Blog: Celebrate 22k stars

Tue, 13 Jun 2023 00:00:00 +0000

Celebrating 22k Stars!

The Apache SkyWalking community is thrilled to reach the milestone of 22k stars on GitHub! This showcases its popularity and impact as an APM and observability tool.

Since launching in 2016 to provide an open source APM solution, SkyWalking has evolved into a full stack observability platform with distributed tracing, metrics monitoring and alerting. It’s seeing widespread adoption globally, especially in Asia where APM needs are expanding rapidly.

The growing user base has enabled SkyWalking to achieve massive deployments demonstrating its ability to scale to extreme levels. There have been reported deployments collecting over 100TB of data from companies’ complex distributed applications, monitoring over 8000 microservices and analyzing 100 billion distributed traces - providing end-to-end visibility, performance monitoring and issue troubleshooting for some of the largest distributed systems in the world.

This success and widespread adoption has attracted an active community of nearly 800 contributors, thanks in part to programs like GSoC and OSPP(Open Source Promotion Plan) that bring in university contributors. The SkyWalking team remains focused on building a reliable, performant platform to observe complex distributed systems. We’ll continue innovating with features like service mesh monitoring and metric analytics.Your ongoing support, feedback and contributions inspire us!

Thank you for helping SkyWalking reach 22k stars on GitHub! This is just the beginning - we have ambitious plans and can’t wait to have you along our journey!

Blog: Monitoring DynamoDB with SkyWalking

Mon, 13 Mar 2023 00:00:00 +0000

Background

Apache SkyWalking is an open-source application performance management system that helps users collect and aggregate logs, traces, metrics, and events, and display them on the UI. Starting from OAP 9.4.0, SkyWalking has added AWS Firehose receiver, which is used to receive and calculate the data of CloudWatch metrics. In this article, we will take DynamoDB as an example to show how to use SkyWalking to receive and calculate CloudWatch metrics data for monitoring Amazon Web Services.

What are Amazon CloudWatch and Amazon Kinesis Data Firehose？

Amazon CloudWatch is a metrics repository, this tool can collect raw data from AWS (e.g. DynamoDB) and process it into readable metrics in near real-time. Also, we can use Metric Stream to continuously stream CloudWatch metrics to a selected target location for near real-time delivery and low latency. SkyWalking takes advantage of this feature to create metric streams and direct them to Amazon Kinesis Data Firehose transport streams for further transport processing.

Amazon Kinesis Data Firehoseis an extract, transform, and load (ETL) service that reliably captures, transforms, and delivers streaming data to data lakes, data stores, and analytics services. SkyWalking takes advantage of this feature to eventually direct the metrics stream to the aws-firehose-receiver for OAP to calculate and ultimately display the metrics.

The flow chart is as follows.

Notice

Due to Kinesis Data Firehose specifications, the URL of the HTTP endpoint must use the HTTPS protocol and must use port 443. Also, this URL must be proxied by Gateway and forwarded to the real aws-firehose-receiver.
The TLS certificate must be signed by a CA and the self-signed certificate will not be trusted by Kinesis Data Firehose.

Setting up DynamoDB monitoring

Next, let’s take DynamoDB as an example to illustrate the necessary settings in aws before using OAP to collect CloudWatch metrics:

Go to Kinesis Console, create a data stream, and select Direct PUT for Source and HTTP Endpoint for Destination. And set HTTP Endpoint URL to Gateway URL. The rest of the configuration options can be configured as needed.

Go to the CloudWatch Console, select Metrics-Stream in the left control panel, and click Create metric stream. Select AWS/DynamoDB for namespace. Also, you can add other namespaces as needed. Kinesis Data Firehose selects the data stream created in the first step. Finally, set the output format to opentelemetry0.7. The rest of the configuration options can be configured as needed.

At this point, the AWS side of DynamoDB monitoring configuration is set up.

SkyWalking OAP metrics processing analysis

SkyWalking uses aws-firehose-receiver to receive and decode AWS metrics streams forwarded by Gateway, and send it to Opentelemetry-receiver for processing and transforming into SkyWalking metrics. Then, the metrics are analyzed and aggregated by Meter Analysis Language (MAL) and finally presented on the UI.

The MAL part and the UI part of SkyWalking support users’ customization, to display the metrics data in a more diversified way. For details, please refer to MAL doc and UI doc.

Typical metrics analysis

Scope

In SkyWalking, there is the concept of scope. By using scopes, we can classify and aggregate metrics more rationally. In the monitoring of DynamoDB, two of these scopes are used - Service and Endpoint.

Service represents a set of workloads that provide the same behavior for incoming requests. Commonly used as cluster-level scopes for services, user accounts are closer to the concept of clusters in AWS. So SkyWalking uses AWS account id as a key to map AWS accounts to Service types.

Similarly, Endpoint represents a logical concept, often used in services for the path of incoming requests, such as HTTP URI path or gRPC service class + method signature, and can also represent the table structure in the database. So SkyWalking maps DynamoDB tables to Endpoint type.

Metrics

Metric Name	Meaning
AccountMaxReads / AccountMaxWrites	The maximum number of read/write capacity units that can be used by an account.
AccountMaxTableLevelReads / AccountMaxTableLevelWrites	The maximum number of read/write capacity units that can be used by a table or global secondary index of an account.
AccountProvisionedReadCapacityUtilization / AccountProvisionedWriteCapacityUtilization	The percentage of provisioned read/write capacity units utilized by an account.
MaxProvisionedTableReadCapacityUtilization / MaxProvisionedTableWriteCapacityUtilization	The percentage of provisioned read/write capacity utilized by the highest provisioned read table or global secondary index of an account.

Above are some common account metrics (Serivce scope). They are various configuration information in DynamoDB, and SkyWalking can show a complete picture of the database configuration changes by monitoring these metrics.

Metric Name	Meaning
ConsumedReadCapacityUnits / ConsumedWriteCapacityUnits	The number of read/write capacity units consumed over the specified time period.
ReturnedItemCount	The number of items returned by Query, Scan or ExecuteStatement (select) operations during the specified time period.
SuccessfulRequestLatency	The latency of successful requests to DynamoDB or Amazon DynamoDB Streams during the specified time period.
TimeToLiveDeletedItemCount	The number of items deleted by Time to Live (TTL) during the specified time period.

The above are some common table metrics (Endpoint scope), which will also be aggregated into account metrics. These metrics are generally used to analyze the performance of the database, and users can use them to determine the reasonable level of database configuration. For example, users can track how much of their provisioned throughput is used through ConsumedReadCapicityUnits / ConsumedReadCapicityUnits to determine the reasonableness of the preconfigured throughput of a table or account. For more information about provisioned throughput, see Provisioned Throughput Intro.

Metric Name	Meaning
UserErrors	Requests to DynamoDB or Amazon DynamoDB Streams that generate an HTTP 400 status code during the specified time period.
SystemErrors	The requests to DynamoDB or Amazon DynamoDB Streams that generate an HTTP 500 status code during the specified time period.
ThrottledRequests	Requests to DynamoDB that exceed the provisioned throughput limits on a resource.
TransactionConflict	Rejected item-level requests due to transactional conflicts between concurrent requests on the same items.

The above are some common error metrics, among which UserErrors are account-level metrics and the rest are table-level metrics. Users can set alarms on these metrics, and if warnings appear, then it may indicate that there are some problems with the use of the database, and users need to check and verify by themselves.

Notice

SkyWalking’s metrics selection for DynamoDB comes directly from CloudWatch metrics, which can also be found at CloudWatch metrics doc to get metrics details.

Demo

In this section, we will demonstrate how to use terraform to create a DynamoDB table and other AWS services that can generate metrics streams, and deploy Skywalking to complete the metrics collection.

First, you need a running gateway instance, such as NGINX, which is responsible for receiving metrics streams from AWS and forwarding them to the aws-firehose-receiver. Note that the gateway needs to be configured with certificates to accept HTTPS protocol requests.

Below is an example configuration for NGINX. The configuration does not need to be identical, as long as it can send incoming HTTPS requests to oap host:12801/aws/firehose/metrics.

server {
        listen       443 ssl;
        
        ssl_certificate     /crt/test.pem;
        ssl_certificate_key  /crt/test.key; 
        
        ssl_session_timeout  5m;   
        ssl_ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE:ECDH:AES:HIGH:!NULL:!aNULL:!MD5:!ADH:!RC4; 
        ssl_protocols TLSv1 TLSv1.1 TLSv1.2;  
        ssl_prefer_server_ciphers on; 
    
        location /aws/firehose/metrics {
            proxy_pass http://test.xyz:12801/aws/firehose/metrics;
        }
    }

Deploying SkyWalking

There are various ways to deploy SkyWalking, and you can get them directly from the release page.

Of course, if you are more comfortable with Kubernetes, you can also find the appropriate deployment method from SkyWalking-kubernetes.

Please note that no matter which deployment method you use, please make sure that the OAP and UI version is 9.4.0 or higher and that port 12801 needs to be open.

The following is an example of a deployment using the helm command.

export SKYWALKING_RELEASE_VERSION=4.3.0 
export SKYWALKING_RELEASE_NAME=skywalking  
export SKYWALKING_RELEASE_NAMESPACE=default

helm install "${SKYWALKING_RELEASE_NAME}" \
  oci://registry-1.docker.io/apache/skywalking-helm \
  --version "${SKYWALKING_RELEASE_VERSION}" \
  -n "${SKYWALKING_RELEASE_NAMESPACE}" \
  --set oap.image.tag=9.4.0 \
  --set oap.storageType=elasticsearch \
  --set ui.image.tag=9.4.0 \
  --set oap.ports.firehose=12801

Start the corresponding AWS service

The terraform configuration file is as follows (example modified inTerraform Registry - kinesis_firehose_delivery_stream）：

terraform configuration file

provider "aws" {
  region = "ap-northeast-1"
  access_key = "[need change]your access_key"
  secret_key = "[need change]your secret_key"
}
resource "aws_dynamodb_table" "basic-dynamodb-table" {
  name           = "GameScores"
  billing_mode   = "PROVISIONED"
  read_capacity  = 20
  write_capacity = 20
  hash_key       = "UserId"
  range_key      = "GameTitle"

  attribute {
    name = "UserId"
    type = "S"
  }

  attribute {
    name = "GameTitle"
    type = "S"
  }

  attribute {
    name = "TopScore"
    type = "N"
  }

  ttl {
    attribute_name = "TimeToExist"
    enabled        = true
  }

  global_secondary_index {
    name               = "GameTitleIndex"
    hash_key           = "GameTitle"
    range_key          = "TopScore"
    write_capacity     = 10
    read_capacity      = 10
    projection_type    = "INCLUDE"
    non_key_attributes = ["UserId"]
  }

  tags = {
    Name        = "dynamodb-table-1"
    Environment = "production"
  }
}

resource "aws_cloudwatch_metric_stream" "main" {
  name          = "my-metric-stream"
  role_arn      = aws_iam_role.metric_stream_to_firehose.arn
  firehose_arn  = aws_kinesis_firehose_delivery_stream.http_stream.arn
  output_format = "opentelemetry0.7"

  include_filter {
    namespace = "AWS/DynamoDB"
  }
}

# https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-metric-streams-trustpolicy.html
data "aws_iam_policy_document" "streams_assume_role" {
  statement {
    effect = "Allow"

    principals {
      type        = "Service"
      identifiers = ["streams.metrics.cloudwatch.amazonaws.com"]
    }

    actions = ["sts:AssumeRole"]
  }
}

resource "aws_iam_role" "metric_stream_to_firehose" {
  name               = "metric_stream_to_firehose_role"
  assume_role_policy = data.aws_iam_policy_document.streams_assume_role.json
}

# https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-metric-streams-trustpolicy.html
data "aws_iam_policy_document" "metric_stream_to_firehose" {
  statement {
    effect = "Allow"

    actions = [
      "firehose:PutRecord",
      "firehose:PutRecordBatch",
    ]

    resources = [aws_kinesis_firehose_delivery_stream.http_stream.arn]
  }
}
resource "aws_iam_role_policy" "metric_stream_to_firehose" {
  name   = "default"
  role   = aws_iam_role.metric_stream_to_firehose.id
  policy = data.aws_iam_policy_document.metric_stream_to_firehose.json
}

resource "aws_s3_bucket" "bucket" {
  bucket = "metric-stream-test-bucket"
}

resource "aws_s3_bucket_acl" "bucket_acl" {
  bucket = aws_s3_bucket.bucket.id
  acl    = "private"
}

data "aws_iam_policy_document" "firehose_assume_role" {
  statement {
    effect = "Allow"

    principals {
      type        = "Service"
      identifiers = ["firehose.amazonaws.com"]
    }

    actions = ["sts:AssumeRole"]
  }
}

resource "aws_iam_role" "firehose_to_s3" {
  assume_role_policy = data.aws_iam_policy_document.firehose_assume_role.json
}

data "aws_iam_policy_document" "firehose_to_s3" {
  statement {
    effect = "Allow"

    actions = [
      "s3:AbortMultipartUpload",
      "s3:GetBucketLocation",
      "s3:GetObject",
      "s3:ListBucket",
      "s3:ListBucketMultipartUploads",
      "s3:PutObject",
    ]

    resources = [
      aws_s3_bucket.bucket.arn,
      "${aws_s3_bucket.bucket.arn}/*",
    ]
  }
}

resource "aws_iam_role_policy" "firehose_to_s3" {
  name   = "default"
  role   = aws_iam_role.firehose_to_s3.id
  policy = data.aws_iam_policy_document.firehose_to_s3.json
}

resource "aws_kinesis_firehose_delivery_stream" "http_stream" {
  name        = "metric-stream-test-stream"
  destination = "http_endpoint"

  http_endpoint_configuration {
    name       = "test_http_endpoint"
    url        = "[need change]Gateway url"
    role_arn   = aws_iam_role.firehose_to_s3.arn
  }
  s3_configuration {
    role_arn   = aws_iam_role.firehose_to_s3.arn
    bucket_arn = aws_s3_bucket.bucket.arn
  }
}

Steps to use.

Get the access_key and secret_key of the AWS account.( For how to get them, please refer to create-access-key )
Fill in the access_key and secret_key you got in the previous step, and fill in the corresponding URL of your gateway in the corresponding location of aws_kinesis_firehose_delivery_stream configuration.
Copy the above content and save it to the main.tf file.
Execute the following code in the corresponding path.

terraform init
terraform apply

At this point, all the required AWS services have been successfully created, and you can check your console to see if the services were successfully created.

Done!

If all the above steps were successful, please wait for about five minutes. After that, you can visit the SkyWalking UI to see the metrics.

Currently, the metrics collected by SkyWalking by default are displayed as follows.

account metrics:

table metrics：

Other services

Currently, SkyWalking officially supports EKS, S3, DynamoDB monitoring. Users also refer to the OpenTelemetry receiver to configure OTel rules to collect and analyze CloudWatch metrics of other AWS services and display them through a custom dashboard.

Material

Blog: Monitoring AWS EKS and S3 with SkyWalking

Sun, 12 Mar 2023 00:00:00 +0000

SKyWalking OAP’s existing OpenTelemetry receiver can receive metrics through the OTLP protocol, and use MAL to analyze related metrics in real time. Starting from OAP 9.4.0, SkyWalking has added an AWS Firehose receiver to receive and analyze CloudWatch metrics data. This article will take EKS and S3 as examples to introduce the process of SkyWalking OAP receiving and analyzing the indicator data of AWS services.

EKS

OpenTelemetry Collector

OpenTelemetry (OTel) is a series of tools, APIs, and SDKs that can generate, collect, and export telemetry data, such as metrics, logs, and traces. OTel Collector is mainly responsible for collecting, processing, and exporting. For telemetry data, Collector consists of the following main components:

Receiver: Responsible for obtaining telemetry data, different receivers support different data sources, such as prometheus, kafka, otlp.
Processor: Process data between receiver and exporter, such as adding or deleting attributes.
Exporter: Responsible for sending data to different backends, such as kafka, SkyWalking OAP (via OTLP).
Service: Components enabled as a unit configuration, only configured components will be enabled.

OpenTelemetry Protocol Specification(OTLP)

OTLP mainly describes how to receive (pull) indicator data through gRPC and HTTP protocols. The OpenTelemetry receiver of SKyWalking OAP implements the OTLP/gRPC protocol, and the indicator data can be exported to OAP through the OTLP/gRPC exporter. Usually the data flow of a Collector is as follows:

Monitor EKS with OTel

EKS monitoring is realized through OTel. You only need to deploy OpenTelemetry Collector in the EKS cluster in the way of DaemonSet – use AWS Container Insights Receiver as the receiver, and set the address of otlp exporter to the address of OAP. In addition, it should be noted that OAP is used job_name : aws-cloud-eks-monitoring as the identifier of EKS metrics according to the attribute, so it is necessary to configure a processor in the collector to add this attribute.

OTel Collector configuration demo


extensions:
  health_check:
receivers:
  awscontainerinsightreceiver:
processors:
# To enable OAP to correctly identify EKS metrics, add the job_name attribute
  resource/job-name:
    attributes:
    - key: job_name   
      value: aws-cloud-eks-monitoring
      action: insert     

# Specify OAP as exporters
exporters:
  otlp:
    endpoint: oap-service:11800 
    tls:
      insecure: true
  logging:
      loglevel: debug          
service:
  pipelines:
    metrics:
      receivers: [awscontainerinsightreceiver]
      processors: [resource/job-name]
      exporters: [otlp,logging]
  extensions: [health_check]

By default, SkyWalking OAP counts the network, disk, CPU and other related indicator data in the three dimensions of Node, Pod, and Service. Only part of the content is shown here.

Pod dimensions

Service dimensions

EKS monitoring complete configuration

Click here to view complete k8s resource configuration


apiVersion: v1
kind: ServiceAccount
metadata:
  name: aws-otel-sa
  namespace: aws-otel-eks

---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: aoc-agent-role
rules:
  - apiGroups: [""]
    resources: ["pods", "nodes", "endpoints"]
    verbs: ["list", "watch"]
  - apiGroups: ["apps"]
    resources: ["replicasets"]
    verbs: ["list", "watch"]
  - apiGroups: ["batch"]
    resources: ["jobs"]
    verbs: ["list", "watch"]
  - apiGroups: [""]
    resources: ["nodes/proxy"]
    verbs: ["get"]
  - apiGroups: [""]
    resources: ["nodes/stats", "configmaps", "events"]
    verbs: ["create", "get"]
  - apiGroups: [""]
    resources: ["configmaps"]
    resourceNames: ["otel-container-insight-clusterleader"]
    verbs: ["get","update"]
  - apiGroups: ["coordination.k8s.io"]
    resources: ["leases"]
    verbs: ["create","get","update"]    

---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: aoc-agent-role-binding
subjects:
  - kind: ServiceAccount
    name: aws-otel-sa
    namespace: aws-otel-eks
roleRef:
  kind: ClusterRole
  name: aoc-agent-role
  apiGroup: rbac.authorization.k8s.io

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: otel-agent-conf
  namespace: aws-otel-eks
  labels:
    app: opentelemetry
    component: otel-agent-conf
data:
  otel-agent-config: |
    extensions:
      health_check:

    receivers:
      awscontainerinsightreceiver:

    processors:
      resource/job-name:
        attributes:
        - key: job_name   
          value: aws-cloud-eks-monitoring
          action: insert     

    exporters:
      otlp:
        endpoint: oap-service:11800
        tls:
          insecure: true
      logging:
          loglevel: debug          

    service:
      pipelines:
        metrics:
          receivers: [awscontainerinsightreceiver]
          processors: [resource/job-name]
          exporters: [otlp,logging]
      extensions: [health_check]

---

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: aws-otel-eks-ci
  namespace: aws-otel-eks
spec:
  selector:
    matchLabels:
      name: aws-otel-eks-ci
  template:
    metadata:
      labels:
        name: aws-otel-eks-ci
    spec:
      containers:
        - name: aws-otel-collector
          image: amazon/aws-otel-collector:v0.23.0
          env:
         	  # Specify region
            - name: AWS_REGION
              value: "ap-northeast-1"
            - name: K8S_NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
            - name: HOST_IP
              valueFrom:
                fieldRef:
                  fieldPath: status.hostIP
            - name: HOST_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
            - name: K8S_NAMESPACE
              valueFrom:
                 fieldRef:
                   fieldPath: metadata.namespace
          imagePullPolicy: Always
          command:
            - "/awscollector"
            - "--config=/conf/otel-agent-config.yaml"
          volumeMounts:
            - name: rootfs
              mountPath: /rootfs
              readOnly: true
            - name: dockersock
              mountPath: /var/run/docker.sock
              readOnly: true
            - name: varlibdocker
              mountPath: /var/lib/docker
              readOnly: true
            - name: containerdsock
              mountPath: /run/containerd/containerd.sock
              readOnly: true
            - name: sys
              mountPath: /sys
              readOnly: true
            - name: devdisk
              mountPath: /dev/disk
              readOnly: true
            - name: otel-agent-config-vol
              mountPath: /conf
            - name: otel-output-vol  
              mountPath: /otel-output
          resources:
            limits:
              cpu:  200m
              memory: 200Mi
            requests:
              cpu: 200m
              memory: 200Mi
      volumes:
        - configMap:
            name: otel-agent-conf
            items:
              - key: otel-agent-config
                path: otel-agent-config.yaml
          name: otel-agent-config-vol
        - name: rootfs
          hostPath:
            path: /
        - name: dockersock
          hostPath:
            path: /var/run/docker.sock
        - name: varlibdocker
          hostPath:
            path: /var/lib/docker
        - name: containerdsock
          hostPath:
            path: /run/containerd/containerd.sock
        - name: sys
          hostPath:
            path: /sys
        - name: devdisk
          hostPath:
            path: /dev/disk/
        - name: otel-output-vol  
          hostPath:
            path: /otel-output
      serviceAccountName: aws-otel-sa

S3

Amazon CloudWatch

Amazon CloudWatch is a monitoring service provided by AWS. It is responsible for collecting indicator data of AWS services and resources. CloudWatch metrics stream is responsible for converting indicator data into stream processing data, and supports output in two formats: json and OTel v0.7.0.

Amazon Kinesis Data Firehose (Firehose)

Firehose is an extract, transform, load (ETL) service that reliably captures, transforms, and serves streaming data into data lakes, data stores (such as S3), and analytics services.

To ensure that external services can correctly receive indicator data, AWS provides Kinesis Data Firehose HTTP Endpoint Delivery Request and Response Specifications (Firehose Specifications) . Firhose pushes Json data by POST

Json data example

{
  "requestId": "ed4acda5-034f-9f42-bba1-f29aea6d7d8f",
  "timestamp": 1578090901599
  "records": [
    {
      "data": "aGVsbG8="
    },
    {
      "data": "aGVsbG8gd29ybGQ="
    }
  ]
}

requestId: Request id, which can achieve deduplication and debugging purposes.
timestamp: Firehose generated the timestamp of the request (in milliseconds).
records: Actual delivery records
1. data: The delivered data, encoded in base64, can be in json or OTel v0.7.0 format, depending on the format of CloudWatch data (described later). Skywalking currently supports OTel v0.7.0 format.

aws-firehose-receiver

aws-firehose-receiver provides an HTTP Endpoint that implements Firehose Specifications: /aws/firehose/metrics. The figure below shows the data flow of monitoring DynamoDB, S3 and other services through CloudWatch, and using Firehose to send indicator data to SKywalking OAP.

Step-by-step setup of S3 monitoring

Enter the S3 console and create a filter forRequest metrics: Amazon S3 >> Buckets >> (Your Bucket) >> Metrics >> metrics >> View additional charts >> Request metrics
Enter the Amazon Kinesis console, create a delivery stream, Source select Direct PUT, Destination select HTTP Endpoint. And set HTTP endpoint URL to https://your_domain/aws/firehose/metrics. Other configuration items:

Buffer hints: Set the size and period of the cache
Access key just matches the AccessKey in aws-firehose-receiver
Retry duration: Retry period
Backup settings: Backup settings, optionally backup the posted data to S3 at the same time.

Enter the CloudWatch console Streams and click Create CloudWatch Stream. And Select your Kinesis Data Firehose stream configure the delivery stream created in the second step in the item. Note that it needs to be set Change output format to OpenTelemetry v0.7.0.

At this point, the S3 monitoring configuration settings are complete. The S3 metrics currently collected by SkyWalking by default are shown below:

Other service

Currently SkyWalking officially supports EKS, S3, DynamoDB monitoring. Users also refer to the OpenTelemetry receiver to configure OTel rules to collect and analyze CloudWatch metrics of other AWS services, and display them through a custom dashboard.

Material

Blog: [License Issue] Aliyun(阿里云)'s trace analysis service copied SkyWalking's trace profiling page.

Tue, 03 Jan 2023 00:00:00 +0000

Apache SkyWalking is an open-source APM for a distributed system, Apache Software Foundation top-level project.

On Jan. 3rd, 2023, we received reports about Aliyun Trace Analysis Service. It provides a cloud service compatible with SkyWalking trace APIs and agents.

On their product page, there is a best-practice document describing about their service is not SkyWalking OAP, but can work with SkyWalking agents to support SkyWalking’s In-Process(Trace) Profiling.

BUT, they copied the whole page of SkyWalking’s profiling UI, including page layout, words, and profiling task setup. The only difference is the color schemes.

SkyWalking UI

Aliyun Trace Analysis UI on their document page

The UI visualization is a part of the copyright. Aliyun declared their backend is NOT a re-distribution of SkyWalking repeatedly on their website, and they never mentioned this page is actually copied from upstream.

This is a LICENSE issue, violating SkyWalking’s copyright and Apache 2.0 License. They don’t respect Apache Software Foundation and Apache SkyWalking’s IP and Branding.

Blog: [Video] Distributed tracing demo using Apache SkyWalking and Kong API Gateway

Thu, 11 Aug 2022 00:00:00 +0000

Observability essential when working with distributed systems. Built on 3 pillars of metrics, logging and tracing, having the right tools in place to quickly identify and determine the root cause of an issue in production is imperative. In this Kongcast interview, we explore the benefits of having observability and demo the use of Apache SkyWalking. We walk through the capabilities that SkyWalking offers out of the box and debug a common HTTP 500 error using the tool.

Andrew Kew is interviewed by Viktor Gamov, a developer advocate at Kong Inc

Andrew is a highly passionate technologist with over 16 valuable years experience in building server side and cloud applications. Having spent the majority of his time in the Financial Services domain, his meritocratic rise to CTO of an Algorithmic Trading firm allowed him to not only steer the business from a technology standpoint, but build robust and scalable trading algorithms. His mantra is “right first time”, thus ensuring the projects or clients he is involved in are left in a better place than they were before he arrived.

He is the founder of a boutique software consultancy in the United Kingdom, QuadCorps Ltd, working in the API and Integration Ecosystem space and is currently on a residency programme at Kong Inc as a senior field engineer and technical account manager working across many of their enterprise strategic accounts.

Blog: Integrating Apache SkyWalking with source code

Thu, 14 Apr 2022 00:00:00 +0000

Introduction

The most profound technologies are those that disappear. They weave themselves into the fabric of everyday life until they are indistinguishable from it. - Mark Weiser

Mark Weiser prophetically argued in the late 1980s, that the most far-reaching technologies are those which vanish into thin air. According to Weiser, “Whenever people learn something sufficiently well, they cease to be aware of it.” This disappearing act, as Weiser claimed, is not limited to technology but rather human psychology. It is this very experience that allows us to escape lower-level thinking into higher-level thinking. For once we are no longer impeded by mundane details, we are then free to focus on new goals.

This realization becomes more relevant as APMs become increasingly popular. As more applications are deployed with APMs, the number of abstract representations of the underlying source code also increases. While this provides great value to many non-development roles within an organization, it does pose additional challenges to those in development roles who must translate these representations into concepts they can work with (i.e. source code). Weiser sums this difficultly up rather succinctly when he states that “Programmers should no more be asked to work without access to source code than auto-mechanics should be asked to work without looking at the engine.”

Still, APMs collect more information only to produce a plethora of new abstract representations. In this article, we will introduce a new concept in Source++, the open-source live-coding platform, specifically designed to allow developers to monitor production applications more intuitively.

Live Views

And we really don’t understand even yet, hundreds of metrics later, what make a program easier to understand or modify or reuse or borrow. I don’t think we’ll find out by looking away from programs to their abstract interfaces. The answers are in the source code. - Mark Weiser

As APMs move from the “nice to have” category to the “must-have” category, there is a fundamental feature holding them back from ubiquity. They must disappear from consciousness. As developers, we should feel no impulse to open our browsers to better understand the underlying source code. The answers are literally in the source code. Instead, we should improve our tools so the source code conveniently tells us what we need to know. Think of how simple life could be if failing code always indicated how and why it failed. This is the idea behind Source++.

In our last blog post, we discussed Extending Apache SkyWalking with non-breaking breakpoints. In that post, we introduced a concept called Live Instruments, which developers can use to easily debug live production applications without leaving their IDE. Today, we will discuss how existing SkyWalking installations can be integrated into your IDE via a new concept called Live Views. Unlike Live Instruments, which are designed for debugging live applications, Live Views are designed for increasing application comprehension and awareness. This is accomplished through a variety of commands which are input into the Live Command Palette.

Live Command Palette

The Live Command Palette (LCP) is a contextual command prompt, included in the Source++ JetBrains Plugin, that allows developers to control and query live applications from their IDE. Opened via keyboard shortcut (Ctrl+Shift+S), the LCP allows developers to easily view metrics relevant to the source code they’re currently viewing. The following Live View commands are currently supported:

Command: view (overview/activity/traces/logs)

The view commands display contextual popups with live operational data of the current source code. These commands allow developers to view traditional SkyWalking operational data filtered down to the relevant metrics.

Command: watch log

The watch log command allows developers to follow individual log statements of a running application in real-time. This command allows developers to negate the need for manually scrolling through the logs to find instances of a specific log statement.

Command: (show/hide) quick stats

The show quick stats command displays live endpoint metrics for a quick idea of an endpoint’s activity. Using this command, developers can quickly assess the status of an endpoint and determine if the endpoint is performing as expected.

Future Work

A good tool is an invisible tool. By invisible, I mean that the tool does not intrude on your consciousness; you focus on the task, not the tool. Eyeglasses are a good tool – you look at the world, not the eyeglasses. - Mark Weiser

Source++ aims to extend SkyWalking in such a way that SkyWalking itself becomes invisible. To accomplish this, we plan to support custom developer commands. Developers will be able to build customized commands for themselves, as well as commands to share with their team. These commands will recognize context, types, and conditions allowing for a wide possibility of operations. As more commands are added, developers will be able to expose everything SkyWalking has to offer while focusing on what matters most, the source code.

If you find these features useful, please consider giving Source++ a try. You can install the plugin directly from your JetBrains IDE, or through the JetBrains Marketplace. If you have any issues or questions, please open an issue. Feedback is always welcome!

Blog: [Resolved][License Issue] Volcengine Inc.(火山引擎) violates the Apache 2.0 License when using SkyWalking.

Fri, 28 Jan 2022 00:00:00 +0000

Apache SkyWalking is an open-source APM for a distributed system, Apache Software Foundation top-level project.

On Jan. 28th, we received a License violation report from one of the committers (anonymously). They have a cloud service called Application Performance Monitoring - Distributed Tracing (应用性能监控全链路版). At the Java service monitoring section, it provides this agent download link

wget https://datarangers.com.cn/apminsight/repo/v2/download/java-agent/apminsight-java-agent_latest.tar.gz

We downloaded it at 23:15 Jan. 28th UTC+8(Beijing), and archived it at here

We have confirmed this is a distribution of SkyWalking Java agent.

We listed several pieces of evidence to prove this here, every reader could compare with the official SkyWalking source codes

The first and the easiest one is agent.config file, which is using the same config keys, and the same config format.

This is the Volcengine’s version, and check SkyWalking agent.config

In the apmplus-agent.jar, Volcengine’s agent core jar, you could easily find several core classes exactly as same as SkyWalking’s.

The ComponentsDefine class is unchanged, even with component ID and name. This is Volcengine’s version, and check SkyWalking’s version

The whole code names, package names, and hierarchy structure are all as same as SkyWalking 6.x version.

This is the Volcengine package hierarchy structure, and check the SkyWalking’s version

Volcengine Inc.’s team changed all package names, removed the Apache Software Foundation’s header, and don’t keep Apache Software Foundation and Apache SkyWalking’s LICENSE and NOTICE file in their redistribution.

Also, we can’t find anything on their website to declare they are distributing SkyWalking.

All above have proved they are violating the Apache 2.0 License, and don’t respect Apache Software Foundation and Apache SkyWalking’s IP and Branding.

We have contacted their legal team, and wait for their official response.

Resolution

On Jan. 30th night, UTC+8, 2022. We received a response from Volcengine’s APMPlus team. They admitted their violation behaviors, and made the following changes.

Volcengine’s APMPlus service page was updated on January 30th and stated that the agent is a fork version(re-distribution) of Apache SkyWalking agent. Below is the screenshot of Volcengine’s APMPlus product page.

Volcengine’s APMPlus agent distributions were also updated and include SkyWalking’s License and NOTICE now. Below is the screenshot of Volcengine’s APMPlus latest agent, you could download from the product page. We keep a copy of their Jan. 30th 2022 at here.

Volcengine’s APMPlus team had restored all license headers of SkyWalking in the agent, and the modifications of the project files are also listed in “SkyWalking-NOTICE”, which you could download from the product page.

We have updated the status to the PMC mail list. This license violation issue has been resolved for now.

Appendix

Inquiries of committers

Q: I hope Volcengine Inc. can give a reason for this license issue, not just an afterthought PR. This will not only let us know where the issue is but also avoid similar problems in the future.

A(apmplus apmplus@volcengine.com):

The developers neglected this repository during submitting compliance
assessment.
Currently, APMPlus team had introduced advanced tools provided by the
company for compliance assessment, and we also strengthened training for
our developers.
In the future, the compliance assessment process will be further improved
from tool assessment and manual assessment.

Blog: Extending Apache SkyWalking with non-breaking breakpoints

Mon, 06 Dec 2021 00:00:00 +0000

Non-breaking breakpoints are breakpoints specifically designed for live production environments. With non-breaking breakpoints, reproducing production bugs locally or in staging is conveniently replaced with capturing them directly in production.

Like regular breakpoints, non-breaking breakpoints can be:

placed almost anywhere
added and removed at will
set to fire on specific conditions
expose internal application state
persist as long as desired (even between application reboots)

The last feature is especially useful given non-breaking breakpoints can be left in production for days, weeks, and even months at a time while waiting to capture behavior that happens rarely and unpredictably.

How do non-breaking breakpoints work?

If you’re familiar with general distributed tracing concepts, such as “traces” and “spans”, then you’re already broadly familiar with how non-breaking breakpoints work. Put simply, non-breaking breakpoints are small fragments of code added during runtime that, upon the proper conditions, save a portion of the application’s current state, and resume normal execution. In SkyWalking, this can be implemented by simply opening a new local span, adding some tags, and closing the local span.

While this process is relatively simple, the range of functionality that can be achieved through this technique is quite impressive. Save the current and global variables to create a non-breaking breakpoint; add the ability to format log messages to create just-in-time logging; add the ability to trigger metric telemetry to create real-time KPI monitoring. If you keep moving in this direction, you eventually enter the realm of live debugging/coding, and this is where Source++ comes in.

Live Coding Platform

Source++ is an open-source live coding platform designed for production environments, powered by Apache SkyWalking. Using Source++, developers can add breakpoints, logs, metrics, and distributed tracing to live production software in real-time on-demand, right from their IDE or CLI. While capable of stand-alone deployment, the latest version of Source++ makes it easier than ever to integrate into existing Apache SkyWalking installations. This process can be completed in a few minutes and is easy to customize for your specific needs.

For a better idea of how Source++ works, take a look at the following diagram:

In this diagram, blue components represent existing SkyWalking architecture, black components represent new Source++ architecture, and the red arrows show how non-breaking breakpoints make their way from production to IDEs. A process that is facilitated by Source++ components: Live Probe, Live Processors, Live Platform, and Live Interface.

Live Probe

The Live Probe is currently available for JVM and Python applications. It runs alongside the SkyWalking agent and is responsible for dynamically adding and removing code fragments based on valid instrumentation requests from developers. These code fragments in turn make use of the SkyWalking agent’s internal APIs to facilitate production instrumentation.

Live Processors

Live Processors are responsible for finding, extracting, and transforming data found in distributed traces produced via live probes. They run alongside SkyWalking collectors and implement additional post-processing logic, such as PII redaction. Live processors work via uniquely identifiable tags (prefix spp.) added previously by live probes.

One could easily view a non-breaking breakpoint ready for processing using Rocketbot, however, it will look like this:

Even though the above does not resemble what’s normally thought of as a breakpoint, the necessary information is there. With live processors added to your SkyWalking installation, this data is refined and may be viewed more traditionally via live interfaces.

Live Platform

The Live Platform is the core part of the Source++ architecture. Unlike the live probe and processors, the live platform does not have a direct correlation with SkyWalking components. It is a standalone server responsible for validating and distributing production breakpoints, logs, metrics, and traces. Each component of the Source++ architecture (probes, processors, interfaces) communicates with each other through the live platform. It is important to ensure the live platform is accessible to all of these components.

Live Interface

Finally, with all the previous parts installed, we’re now at the component software developers will find the most useful. A Live Interface is what developers use to create, manage, and view non-breaking breakpoints, and so on. There are a few live interfaces available:

With the Live Instrument Processor enabled, and the JetBrains Plugin installed, non-breaking breakpoints appear as such:

The above should be a sight far more familiar to software developers. Beyond the fact that you can’t step through execution, non-breaking breakpoints look and feel just like regular breakpoints.

For more details and complete setup instructions, please visit:

https://github.com/sourceplusplus/deploy-skywalking

Blog: [Webinar] SkyWalking 8.x Introduction

Wed, 04 Aug 2021 00:00:00 +0000

GOUP hosted a webinar, and invited Sheng Wu to introduce Apache SkyWalking. This is a 1.5 hours presentation including the full landscape of Apache SkyWalking 8.x.

Chapter04 Session10 - Apache Skywalking by Sheng Wu

Blog: [Community win] SkyWalking achieved 500 contributors milestone.

Sat, 10 Jul 2021 00:00:00 +0000

SkyWalking is an open source APM (application performance monitor) system, especially designed for microservices, cloud native, and container-based architectures.

From 2020, it has dominated the open source APM market in China, and expanded aggressively in North American, Europe and Asia’s other countries.

With over 6 years (2015-2021) of development, driven by the global open source community, SkyWalking now provides full stack observability covering metrics, tracing and logging, plus event detector, which are built based on various native and ecosystem solutions.

Language agent-based(Java, Dot Net, Golang, PHP, NodeJS, Python, C++, LUA) in-process monitoring, is as powerful as commercial APM vendors’ agents. Mostly auto-instrumentation, and good interactivity.
Service Mesh Observability, working closely with Envoy and Istio teams.
Transparent integration of popular metrics ecosystem. Accept metrics from Prometheus SDK, OpenTelemetry collectors, Zabbix agents, etc.
Log collection with analysis capability from FluentD, Fluent-bit, Filebeat, etc. agents.
Infrastructure monitoring, such as Linux and k8s, is out of the box.

The SkyWalking ecosystem was started by very few people. The community drives the project to cover real scenarios, from tracing to the whole APM field. Even today, more professional open source developers, powered by the vendors behind them, are bringing the project to a different level.

Typically and most attractively, SkyWalking is going to build the first known open source APM specific database in the world, at least providing

Time series-based database engine.
Support traces/logs and metrics in the database core level.
High performance with cluster mode and HPA.
Reasonable resource cost.

We nearly doubled the number of contributors in the last year, from ~300 to over 500. The whole community is very energetic. Here, we want to thank our 47 committers(28 PMC members included), listed here, and over 400 other contributors.

We together built this humongous Apache Top Level project, and proved the stronge competitiveness of an open-source project.

This is a hard-won and impressive achievement. We won’t stop here. The trend is there, the ground is solid. We are going to build the top-level APM system relying on our open-source community.

500 Contributors List

GitHub
1095071913	182148432**	295198088**	394102339**	437376068**
50168383	55846420**	826245622**	844067874	Ahoo-Wang
AirTrioa	AlexanderWert	AlseinX	AngryMills	Ax1an
BFergerson	BZFYS	CalvinKirs	CharlesMaster	ChaunceyLin5152
CommissarXia	Cvimer	DeadLion	Doublemine	Du-fei
ElderJames	EvanLjp	FatihErdem	FeynmanZhou	Fine0830
FingerLiu	FrankyXu	Gallardot	GerryYuan	HackerRookie
HarryFQ	Heguoya	Hen1ng	HendSame	Humbertzhang
IanCao	IluckySi	Indifer	J-Cod3r	JaredTan95
Jargon96	Jijun	JoeKerouac	JohnNiang	Johor03
Jozdortraz	Jtrust	Just-maple	KangZhiDong	LazyLei
LiWenGu	Lin1997	Linda-pan	LiteSun	Liu-XinYuan
MiracleDx	Miss-you	MoGuGuai-hzr	MrYzys	O-ll-O
Patrick0308	QHWG67	Qiliang	QuanjieDeng	RandyAbernethy
RedzRedz	Runrioter	SataQiu	ScienJus	SevenBlue2018
ShaoHans	Shikugawa	SoberChina	SummerOfServenteen	Switch-vov
TJ666	Technoboy-	TerrellChen	TeslaCN	TheRealHaui
TinyAllen	TomMD	ViberW	Videl	WALL-E
WeihanLi	WildWolfBang	WillemJiang	Wooo0	XhangUeiJong
Xlinlin	YczYanchengzhe	Yebemeto	YoungHu	YunaiV
YunfengGao	Z-Beatles	ZS-Oliver	ZhHong	ZhuoSiChen
a198720	a1vin-tian	a526672351	acurtain	adamni135
adermxzs	adriancole**	aeolusheath	agile6v	aix3
aiyanbo	ajanthan	alexkarezin	alonelaval	amogege
amwyyyy	andyliyuze	andyzzl	aoxls	arugal
ascrutae	ascrutae**	augustowebd	aviaviavi	bai-yang
beckhampu	beckjin	beiwangnull	bigflybrother	bootsrc
bostin	brucewu-fly	buxingzhe	buzuotaxuan	bwh12398**
c feng	c1ay	candyleer	carllhw	carlvine500
carrypann	cheenursn	cheetah012	chenbeitang	chenglei**
chengshiwen	chenmudu	chenpengfei	chenvista	chess-equality
chestarss	chidaodezhongsheng	chopin-d	clevertension	clk1st
cngdkxw	cnlangzi	codeglzhang	codelipenghui	coder-yqj
coki230	compilerduck	constanine	coolbeevip	crystaldust
cui-liqiang	cuiweiwei	cutePanda123	cyberdak	cyejing
cyhii	dafu-wu	dagmom	dalekliuhan**	darcydai
dengliming	devkanro	devon-ye	dickens7	dimaaan
dingdongnigetou	dio	divyakumarjain	dmsolr	dominicqi
donbing007	dsc6636926	dvsv2	dzx2018	echooymxq
efekaptan	elk-g	emschu	eoeac	evanljp**
evanxuhe	feelwing1314	fgksgf	fredster33	fuhuo
fulmicoton	fushiqinghuan111	geektcp	geomonlin	ggndnn
gitter-badger	givingwu	glongzh	gnr163	gonedays
grissom-grissom	grissomsh	guodongq	guyukou	gxthrj
gy09535	gzshilu	hailin0	hanahmily	haotian2015
haoyann	hardzhang	harvies	heihaozi	hepyu
heyanlong	hi-sb	honganan	horber	hsoftxl
huangyoje	huliangdream	huohuanhuan	iluckysi	innerpeacez
itsvse	jasper-zsh	jbampton	jialong121	jinlongwang
jjlu521016	jjtyro	jmjoy	jsbxyyx	justeene
juzhiyuan	jy00464346	kaanid	kagaya85	karott
kayleyang	kevinyyyy	kezhenxu94	kikupotter	kilingzhang
killGC	kkl129	klboke	ksewen	kuaikuai
kun-song	kylixs	landonzeng	langke93	langyan1022
langyizhao	lazycathome	leemove	leizhiyuan	libinglong
lijial	lilien1010	limfriend	linkinshi	linliaoy
liqiangz	liu-junchi	liufei**	liuhaoXD	liuhaoyang
liuweiyi**	liuyanggithup	liuzhengyang	liweiv	lixin40**
lizl9**	lkxiaolou	llissery	louis-zhou	lpcy
lpf32	lsyf	lucperkins	lujiajing1126	lunamagic1978
lunchboxav	lxin96**	lxliuxuankb	lytscu	lyzhang1999
mage3k	makefriend8	makingtime	mantuliu	maolie
margauxcabrera	masterxxo	maxiaoguang64	me**	membphis
mestarshine	mgsheng	michaelsembwever	mikkeschiren	ming_flycash**
minquan.chen**	misaya	momo0313	moonming	mrproliu
mrproliu**	muyun12	nacx	neatlife	neeuq
nic-chen	nickwongwong	nikitap492	nileblack	nisiyong
novayoung	oatiz	oflebbe	olzhy	onecloud360
osiriswd	panniyuyu	peng-yongsheng	pengweiqhca	potiuk
probeyang	purgeyao	qijianbo010	qinhang3	qiuyu-d
qjgszzx	qq362220083	qqeasonchen	qxo	ralphgj
raybi-asus	refactor2	remicollet	rlenferink	rootsongjc
rovast	ruibaby	s00373198	scolia	sdanzo
seifeHu	sergicastro	shiluo34	sikelangya	simonlei
sk163	snakorse	songzhendong	songzhian	songzhian**
sonxy	spacewander	stalary	stenio2011	stevehu
stone-wlg	sungitly	surechen	swartz-k	sxzaihua
tangxqa	tanjunchen	tankilo	tanzhen**	taskmgr
tbdpmi	terranhu	terrymanu	tevahp	thanq
thebouv	tianyk	tianyuak	tincopper	tinyu0
tom-pytel	tristaZero	tristan-tsl	trustin	tsuilouis
tuohai666	tzsword-2020	tzy1316106836	vcjmhg	viktoryi
vision-ken	viswaramamoorthy	wallezhang	wang-yeliang	wang_weihan**
wangrzneu	wankai123	wbpcode	web-xiaxia	webb2019
weiqiang-w	weiqiang333	wendal	wengangJi	wenjianzhang
whfjam	whl12345	willseeyou	wilsonwu	wind2008hxy
wingwong-knh	withlin	wl4g	wqr2016	wu-sheng
wuguangkuo	wujun8	wuwen5	wuxingye	x22x22
xbkaishui	xcaspar	xdRight	xiaoweiyu**	xiaoxiangmoe
xiaoy00	xinfeingxia85	xingren23	xinzhuxiansheng	xonze
xuanyu66	xuchangjunjx	xudianyang	yanbw	yanfch
yang-xiaodong	yangxb2010000	yanickxia	yanmaipian	yanmingbi
yantaowu	yaojingguo	yaowenqiang	yazong	ychandu
ycoe	yimeng	yu199195	yuqichou	yushuqiang**
yuyujulin	yxudong	yymoth	zaunist	zaygrzx
zcai2	zeaposs	zhang98722	zhanghao001	zhangjianweibj
zhangkewei	zhangsean	zhangxin**	zhaoyuguang	zhe1926
zhentaoJin	zhongjianno1**	zhousiliang163	zhuCheer	zhyyu
zifeihan	zijin-m	zkscpqm	zoidbergwill	zoumingzm
zouyx	zpf1989	zshit	zxbu	zygfengyuwuzu

Blog: OpenSearch, a new storage option to avoid ElasticSearch's SSPL

Sun, 09 May 2021 00:00:00 +0000

We posted our Response to Elastic 2021 License Change blog 4 months ago. It doesn’t have a big impact in the short term, but because of the incompatibility between SSPL and Apache 2.0, we lost the chance of upgrading the storage server, which concerns the community and our users. So, we have to keep looking for a new option as a replacement.

There was an open source project, Open Distro for Elasticsearch, maintained by the AWS team. It is an Apache 2.0-licensed distribution of Elasticsearch enhanced with enterprise security, alerting, SQL, and more. After Elastic relicensed its projects, we talked with their team, and they have an agenda to take over the community leadship and keep maintaining Elasticsearch, as it was licensed by Apache 2.0. So, they are good to fork and continue.

On April 12th, 2021, AWS announced the new project, OpenSearch, driven by the community, which is initialized from people of AWS, Red Hat, SAP, Capital One, and Logz.io. Read this Introducing OpenSearch blog for more detail.

Once we had this news in public, we begin to plan the process of evaluating and testing OpenSearch as SkyWalking’s storage option. Read our issue.

Today, we are glad to ANNOUNCE, OpenSearch could replace ElastcSearch as the storage, and it is still licensed under Apache 2.0.

This has been merged in the main stream, and you can find it in the dev doc already.

OpenSearch

OpenSearch storage shares the same configurations as Elasticsearch 7. In order to activate Elasticsearch 7 as storage, set storage provider to elasticsearch7. Please download the apache-skywalking-bin-es7.tar.gz if you want to use OpenSearch as storage.

SkyWalking community will keep our eyes on the OpenSearch project, and look forward to their first GA release.

NOTE: we have to add a warning NOTICE to the Elasticsearch storage doc:

NOTICE: Elastic announced through their blog that Elasticsearch will be moving over to a Server Side Public License (SSPL), which is incompatible with Apache License 2.0. This license change is effective from Elasticsearch version 7.11. So please choose the suitable Elasticsearch version according to your usage.

Blog: [Resolved][License Issue] Tencent Cloud TSW service violates the Apache 2.0 License when using SkyWalking.

Fri, 22 Jan 2021 00:00:00 +0000

Apache SkyWalking is an open source APM for distributed system, Apache Software Foundation top-level project.

At Jan. 11th, 2021, we noticed the Tencent Cloud Service, Tencent Service Watcher - TSW, for first time. Due to the similar short name, which SkyWalking is also called SW in the community, we connected with the service team of Tencent Cloud, and kindly asked.

They used to replay, TSW is purely developed by Tencent team itself, which doesn’t have any code dependency on SkyWalking.. We didn’t push harder.

But one week later, Jan 18th, 2021, our V.P., Sheng got the report again from Haoyang SkyWalking PMC member, through WeChat DM(direct message),. He provided complete evidence to prove TSW actually re-distributed the SkyWalking’s Java agent. We keep one copy of their agent’s distribution(at Jan. 18th), you could be downloaded here.

Some typically evidences are here

ServiceManager is copied and package-name changed in the TSW’s agent.
ContextManager is copied and ackage-name changed in the TSW’s agent.

At the same time, we checked their tsw-client-package.zip, it didn’t include the SkyWalking’s LICENSE and NOTICE. Also, they didn’t mention TSW agent is the re-ditribution SkyWalking on their website.

With all above information, we had enough reason to believe, from the tech perspective, they were violating the Apache 2.0 License.

From the 18th Jan., 2021, we sent mail [Apache 2.0 License Violation] Tencent Cloud TSW service doesn't follow the Apache 2.0 License to brief the SkyWalking PMC, and took the following actions to connect with Tencent.

Made direct call to Tencent Open Source Office.
Connected with Tencent Cloud TVP program committee, as Sheng Wu(Our VP) is a Tencent Cloud TVP.
Talked with the Tencent Cloud team lead.

In all above channels, we provided the evidences of copy-redistribution hebaviors, requested them to revaluate their statements on the website, and follow the License’s requirements.

Resolution

At Jan. 19th night, UTC+8, 2021. We received response from the Tencent cloud team. They admited their violation behaviors, and did following changes

Tencent Cloud TSW service page states, the agent is the fork version(re-distribution) of Apache SkyWalking agent.
TSW agent distributions include the SkyWalking’s License and NOTICE. Below is the screenshot, you could download from their product page. We keep a copy of their Jan. 19th 2021 at here.

We have updated the status to the PMC mail list. This license violation issue has been resolved for now.

The SkyWalking community and program management committee will keep our eyes on Tencent TSW.

Blog: Response to Elastic 2021 License Change

Sun, 17 Jan 2021 00:00:00 +0000

Elastic announced their license change, Upcoming licensing changes to Elasticsearch and Kibana.

We are moving our Apache 2.0-licensed source code in Elasticsearch and Kibana to be dual licensed under Server Side Public License (SSPL) and the Elastic License, giving users the choice of which license to apply. This license change ensures our community and customers have free and open access to use, modify, redistribute, and collaborate on the code. It also protects our continued investment in developing products that we distribute for free and in the open by restricting cloud service providers from offering Elasticsearch and Kibana as a service without contributing back. This will apply to all maintained branches of these two products and will take place before our upcoming 7.11 release. Our releases will continue to be under the Elastic License as they have been for the last three years.

Also, they provide the FAQ page for more information about the impact for the users, developers, and vendors.

In the perspective of Apache Software Foundation, SSPL has been confirmed as a Catalog X LICENSE(https://www.apache.org/legal/resolved.html#category-x), which means hard-dependency as a part of the core is not allowed. With that, we can’t only focus on it anymore. We need to consider other storage options. Right now, we still have InfluxDB, TiDB, H2 server still in Apache 2.0 licensed. Right now, we still have InfluxDB, TiDB, H2 server as storage options still in Apache 2.0 licensed.

As one optional plugin, we need to focus on the client driver license. Right now, we are only using ElasticSearch 7.5.0 and 6.3.2 drivers, which are both Apache 2.0 licensed. So, we are safe. For further upgrade, here is their announcement. They answer these typical cases in the FAQ page.

I build a SaaS application using Elasticsearch as the backend, how does this affect me?

This source code license change should not affect you - you can use our default distribution or develop applications on top of it for free, under the Elastic License. This source-available license does not contain any copyleft provisions and the default functionality is free of charge. For a specific example, you can see our response to a question around this at Magento.

Our users still could use, redistribute, sale the products/services, based on SkyWalking, even they are using self hosting Elastic Search unmodified server.

I’m using Elasticsearch via APIs, how does this change affect me?

This change does not affect how you use client libraries to access Elasticsearch. Our client libraries remain licensed under Apache 2.0, with the exception of our Java High Level Rest Client (Java HLRC). The Java HLRC has dependencies on the core of Elasticsearch, and as a result this client library will be licensed under the Elastic License. Over time, we will eliminate this dependency and move the Java HLRC to be licensed under Apache 2.0. Until that time, for the avoidance of doubt, we do not consider using the Java HLRC as a client library in development of an application or library used to access Elasticsearch to constitute a derivative work under the Elastic License, and this will not have any impact on how you license the source code of your application using this client library or how you distribute it.

The client driver license incompatible issue will exist, we can’t upgrade the driver(s) until they release the Apache 2.0 licensed driver jars. But users are still safe to upgrade the drivers by themselves.

Apache SkyWalking will discuss the further actions here. If you have any question, welcome to ask. In the later 2021, we will begin to invest the posibility of creating SkyWalking’s observability database implementation.

Blog: Celebrate SkyWalking single repository hits the 300 contributors mark

Tue, 22 Dec 2020 00:00:00 +0000

Apache SkyWalking is an open source APM for distributed system. Provide tracing, service mesh observability, metrics analysis, alarm and visualization.

Just 11 months ago, on Jan. 20th, 2020, SkyWalking hit the 200 contributors mark. With the growth of the project and the community, SkyWalking now includes over 20 sub(ecosystem) projects covering multiple language agents and service mesh, integration with mature open source projects, like Prometheus, Spring(Sleuth), hundreds of libraries to support all tracing/metrics/logs fields. In the past year, the number of contributors grows super astoundingly , and all its metrics point to its community vibrancy. Many corporate titans are already using SkyWalking in a large-scale production environment, including, Alibaba, Huawei, Baidu, Tencent, etc.

Recently, our SkyWalking main repository overs 300 contributors.

Our website has thousands of views from most countries in the world every week.

Although we know that, the metrics like GitHub stars and the numbers of open users and contributors, are not a determinant of vibrancy, they do show the trend, we are very proud to share the increased numbers here, too.

We double those numbers and are honored with the development of our community.

Thank you, all of our contributors. Not just these 300 contributors of the main repository, or nearly 400 contributors in all repositories, counted by GitHub. There are countless people contributing codes to SkyWalking’s subprojects, ecosystem projects, and private fork versions; writing blogs and guidances, translating documents, books, and presentations; setting up learning sessions for new users; convincing friends to join the community as end-users, contributors, even committers. Companies behinds those contributors support their employees to work with the community to provide feedback and contribute the improvements and features upstream. Conference organizers share the stages with speakers from the SkyWalking community.

SkyWalking can’t make this happen without your help. You made this community extraordinary.

At this crazy distributed computing and cloud native age, we as a community could make DEV, OPS, and SRE teams’ work easier by locating the issue(s) in the haystack quicker than before, like why we named the project as SkyWalking, we will have a clear site line when you stand on the glass bridge Skywalk at Grand Canyon West.

376 Contributors counted by GitHub account are following. Dec. 22st, 2020. Generated by a tool deveoped by Yousa

1095071913
50168383
Ahoo-Wang
AirTrioa
AlexanderWert
AlseinX
Ax1an
BFergerson
BZFYS
CharlesMaster
ChaunceyLin5152
CommissarXia
Cvimer
Doublemine
ElderJames
EvanLjp
FatihErdem
FeynmanZhou
Fine0830
FingerLiu
Gallardot
GerryYuan
HackerRookie
Heguoya
Hen1ng
Humbertzhang
IanCao
IluckySi
Indifer
J-Cod3r
JaredTan95
Jargon96
Jijun
JohnNiang
Jozdortraz
Jtrust
Just-maple
KangZhiDong
LazyLei
LiWenGu
Liu-XinYuan
Miss-you
O-ll-O
Patrick0308
QHWG67
Qiliang
RandyAbernethy
RedzRedz
Runrioter
SataQiu
ScienJus
SevenPointOld
ShaoHans
Shikugawa
SoberChina
SummerOfServenteen
TJ666
TerrellChen
TheRealHaui
TinyAllen
TomMD
ViberW
Videl
WALL-E
WeihanLi
WildWolfBang
WillemJiang
Wooo0
XhangUeiJong
Xlinlin
YczYanchengzhe
YoungHu
YunaiV
ZhHong
ZhuoSiChen
ZS-Oliver
a198720
a526672351
acurtain
adamni135
adermxzs
adriancole
aeolusheath
agile6v
aix3
aiyanbo
ajanthan
alexkarezin
alonelaval
amogege
amwyyyy
arugal
ascrutae
augustowebd
bai-yang
beckhampu
beckjin
beiwangnull
bigflybrother
bostin
brucewu-fly
c1ay
candyleer
carlvine500
carrypann
cheenursn
cheetah012
chenpengfei
chenvista
chess-equality
chestarss
chidaodezhongsheng
chopin-d
clevertension
clk1st
cngdkxw
codeglzhang
codelipenghui
coder-yqj
coki230
coolbeevip
crystaldust
cui-liqiang
cuiweiwei
cyberdak
cyejing
dagmom
dengliming
devkanro
devon-ye
dimaaan
dingdongnigetou
dio
dmsolr
dominicqi
donbing007
dsc6636926
duotai
dvsv2
dzx2018
echooymxq
efekaptan
eoeac
evanxuhe
feelwing1314
fgksgf
fuhuo
geektcp
geomonlin
ggndnn
gitter-badger
glongzh
gnr163
gonedays
grissom-grissom
grissomsh
guodongq
guyukou
gxthrj
gzshilu
hailin0
hanahmily
haotian2015
haoyann
hardzhang
harvies
hepyu
heyanlong
hi-sb
honganan
hsoftxl
huangyoje
huliangdream
huohuanhuan
innerpeacez
itsvse
jasonz93
jialong121
jinlongwang
jjlu521016
jjtyro
jmjoy
jsbxyyx
justeene
juzhiyuan
jy00464346
kaanid
karott
kayleyang
kevinyyyy
kezhenxu94
kikupotter
kilingzhang
killGC
klboke
ksewen
kuaikuai
kun-song
kylixs
landonzeng
langke93
langyan1022
langyizhao
lazycathome
leemove
leizhiyuan
libinglong
lilien1010
limfriend
linkinshi
linliaoy
liuhaoXD
liuhaoyang
liuyanggithup
liuzhengyang
liweiv
lkxiaolou
llissery
louis-zhou
lpf32
lsyf
lucperkins
lujiajing1126
lunamagic1978
lunchboxav
lxliuxuankb
lytscu
lyzhang1999
magic-akari
makingtime
maolie
masterxxo
maxiaoguang64
membphis
mestarshine
mgsheng
michaelsembwever
mikkeschiren
mm23504570
momo0313
moonming
mrproliu
muyun12
nacx
neatlife
neeuq
nic-chen
nikitap492
nileblack
nisiyong
novayoung
oatiz
oflebbe
olzhy
onecloud360
osiriswd
peng-yongsheng
pengweiqhca
potiuk
purgeyao
qijianbo010
qinhang3
qiuyu-d
qqeasonchen
qxo
raybi-asus
refactor2
remicollet
rlenferink
rootsongjc
rovast
scolia
sdanzo
seifeHu
shiluo34
sikelangya
simonlei
sk163
snakorse
songzhendong
songzhian
sonxy
spacewander
stalary
stenio2011
stevehu
stone-wlg
sungitly
surechen
swartz-k
sxzaihua
tanjunchen
tankilo
taskmgr
tbdpmi
terranhu
terrymanu
tevahp
thanq
thebouv
tianyuak
tincopper
tinyu0
tom-pytel
tristaZero
tristan-tsl
trustin
tsuilouis
tuohai666
tzsword-2020
tzy1316106836
vcjmhg
vision-ken
viswaramamoorthy
wankai123
wbpcode
web-xiaxia
webb2019
weiqiang333
wendal
wengangJi
wenjianzhang
whfjam
wind2008hxy
withlin
wqr2016
wu-sheng
wuguangkuo
wujun8
wuxingye
x22x22
xbkaishui
xcaspar
xiaoxiangmoe
xiaoy00
xinfeingxia85
xinzhuxiansheng
xudianyang
yanbw
yanfch
yang-xiaodong
yangxb2010000
yanickxia
yanmaipian
yanmingbi
yantaowu
yaowenqiang
yazong
ychandu
ycoe
yimeng
yu199195
yuqichou
yuyujulin
yymoth
zaunist
zaygrzx
zcai2
zeaposs
zhang98722
zhanghao001
zhangjianweibj
zhangkewei
zhangsean
zhaoyuguang
zhentaoJin
zhousiliang163
zhuCheer
zifeihan
zkscpqm
zoidbergwill
zoumingzm
zouyx
zshit
zxbu
zygfengyuwuzu

Blog: [Video] Another backend storage solution for the APM system

Sat, 21 Nov 2020 00:00:00 +0000

The APM system provides the tracing or metrics for distributed systems or microservice architectures. Back to APM themselves, they always need backend storage to store the necessary massive data. What are the features required for backend storage? Simple, fewer dependencies, widely used query language, and the efficiency could be into your consideration. Based on that, traditional SQL databases (like MySQL) or NoSQL databases would be better choices. However, this topic will present another backend storage solution for the APM system viewing from NewSQL. Taking Apache Skywalking for instance, this talking will share how to make use of Apache ShardingSphere, a distributed database middleware ecosystem to extend the APM system’s storage capability.

As a senior DBA worked at JD.com, the responsibility is to develop the distributed database and middleware, and the automated management platform for database clusters. As a PMC of Apache ShardingSphere, I am willing to contribute to the OS community and explore the area of distributed databases and NewSQL.

Blog: [Video] Improve Apache APISIX observability with Apache SkyWalking

Sat, 21 Nov 2020 00:00:00 +0000

Apache APISIX is a cloud-native microservices API gateway, delivering the ultimate performance, security, open-source and scalable platform for all your APIs and microservices. Apache SkyWalking: an APM(application performance monitor) system, especially designed for microservices, cloud-native and container-based (Docker, Kubernetes, Mesos) architectures. Through the powerful plug-in mechanism of Apache APISIX, Apache Skywalking is quickly supported, so that we can see the complete life cycle of requests from the edge to the internal service. Monitor and manage each request in a visual way, and improve the observability of the service.

Blog: [Video] SourceMarker - Continuous Feedback for Developers

Sat, 21 Nov 2020 00:00:00 +0000

Today’s monitoring solutions are geared towards operational tasks, displaying behavior as time-series graphs inside dashboards and other abstractions. These abstractions are immensely useful but are largely designed for software operators, whose responsibilities require them to think in systems, rather than the underlying source code. This is problematic given that an ongoing trend of software development is the blurring boundaries between building and operating software. This trend makes it increasingly necessary for programming environments to not just support development-centric activities, but operation-centric activities as well. Such is the goal of the feedback-driven development approach. By combining IDE and APM technology, software developers can intuitively explore multiple dimensions of their software simultaneously with continuous feedback about their software from inception to production.

Brandon Fergerson is an open-source software developer who does not regard himself as a specialist in the field of programming, but rather as someone who is a devoted admirer. He discovered the beauty of programming at a young age and views programming as an art and those who do it well to be artists. He has an affinity towards getting meta and combining that with admiration of programming, has found source code analysis to be exceptionally interesting. Lately, his primary focus involves researching and building AI-based pair programming technology.

Blog: [Video] The history of distributed tracing storage

Sat, 21 Nov 2020 00:00:00 +0000

Over the past few years, and coupled with the growing adoption of microservices, distributed tracing has emerged as one of the most commonly used monitoring and troubleshooting methodologies. New tracing tools are increasingly being introduced, driving adoption even further. One of these tools is Apache SkyWalking, a popular open-source tracing, and APM platform. This talk explores the history of the SkyWalking storage module, shows the evolution of distributed tracing storage layers, from the traditional relational database to document-based search engine. I hope that this talk contributes to the understanding of history and also that it helps to clarify the different types of storage that are available to organizations today.

Hongtao Gao is the engineer of tetrate.io and the former Huawei Cloud expert. One of PMC members of Apache SkyWalking and participates in some popular open-source projects such as Apache ShardingSphere and Elastic-Job. He has an in-depth understanding of distributed databases, container scheduling, microservices, ServicMesh, and other technologies.

Blog: [Video] Apache grows in China

Fri, 20 Nov 2020 00:00:00 +0000

Sheng Wu is a founding engineer at tetrate.io, leads the observability for service mesh and hybrid cloud. A searcher, evangelist, and developer in the observability, distributed tracing, and APM. He is a member of the Apache Software Foundation. Love open source software and culture. Created the Apache SkyWalking project and being its VP and PMC member. Co-founder and PMC member of Apache ShardingSphere. Also as a PMC member of Apache Incubator and APISIX. He is awarded as Microsoft MVP, Alibaba Cloud MVP, Tencent Cloud TVP.

In the Apache FY2020 report, China is on the top of the download statistics. More China initiated projects joined the incubator, and some of them graduated as the Apache TLP. Sheng joined the Apache community since 2017, in the past 3 years, he witnessed the growth of the open-source culture and Apache way in China. Many developers have joined the ASF as new contributors, committers, foundation members. Chinese enterprises and companies paid more attention to open source contributions, rather than simply using the project like before. In the keynote, he would share the progress about China embracing the Apache culture, and willing of enhancing the whole Apache community.

Blog: Observability at Scale: SkyWalking it is

Tue, 11 Aug 2020 00:00:00 +0000

Author: Sheng Wu
Original link, Tetrate.io blog

SkyWalking, a top-level Apache project, is the open source APM and observability analysis platform that is solving the problems of 21st-century systems that are increasingly large, distributed, and heterogenous. It’s built for the struggles system admins face today: To identify and locate needles in a haystack of interdependent services, to get apples-to-apples metrics across polyglot apps, and to get a complete and meaningful view of performance.

SkyWalking is a holistic platform that can observe microservices on or off a mesh, and can provide consistent monitoring with a lightweight payload.

Let’s take a look at how SkyWalking evolved to address the problem of observability at scale, and grew from a pure tracing system to a feature-rich observability platform that is now used to analyze deployments that collect tens of billions of traces per day.

Designing for scale

When SkyWalking was first initialized back in 2015, its primary use case was monitoring the first-generation distributed core system of China Top Telecom companies, China Unicom and China Mobile. In 2013-2014, the telecom companies planned to replace their old traditional monolithic applications with a distributed system. Supporting a super-large distributed system and scaleablity were the high-priority design goals from Day one. So, what matters at scale?

Pull vs. push

Pull and push modes relate to the direction of data flow. If the agent collects data and pushes them to the backend for further analysis, we call it “push” mode. Debate over pull vs. push has gone on for a long time. The key for an observability system is to minimize the cost of the agent, and to be generally suitable for different kinds of observability data.

The agent would send the data out a short period after it is collected. Then, we would have less concern about overloading the local cache. One typical case would be endpoint (URI of HTTP, service of gRPC) metrics. Any service could easily have hundreds, even thousands of endpoints. An APM system must have these metrics analysis capabilities.

Furthermore, metrics aren’t the only thing in the observability landscape; traces and logs are important too. SkyWalking is designed to provide a 100% sampling rate tracing capability in the production environment. Clearly, push mode is the only solution.

At the same time, using push mode natively doesn’t mean SkyWalking can’t do data pulling. In recent 8.x releases, SkyWalking supports fetching data from Prometheus-instrumented services for reducing the Non-Recurring Engineering of the end users. Also, pull mode is popular in the MQ based transport, typically as a Kafka consumer. The SkyWalking agent side uses the push mode, and the OAP server uses the pull mode.

The conclusion: push mode is the native way, but pull mode works in some special cases too.

Metrics analysis isn’t just mathematical calculation

Metrics rely on mathematical theories and calculations. Percentile is a good measure for identifying the long tail issue, and reasonable average response time and successful rate are good SLO(s). But those are not all. Distributed tracing provides not just traces with detailed information, but high values metrics that can be analyzed.

The service topology map is required from Ops and SRE teams for the NOC dashboard and confirmation of system data flow. SkyWalking uses the STAM (Streaming Topology Analysis Method) to analyze topology from the traces, or based on ALS (Envoy Access Log Service) in the service mesh environment. This topology and metrics of nodes (services) and lines (service relationships) can’t be pulled from simple metrics SDKs.

As with fixing the limitation of endpoint metrics collection, SkyWalking needs to do endpoint dependency analysis from trace data too. Endpoint dependency analysis provides more important and specific information, including upstream and downstream. Those dependency relationships and metrics help the developer team to locate the boundaries of a performance issue, to specific code blocks.

Pre-calculation vs. query stage calculation?

Query stage calculation provides flexibility. Pre-calculation, in the analysis stage, provides better and much more stable performance. Recall our design principle: SkyWalking targets a large-scale distributed system. Query stage calculation was very limited in scope, and most metrics calculations need to be pre-defined and pre-calculated. The key of supporting large datasets is reducing the size of datasets in the design level. Pre-calculation allows the original data to be merged into aggregated results downstream, to be used in a query or even for an alert check.

TTL of metrics is another important business enabler. With the near linear performance offered by queries because of pre-calculation, with a similar query infrastructure, organizations can offer higher TTL, thereby providing extended visibility of performance.

Speaking of alerts, query-stage calculation also means the alerting query is required to be based on the query engine. But in this case, when the dataset increasing, the query performance could be inconsistent. The same thing happens in a different metrics query.

Cases today

Today, SkyWalking is monitoring super large-scale distributed systems in many large enterprises, including Alibaba, Huawei, Tencent, Baidu, China Telecom, and various banks and insurance companies. The online service companies have more traffic than the traditional companies, like banks and telecom suppliers.

SkyWalking is the observability platform used for a variety of use cases for distributed systems that are super-large by many measures:

Lagou.com, an online job recruitment platform
- SkyWalking is observing >100 services, 500+ JVM instances
- SkyWalking collects and analyzes 4+ billion traces per day to analyze performance data, including metrics of 300k+ endpoints and dependencies
- Monitoring >50k traffic per second in the whole cluster
Yonghui SuperMarket, online service
- SkyWalking analyzes at least 10+ billion (3B) traces with metrics per day
- SkyWalking’s second, smaller deployment, analyzes 200+ million traces per day
Baidu, internet and AI company, Kubernetes deployment
- SkyWalking collects 1T+ traces a day from 1,400+ pods of 120+ services
- Continues to scale out as more services are added
Beike Zhaofang(ke.com), a Chinese online property brokerage backed by Tencent Holdings and SoftBank Group
- Has used SkyWalking from its very beginning, and has two members in the PMC team.
- Deployments collect 16+ billion traces per day
Ali Yunxiao, DevOps service on the Alibaba Cloud,
- SkyWalking collects and analyzes billions of spans per day
- SkyWalking keeps AliCloud’s 45 services and ~300 instances stable
A department of Alibaba TMall, one of the largest business-to-consumer online retailers, spun off from Taobao
- A customized version of SkyWalking monitors billions of traces per day
- At the same time, they are building a load testing platform based on SkyWalking’s agent tech stack, leveraging its tracing and context propagation cabilities

Conclusion

SkyWalking’s approach to observability follows these principles:

Understand the logic model: don’t treat observability as a mathematical tool.
Identify dependencies first, then their metrics.
Scaling should be accomplished easily and natively.
Maintain consistency across different architectures, and in the performance of APM itself.

Resources

Read about the SkyWalking 8.1 release highlights.
Get more SkyWalking updates on Twitter.
Sign up to hear more about SkyWalking and observability from Tetrate.

Blog: SkyWalking hits 200 contributors mark

Mon, 20 Jan 2020 00:00:00 +0000

Author: Wu Sheng, tetrate.io, SkyWalking original creator, SkyWalking V.P.
GitHub, Twitter, Linkedin

The SkyWalking project provides distributed tracing, topology map analysis, service mesh telemetry analysis, metrics analysis and a super cool visualization targeting distributed systems in k8s or traditional VM deployments.

The project is widely used in Alibaba, Huawei, Tencent, DiDi, xiaomi, Pingan, China’s top 3 telecom companies (China Mobile, China telecom, China Unicom), airlines, banks and more. It has over 140 company users listed on our powered by page.

Today, we welcome and celebrate reaching 200 code contributors on our main repo. We hereby mark this milestone as official today, : Jan. 20th 2020.

At this great moment, I would like to share SkyWalking’s 4-year open source journey.

I wrote the first line on Nov. 1st, 2015, guiding people to understand a distributed system just as micro-services and distributed architecture were becoming popular. In the first 2 years, I never thought it would become such a big and active community. I didn’t even expect it would be an open source project. Initially, the goal was primarily to teach others about distributed tracing and analysis.

It was a typical open source project in obscurity in its first two years. But people still showed up, asked questions, and tried to improve the project. I got several invitations to share the project at local meetups.All these made me realize people really needed a good open source APM project.

In 2017, I decided to dedicate myself as much as possible to make the project successful, and it became my day job. To be honest, I had no clue about how to do that; at that time in China, it was rare to have this kind of job. So, I began to ask friends around me, “Do you want to collaborate on the open source APM with me?” Most people were busy and gave a clear NO, but two of them agreed to help: Xin Zhang and Yongsheng Peng. We built SkyWalking 3.x and shared the 3.2 release at GOPS Shanghai, China.

It became the first adoption version used in production

Compared to today’s SkyWalking, it was a toy prototype, but it had the same tracing design, protocol and analysis method.

That year the contributor team was 15-20, and the project had obvious potential to expand. I began to consider bringing the project into a worldwide, top-level open source foundation. Thanks to our initial incubator mentors, Michael Semb Wever, William Jiang, and Luke Han, this really worked. At the end of 2017, SkyWalking joined the Apache Incubator, and kept following the Apache Way to build community. More contributors joined the community.

With more people spending time on the project collaborations, including codes, tests, blogs, conference talks, books and uses of the project, a chemical reaction happens. New developers begin to provide bug fixes, new feature requirements and new proposals. At the moment of graduation in spring 2019, the project had 100 contributors. Now, only 9 months later, it’s surged to 200 super quickly. They enhance the project and extend it to frontiers we never imaged: 5 popular language agents, service mesh adoption, CLI tool, super cool visualization. We are even moving on thread profiling, browser performance and Nginx tracing NOW.

Over the whole 4+ years open source journey, we have had supports from leaders in the tracing open source community around the world, including Adrian Cole, William Jiang, Luke Han, Michael Semb Wever, Ben Sigelman, and Jonah Kowall. And we’ve had critical foundations’ help, especially Apache Software Foundation and the Cloud Native Computing Foundation.

Our contributors also have their support from their employers, including, to the best of my knowledge, Alibaba, Huawei, China Mobile, ke.com, DaoCloud, Lizhi.fm, Yonghui Supermarket, and dangdang.com. I also have support from my employers, tetrate.io, Huawei, and OneAPM.

Thanks to our 200+ contributors and the companies behind them. You make this magic happen.