new-plugin

# SkyWalking Java Agent Plugin Development Develop a new plugin for the Apache SkyWalking Java Agent. Ask the user what library/framework to instrument and what to observe (tracing, metrics, or both), then generate all required files. ## Step 0 - Gather Requirements Ask the user: 1. **Target library/framework** and version range (e.g., "Jedis 3.x-4.x", "Spring Kafka 2.7+") 2. **Observation type**: tracing plugin, meter plugin, or both 3. **Plugin category**: SDK plugin (default), bootstrap plugin, or optional plugin 4. **Span type needed**: Entry (server/consumer), Exit (client/producer), Local (internal), or combination If the user already provided this info, skip asking. ## Step 1 - Understand the Library and Identify Interception Points This is the most critical step. Do NOT jump to picking method names. Follow these phases in order. ### Phase 1: Understand How the Library Is Used Read the target library's documentation, quickstart guides, or sample code. Understand the **user-facing API** — how developers create clients, make calls, and handle responses. This tells you: - What objects are long-lived (clients, connections, pools) vs. per-request (requests, commands) - Where configuration lives (server address, credentials, timeouts) - Whether the library is sync, async, or reactive - Whether it uses callbacks, futures, or blocking calls Example thought process for a Redis client: ``` User creates: RedisClient client = RedisClient.create("redis://localhost:6379"); User connects: StatefulRedisConnection conn = client.connect(); User executes: conn.sync().get("key"); // or conn.async().get("key") ``` This tells you: connection holds the server address, commands are executed on the connection. ### Phase 2: Trace the Execution Flow Starting from the user-facing API, trace inward through the library source code to understand the execution workflow: 1. What happens when the user calls the API method? 2. Where does the request object get built? 3. Where is the actual network I/O or dispatch? 4. Where is the response available? 5. For RPC/MQ: where are headers/metadata accessible for inject/extract? **Key question at each point:** What data is directly accessible as method arguments, return values, or fields on `this`? You want interception points where you can read the data you need **without reflection**. ### Phase 3: Choose Interception Points Pick interception points based on these principles: **Principle 1: Data accessibility without reflection.** Choose methods where the information you need (peer address, operation name, request/response details, headers for inject/extract) is directly available as method arguments, return values, or accessible through the `this` object's public API. **Never use reflection to read private fields.** If the data is not accessible at one method, look at a different point in the execution flow. If the target class is **package-private** (e.g., `final class` without `public`), you cannot import or cast to it. **Same-package helper classes do NOT work** because the agent and application use different classloaders — Java treats them as different runtime packages even with the same package name (`IllegalAccessError`). Use `setAccessible` reflection to call public methods: ```java try { java.lang.reflect.Method method = objInst.getClass().getMethod("publicMethodName"); method.setAccessible(true); // Required for package-private class Object result = method.invoke(objInst); } catch (Exception e) { LOGGER.warn("Failed to access method", e); } ``` **Principle 2: Use `EnhancedInstance` dynamic field to propagate context inside the library.** This is the primary mechanism for passing data between interception points. The agent adds a dynamic field to every enhanced class via `EnhancedInstance`. Use it to: - Store server address (peer) at connection/client creation time, retrieve it at command execution time - Store request info at request-build time, retrieve it at send time - Pass span references from the initiating method to the completion callback **Do NOT use `Map` or other caches to store per-instance context.** Always use the dynamic field on the relevant `EnhancedInstance`. Maps introduce memory leaks, concurrency issues, and are slower than the direct field access that `EnhancedInstance` provides. **Principle 3: Intercept the minimal set of methods.** Prefer one well-chosen interception point over many surface-level ones. If a library has 20 command methods that all flow through a single `dispatch()` method internally, intercept `dispatch()` — not all 20. **Principle 4: Pick points where you can do inject/extract for cross-process propagation.** For RPC/HTTP/MQ plugins, you need to inject trace context into outgoing requests (ExitSpan) or extract from incoming requests (EntrySpan). The interception point MUST be where headers/metadata are writable (inject) or readable (extract). If headers are not accessible at the execution method, look for: - A request builder/decorator stage where headers can be added - A channel/transport layer where metadata is attached - A message properties object accessible from the method arguments **Principle 5: Consider the span lifecycle across threads.** If the library dispatches work asynchronously: - Identify where the work is submitted (original thread) and where the result arrives (callback/future thread) - You may need interception points in both threads - Use `EnhancedInstance` dynamic field on the task/callback/future object to carry the span or `ContextSnapshot` across the thread boundary - Use `prepareForAsync()` / `asyncFinish()` if the span must stay open across threads ### Phase 4: Map Out the Interception Plan Before writing code, create a clear plan listing: | Target Class | Method/Constructor | What to Do | Data Available | |---|---|---|---| | `XxxClient` | constructor | Store peer address in dynamic field | host, port from args | | `XxxConnection` | `execute(Command)` | Create ExitSpan, inject carrier into command headers | command name, peer from dynamic field | | `XxxResponseHandler` | `onComplete(Response)` | Set response tags, stop span or asyncFinish | status code, error from args | For each interception point, verify: - [ ] The data I need is readable from method args, `this`, or `EnhancedInstance` dynamic field — no reflection needed - [ ] For inject: I can write headers/metadata through a public API on a method argument - [ ] For extract: I can read headers/metadata through a public API on a method argument - [ ] The `this` object (or a method argument) will be enhanced as `EnhancedInstance`, so I can use the dynamic field ### Choosing Span Type | Scenario | Span Type | Requires | |----------|-----------|----------| | Receiving requests (HTTP server, MQ consumer, RPC provider) | EntrySpan | Extract ContextCarrier from incoming headers | | Making outgoing calls (HTTP client, DB, cache, MQ producer, RPC consumer) | ExitSpan | Peer address; inject ContextCarrier into outgoing headers (for RPC/HTTP/MQ) | | Internal processing (annotation-driven, local logic) | LocalSpan | Nothing extra | ### For Meter Plugins Meter plugins follow the same understand-then-intercept process, but the goal is to find objects that expose numeric state: - **Gauges**: Intercept the creation of pool/executor/connection-manager objects. Register a `MeterFactory.gauge()` with a supplier lambda that calls the object's own getter methods (e.g., `pool.getActiveCount()`). Store the gauge reference in the dynamic field if needed. - **Counters**: Intercept execution methods and call `counter.increment()` on each invocation. - **Histograms**: Intercept methods where duration or size is computable (measure between before/after, or read from response). ### Check Existing Plugins for Reference Before writing a new plugin, check if a similar library already has a plugin: ``` apm-sniffer/apm-sdk-plugin/ # 70+ standard plugins apm-sniffer/optional-plugins/ # Optional plugins apm-sniffer/bootstrap-plugins/ # JDK-level plugins ``` Similar libraries often share execution patterns. Study how an existing plugin for a similar library solved the same problems — especially how it chains dynamic fields across multiple interception points and where it does inject/extract. ### Verify Against Actual Source Code — Never Speculate **This applies to both new plugin development AND extending existing plugins to newer library versions.** When assessing whether a plugin works with a new library version, or when choosing interception points for a new plugin, you MUST read the **actual source code** of the target library at the specific version. Do NOT rely on: - Version number assumptions ("it's still 4.x so it should be compatible") - Changelog summaries (they don't list every internal class rename or method removal) - General knowledge about the library's public API (plugins intercept internal classes, which change without notice) **What to verify for each intercepted class/method:** 1. Does the target class still exist at the exact FQCN? (internal classes get renamed, extracted, or removed between minor versions) 2. Does the intercepted method still exist with a compatible signature? (parameters may be added/removed/reordered) 3. Do the witness classes still correctly distinguish versions? (a witness class that exists in both old and new versions won't prevent the plugin from loading on an incompatible version) 4. Do the runtime APIs called by the interceptor still exist? (e.g., calling `cluster.getDescription()` will crash if that method was removed, even if the plugin loaded successfully) **How to verify — clone the library source locally:** ```bash # Clone specific version tag to /tmp for easy source code inspection cd /tmp && git clone --depth 1 --branch {tag} https://github.com/{org}/{repo}.git {local-name} # Examples: git clone --depth 1 --branch r4.9.0 https://github.com/mongodb/mongo-java-driver.git mongo-4.9 git clone --depth 1 --branch v2.4.13.RELEASE https://github.com/spring-projects/spring-kafka.git spring-kafka-2.4 # Then grep/read the actual source files to check class/method existence grep -rn "getFilter\|getWriteRequests" /tmp/mongo-4.9/driver-core/src/main/com/mongodb/internal/operation/ ``` This is faster and more reliable than fetching individual files via raw GitHub URLs. You can `grep`, `diff` between versions, and trace the full execution path. **Also check for import-time class loading failures:** If a plugin helper class (not just the instrumentation class) imports a library class that was removed, the entire helper class will fail to load with `NoClassDefFoundError` at runtime. This silently breaks ALL functionality in that helper — not just the code paths using the removed class. Verify that every `import` statement in plugin support classes resolves to an existing class in the target version. **Real examples of why this matters:** - MongoDB driver 4.9 removed `InsertOperation`, `DeleteOperation`, `UpdateOperation` — `MongoOperationHelper` imported all three, causing the entire class to fail loading with `NoClassDefFoundError`, silently losing ALL `db.bind_vars` tags even for operations that still exist (like `FindOperation`, `AggregateOperation`) - MongoDB driver 4.11 removed `Cluster.getDescription()` — the plugin loads (witness classes pass) but crashes at runtime with `NoSuchMethodError` - Feign 12.2 moved `ReflectiveFeign$BuildTemplateByResolvingArgs` to `RequestTemplateFactoryResolver$BuildTemplateByResolvingArgs` — the path variable interception silently stops working - MariaDB 3.0 renamed every JDBC wrapper class (`MariaDbConnection` → `Connection`) — none of the plugin's `byName` matchers match anything **When extending `support-version.list` to add newer versions:** Before adding a version, verify that every class and method the plugin intercepts still exists in that version's source. A plugin test passing does not mean everything works — it only means the test scenario's specific code path exercised the intercepted methods. Missing interception points may go undetected if the test doesn't cover them. ## Step 2 - Create Plugin Module ### Directory Structure **SDK plugin** (most common): ``` apm-sniffer/apm-sdk-plugin/{framework}-{version}-plugin/ pom.xml src/main/java/org/apache/skywalking/apm/plugin/{framework}/v{N}/ define/ {Target}Instrumentation.java # One per target class {Target}Interceptor.java # One per interception concern {Target}ConstructorInterceptor.java # If intercepting constructors {PluginName}PluginConfig.java # If plugin needs configuration src/main/resources/ skywalking-plugin.def # Plugin registration src/test/java/org/apache/skywalking/apm/plugin/{framework}/v{N}/ {Target}InterceptorTest.java # Unit tests ``` **Bootstrap plugin** (for JDK classes): ``` apm-sniffer/bootstrap-plugins/{name}-plugin/ (same structure, but instrumentation class overrides isBootstrapInstrumentation) ``` **Optional plugin**: ``` apm-sniffer/optional-plugins/{name}-plugin/ (same structure as SDK plugin) ``` ### pom.xml Template ```xml <?xml version="1.0" encoding="UTF-8"?> <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> <modelVersion>4.0.0</modelVersion> <parent> <groupId>org.apache.skywalking</groupId> <artifactId>apm-sdk-plugin</artifactId> <!-- or bootstrap-plugins / optional-plugins --> <version>${revision}</version> </parent> <artifactId>{framework}-{version}-plugin</artifactId> <packaging>jar</packaging> <properties> <target-library.version>X.Y.Z</target-library.version> </properties> <dependencies> <!-- Target library - MUST be provided scope --> <dependency> <groupId>com.example</groupId> <artifactId>target-library</artifactId> <version>${target-library.version}</version> <scope>provided</scope> </dependency> </dependencies> </project> ``` **CRITICAL dependency rules:** - Target library: **always `provided` scope** (supplied by the application at runtime) - `apm-agent-core`: inherited from parent POM as `provided` - `apm-util`: inherited from parent POM as `provided` - Never bundle target library classes into the plugin JAR - If the plugin needs a 3rd-party utility not already in agent-core, discuss with maintainers first **CRITICAL compiler level rule:** - Do NOT set `maven.compiler.release` or `maven.compiler.source/target` unless the plugin source code itself uses JDK 9+ language features (e.g., `var`, records, sealed classes, `java.net.http.HttpClient`). - A `provided`-scope dependency targeting a higher JDK (e.g., Spring AI requires JDK 17+) does NOT require raising the compiler level — the plugin only references the library's API at compile time. - Bootstrap plugins like `jdk-httpclient-plugin` (which uses JDK 11 `HttpClient` API directly in plugin source code) legitimately need a higher compiler target. SDK plugins and optional plugins generally should not. ### Register in Parent POM Add the new module to the parent `pom.xml`: ```xml <modules> ... <module>{framework}-{version}-plugin</module> </modules> ``` ## Step 3 - Implement Instrumentation Class (V2 API) **ALWAYS use V2 API for new plugins.** V1 is legacy. ### Import Rules (Enforced by Checkstyle) Plugins may ONLY import from: - `java.*` - Java standard library - `org.apache.skywalking.*` - SkyWalking modules - `net.bytebuddy.*` - ByteBuddy (for matchers in instrumentation classes) **No other 3rd-party imports are allowed in instrumentation/activation files.** This is enforced by `apm-checkstyle/importControl.xml`. Interceptor classes CAN reference target library classes (they're loaded after the target library). ### Class Matching **CRITICAL: NEVER use `.class` references in instrumentation definitions.** Always use string literals. ```java // WRONG - breaks agent if class doesn't exist at runtime byName(SomeThirdPartyClass.class.getName()) takesArgument(0, SomeThirdPartyClass.class) // CORRECT - safe string literals byName("com.example.SomeThirdPartyClass") takesArgumentWithType(0, "com.example.SomeThirdPartyClass") ``` Available ClassMatch types (from `org.apache.skywalking.apm.agent.core.plugin.match`): | Matcher | Usage | Performance | |---------|-------|-------------| | `NameMatch.byName(String)` | Exact class name | Best (HashMap lookup) | | `MultiClassNameMatch.byMultiClassMatch(String...)` | Multiple exact names | Good | | `HierarchyMatch.byHierarchyMatch(String...)` | Implements interface / extends class | Expensive - avoid unless necessary | | `ClassAnnotationMatch.byClassAnnotationMatch(String...)` | Has annotation(s) | Moderate | | `MethodAnnotationMatch.byMethodAnnotationMatch(String...)` | Has method with annotation | Moderate | | `PrefixMatch.nameStartsWith(String...)` | Class name prefix | Moderate | | `RegexMatch.byRegexMatch(String...)` | Regex on class name | Expensive | | `LogicalMatchOperation.and(match1, match2)` | AND composition | Depends on operands | | `LogicalMatchOperation.or(match1, match2)` | OR composition | Depends on operands | **Prefer `NameMatch.byName()` whenever possible.** It uses a fast HashMap lookup. All other matchers require linear scanning. ### Method Matching (ByteBuddy ElementMatcher API) Common matchers from `net.bytebuddy.matcher.ElementMatchers`: ```java // By name named("methodName")